EP3493548B1 - Transmission device, transmission method, reception device, and reception method - Google Patents
Transmission device, transmission method, reception device, and reception method Download PDFInfo
- Publication number
- EP3493548B1 EP3493548B1 EP17834115.2A EP17834115A EP3493548B1 EP 3493548 B1 EP3493548 B1 EP 3493548B1 EP 17834115 A EP17834115 A EP 17834115A EP 3493548 B1 EP3493548 B1 EP 3493548B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- subtitle
- display
- region
- pes
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005540 biological transmission Effects 0.000 title claims description 33
- 238000000034 method Methods 0.000 title claims description 31
- 239000000523 sample Substances 0.000 description 96
- 230000032258 transport Effects 0.000 description 41
- 238000005516 engineering process Methods 0.000 description 14
- 238000004458 analytical method Methods 0.000 description 10
- 238000006243 chemical reaction Methods 0.000 description 9
- 239000000284 extract Substances 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 101100190466 Caenorhabditis elegans pid-3 gene Proteins 0.000 description 1
- 101000609957 Homo sapiens PTB-containing, cubilin and LRP1-interacting protein Proteins 0.000 description 1
- 101150109471 PID2 gene Proteins 0.000 description 1
- 102100039157 PTB-containing, cubilin and LRP1-interacting protein Human genes 0.000 description 1
- 230000002730 additional effect Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/85406—Content authoring involving a specific file format, e.g. MP4 format
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/23614—Multiplexing of additional data and video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/488—Data services, e.g. news ticker
- H04N21/4884—Data services, e.g. news ticker for displaying subtitles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/08—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division
- H04N7/087—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division with signal insertion during the vertical blanking interval only
- H04N7/088—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division with signal insertion during the vertical blanking interval only the inserted signal being digital
- H04N7/0884—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division with signal insertion during the vertical blanking interval only the inserted signal being digital for the transmission of additional display-information, e.g. menu for programme or channel selection
- H04N7/0885—Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band, e.g. by time division with signal insertion during the vertical blanking interval only the inserted signal being digital for the transmission of additional display-information, e.g. menu for programme or channel selection for the transmission of subtitles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/61—Network physical structure; Signal processing
- H04N21/6106—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network
- H04N21/6125—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network involving transmission via Internet
Definitions
- the present technology relates to a transmission device, a transmission method, a reception device, and a reception method, and particularly relates to, for example, a transmission device that transmits a document of subtitle text information.
- TTML Timed Text Markup Language
- W3C World Wide Web Consortium
- Patent Document 2 further proposes a streaming device for streaming data in the MP4 format, including subtitle data stored in fragment boxes.
- the subtitle data contains TTML document instances which are identified by instance classification distinguishing information.
- Patent Document 3 proposes a method for adapting the display time of subtitle data taking into account the time necessary for processing the encoded subtitle data, which is dependent on the size of the encoded data, and the subtitle presentation time.
- Patent Document 4 proposes a reproduction that enables the correct display of subtitles after conversion from TS to PS, even during trick play modes.
- the beginning of a subtitle, divided into subtitle packets is the time stamp aligned with the beginning of an I-picture, divided into I-picture packets.
- Patent Document 5 proposes a content reproduction device that reproduces content items containing audio, video and subtitles organized in a specified data structure.
- Each content item is split into segment data items which are described in a XML file.
- Each segment data item corresponds to a plurality of segments of playback time periods.
- To each playback time period corresponds a group of subtitle data items that are reproduced within each of the time periods.
- Patent Document 6 proposes a reproduction device that controls the display of timed subtitles.
- a subtitle encoder generates a subtitle stream having the text information and the display timing information.
- the subtitles are in the TTML format.
- the display start time is indicated in the PTS of a PES packet and the text information and display timing is indicated in the PES payload.
- Patent Document 7 proposes TTML subtitle transmission management per regions and samples with start and end time.
- An object of the present technology is to improve the transport efficiency of a document of subtitle text information.
- an object of the present technology is to allow region display end processing to be performed properly on the reception side in live delivery.
- a transmission device for transmitting a plurality of subtitles for different regions comprises:
- a reception device for receiving a plurality of subtitles for different regions comprises:
- a transmission method for transmitting a plurality of subtitles for different regions comprises:
- a reception method for receiving a plurality of subtitles for different regions comprises:
- the transport efficiency of a document of subtitle text information can improve.
- region display end processing can be performed properly on the reception side in live delivery. Note that the effects described in the present specification are, but are not limited to, just exemplifications, and thus additional effects may be provided.
- Fig. 1 illustrates an exemplary configuration of a transmission and reception system 10 as an embodiment.
- the transmission and reception system 10 includes a broadcast output system 100 and a television receiver 200.
- the broadcast output system 100 superimposes an MPEG-2 transport stream (hereinafter, simply referred to as a "transport stream”) TS as a multiplexed stream, on a broadcast wave, to transmit the transport stream TS.
- transport stream an MPEG-2 transport stream
- the transport stream TS has a subtitle stream together with a video stream and an audio stream.
- the video stream includes video PES packets each having coded image data disposed in a payload.
- the audio stream includes audio PES packets each having coded audio data disposed in a payload.
- the subtitle stream includes subtitle PES packets each having a segment packet disposed in a payload, according to the present embodiment, the segment packet having a document in TTML (TTML document) of subtitle text information having display timing information.
- TTML TTML document
- Such a subtitle PES packet is generated for every sample.
- each sample is, as conventionally known, a fixed sample in which the period thereof is fixed, or each sample is, as newly proposed, a variable sample in which the period thereof is variable in accordance with a variation period of subtitle display.
- subtitle display corresponding to each sample is individually referred to as an event.
- the transmission frequency of subtitle PES packets can be inhibited, so that transport efficiency can improve.
- region display end time For pre-authored subtitle delivery, because region display end time is known, the region display end time can be inserted into a TTML document. However, for live subtitle delivery, because region display end time is unknown at the beginning of a display start, it is impossible that the region display end time is inserted into a TTML document.
- region display-end-time information is given to the reception side with a first method or a second method. This arrangement allows region display end processing on the reception side to be performed properly on the reception side in live delivery.
- a PES packet having a segment packet having a TTML document in which region end-time information is filled, inserted in a payload, is newly generated and is transmitted.
- the first method is limited to, for the fixed samples, a sample unit thereof.
- a PES packet having a segment packet having a control command for giving region display-end-time information, inserted in a payload is newly generated and is transmitted.
- the television receiver 200 receives the transport stream TS sent by the broadcast wave from the broadcast output system 100.
- the transport stream TS has, as described above, the subtitle stream together with the video stream and the audio stream.
- the subtitle stream includes, as described above, the subtitle PES packets each having a segment packet having a TTML document, disposed in a payload.
- the television receiver 200 receives, in accordance with each sample, a subtitle PES packet for the sample before the beginning of the period of the sample.
- the television receiver 200 controls region display on the basis of the TTML document included in the subtitle PES packet.
- the television receiver 200 has the reception frequency of PES packets lower than that in a case where each sample is a fixed sample, so that a processing load is reduced.
- the television receiver 200 controls an end of the region display, on the basis of the TTML document in which region display-end-time information is filled, included in a subtitle PES packet received after the region display starts. Furthermore, in a similar case, with the second method adopted, the television receiver 200 acquires the region display-end-time information and controls an end of the region display, on the basis of the control command for giving region display-end-time information, included in a subtitle PES packet received after the region display starts. This arrangement allows the television receiver 200 to perform the region display end processing properly in the live delivery.
- Fig. 2 illustrates an exemplary configuration of a stream generation unit 110 of the broadcast output system 100.
- the stream generation unit 110 has a control unit 111, a video encoder 112, an audio encoder 113, a text format conversion unit 114, a subtitle encoder 115, and a TS formatter (multiplexer) 116.
- the control unit 111 includes, for example, a central processing unit (CPU), and controls the operation of each unit of the stream generation unit 110.
- the video encoder 112 inputs image data DV and performs coding to the image data DV, to generate a video stream (PES stream) including video PES packets each having coded image data in a payload.
- the audio encoder 113 inputs audio data DA and performs coding to the audio data DA, to generate an audio stream (PES stream) including audio PES packets each having coded audio data.
- the text format conversion unit 114 inputs text data (character code) DT as subtitle information, and acquires subtitle text information having display timing information, in a predetermined format.
- text data character code
- TTML or a TTML derived format is considered as the text information, but TTML is adopted in the present embodiment.
- Fig. 3 illustrates an exemplary structure of a document (file) in TTML.
- the TTML is described on an XML basis. Head and body are present in the TTML. Then, for example, metadata, styling, and layout are present in the head.
- the metadata includes, for example, title information and copyright information.
- a region identifier, and color information, font information, and position information in association with the region identifier are disposed in the styling.
- background color information and alignment information are disposed in association with the region identifier in the layout.
- subtitle text data to be displayed and timing information regarding a region display start and a region display end are disposed in association with the region identifier in the body.
- the subtitle encoder 115 generates a segment packet including a document in TTML acquired by the text format conversion unit 114 (timed text subtitling segment: TT subtitling segment). Then, the subtitle encoder 115 generates a subtitle stream (PES stream) including a subtitle PES packet having the segment packet disposed in a payload.
- TT subtitling segment timed text subtitling segment
- the subtitle encoder 115 generates a subtitle PES packet for every sample. Furthermore, because the region display end time is unknown initially for the live subtitle delivery, the subtitle encoder 115 cannot insert the region display end time into the TTML document included in a subtitle PES packet, and thus renders a blank.
- the subtitle encoder 115 generates a new subtitle PES packet in order to give the region display-end-time information to the reception side.
- the subtitle encoder 115 With the first method adopted, the subtitle encoder 115 generates a segment packet including a TTML document in which the region end-time information is filled, and newly generates a subtitle PES packet having the segment packet disposed in a payload.
- the subtitle encoder 115 With the second method adopted, the subtitle encoder 115 generates a segment packet including a control command for giving the region display-end-time information, and newly generates a subtitle PES packet having the segment packet disposed in a payload.
- the subtitle encoder 115 generates a segment packet including download information for downloading a font file designated in TTML font designation information, and disposes the segment packet in the payload of the PES packet.
- Fig. 4(a) illustrates an exemplary structure of a subtitle PES packet (PES_packet).
- a fixed pattern of "0x000001” is disposed in the 24-bit field of "PES_startcode_prefix".
- the 8-bit field of "stream_id” indicates a stream identifier.
- For the subtitle PES packet "10111101” that is a value indicating "private stream1”, is provided.
- the 16-bit field of "PES_packet_length” indicates a byte length thereafter as the length (size) of the PES packet.
- the field of "Optional_PES_header()" is present after "PES_packet_length". For example, the time stamp of PTS or DTS is disposed in the field. After the field, the field of "PES_packet_data_byte” is present. The field corresponds to a PES payload. "PES_data_byte_field()" for containing data is disposed in the field.
- Fig. 4(b) illustrates an exemplary structure of "PES_data_byte_field()".
- the 8-bit field of "data_identifier” indicates an identifier that identifies the type of data in a container portion. Because conventional subtitles (in bitmap) are indicated with “0x20”, identification of text can be made with a new value, for example, "0x21”.
- the 8-bit field of "subtitle_stream_id” indicates an identifier that identifies the type of a subtitle stream.
- a new value for example, "0x01” is adopted for the subtitle stream that transports text information, so that a distinction can be made from the subtitle stream "0x00" that transports the conventional bitmap.
- the field of "Subtitling_segments()" is present, following a pattern of "00001111".
- a segment packet is disposed in the field.
- the 8-bit field of "end_of_PES_data_field_marker” is present. The field is a marker indicating the end of the PES packet.
- Fig. 5(a) illustrates exemplary definitions of subtitle segment types (subtitle segment_type).
- "0xA0” indicates a segment packet that transports a TTML document.
- "0xA1” indicates a segment packet that transports a compressed TTML document.
- "0xA2” indicates a segment packet that transports a control command.
- "0xA3” indicates a segment packet that transports font download information.
- segments having different subtitle segment types in value can be disposed in one PES payload.
- Fig. 6 illustrates an exemplary structure of a subtitle stream (PES stream) including subtitle PES packets.
- the subtitle PES packets for example, a subtitle PES packet including only a segment packet that transports a TTML document, a subtitle PES packet including only a segment packet that transports a control command, and furthermore a subtitle PES packet including both thereof are present.
- Fig. 7 illustrates an exemplary structure of a TTML-document segment (TTL_document_segment()).
- Fig. 8 illustrates the descriptions of main information in the exemplary structure.
- the 8-bit field of "sync_byte” is a unique word indicating a segment start.
- the 8-bit field of "segment_type” is a segment type, and indicates the type of the segment.
- "segment_type” is "0xA0" (refer to Fig. 5(a) ).
- the 8-bit field of "number_of_regions” indicates the number of regions included in a TTML document.
- the 8-bit field of "region_id” is repeatedly present by the number of regions.
- the field of "region_id” indicates region identification information (id).
- the region identification information (id) corresponds to the region identification information (id) in the TTML document.
- the 16-bit field of "latency” indicates the delay in time from PTS until caption display (subtitle display) transported in byte data of "segment_data_field()" starts.
- the value is, for example, ten times the amount of actual time (by the second).
- the 16-bit field of "document_version_number” indicates information update of the TTML document. In a case where an update is made, the value is incremented by one.
- the 16-bit field of "segment_length” indicates a byte length thereafter as the length (size) of the TTML-document segment. After “segment_length”, the field of "segment_data_field()" is present. The field corresponds to the payload of the segment (segment packet), and the entire TTML document is disposed in the field.
- Fig. 9 illustrates an exemplary structure of a control-command segment (control_command_segment()).
- Fig. 10 illustrates the descriptions of main information in the exemplary structure.
- the 8-bit field of "sync_byte” is a unique word indicating a segment start.
- the 8-bit field of "segment_type” is a segment type, and indicates the type of the segment.
- "segment_type” is "0xA2" (refer to Fig. 5(a) ).
- the 8-bit field of "number_of_regions” indicates the number of regions.
- the 8-bit field of "region_id” and the 8-bit field of “command” are repeatedly present by the number of regions.
- the field of "region_id” indicates region identification information (id).
- the field of "latency” expresses the timing of command execution as a delay offset from PTS. The value is indicated with a tenfold value, and thus “1” and “16” express an execution delay of 0.1 seconds and an execution delay of 1.6 seconds, respectively.
- the field of "command” indicates a command. For example, "0x01” indicates an end of region display. "0x02” indicates that information relating to the region display is to be deleted from a buffer. "0x03" indicates that a region display state is to be indicated with a count (time count) at previously defined time intervals.
- time_offset indicates a time offset (time_offset) from the time stamp (timestamp) of a display end command.
- Fig. 11 illustrates an exemplary structure of a font-download-information segment (font_download_information_segment()).
- Fig. 12 illustrates the descriptions of main information in the exemplary structure.
- the 8-bit field of "sync_byte” is a unique word indicating a segment start.
- the 8-bit field of "segment_type” is a segment type, and indicates the type of the segment. For the control-command segment, "segment_type” is "0xA3" (refer to Fig. 5(a) ).
- the 16-bit field of "original_network_id” indicates identification information regarding a network through which download data is to be transported.
- the 16-bit field of "transport_stream_id” indicates identification information regarding an individual transport stream.
- the 16-bit field of "service_id” indicates identification information regarding a service in which downloading is to be performed. For a common object to be downloaded between delivery media, there is a possibility that, instead of in the host transport stream, a font file is sent in another transport stream. In that case, information regarding "original_network_id", “transport_stream_id”, and “service_id” can be designated as information for specifying a private section in a reference destination.
- the 8-bit field of "font_file_id” indicates an identification number assigned to the font file.
- the 24-bit field of "ISO_639_language_code” indicates a code having three characters allowing a language to be identified. For example, “jpn” indicates Japanese, and “eng” indicates English.
- the 8-bit field of "font_group_id” indicates identification information regarding a font group, and corresponds to a TTML generic family.
- the 8-bit field of "font_name_id” indicates an individual font name.
- the 8-bit field of "url_type” indicates the type of a server. For example, “0x01” indicates a font server (uncompressed URL) and “0x02” indicates a general server (uncompressed URL). “0x11” indicates a font server (compressed URL) and “0x12” indicates a general server (compressed URL).
- the 8-bit field of "url_string_length” indicates, in byte length, the length (size) of a character code portion indicating the character string of a URL, thereafter. The character code is disposed in the field of "char”.
- a TTML layer and a container layer will be further described.
- subtitle delivery produced at expense in time namely, pre-authored subtitle delivery will be described with fixed samples.
- Fig. 13 illustrates an exemplary TTML layer and an exemplary container layer in that case.
- Information regarding a region A is present in the TTML layer.
- the region A is displayed from timing T1 to timing T3.
- a start time "start time” corresponding to the point in time past by a delay LTa from Tf1 in which PTS is reflected is T1 and an end time "end time” is T3.
- T3 is previously known.
- the fact that the end time "end time” is T3 is initially written in a TTML document in addition to the fact that "start time” is T1, in relation to the region A.
- a region B is present in the TTML layer.
- the region B is displayed from timing T2 to timing T5, partially overlapping with the region A described above, temporally.
- a start time "start time” corresponding to the point in time past by a delay LTb from Tf1 in which PTS is reflected, is T2 and an end time "end time” is T5.
- T5 is previously known.
- the fact that the end time "end time” is T5 is initially written in a TTML document in addition to the fact that "start time” is T2, in relation to the region B.
- a region C is present in the TTML layer.
- the region C is displayed from timing T4 to timing T6, partially overlapping with the region B described above, temporally.
- a start time "start time” corresponding to the point in time past by a delay LTc from Tf3 in which PTS is reflected, is T4 and an end time "end time” is T6.
- T6 is previously known.
- the fact that the end time "end time” is T6 is initially written in a TTML document in addition to the fact that "start time” is T4, in relation to the region C.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- each PES packet is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 1, disposed in a payload is transmitted so as to arrive at the reception side before time Tf1.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 2, disposed in a payload is transmitted so as to arrive at the reception side before time Tf2.
- Figs. 14(a) to 14(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.
- Fig. 14(a) illustrates the TTML document corresponding to the event 1.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- Fig. 14(b) illustrates the TTML document corresponding to the event 2.
- Fig. 14(c) illustrates the TTML document corresponding to an event 3.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 14(d) illustrates the TTML document corresponding to an event 4.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 14(e) illustrates the TTML document corresponding to an event 5.
- FIG. 15 illustrates an exemplary TTML layer and an exemplary container layer in that case.
- T1 corresponds to the point in time past by a delay LTa from Tf1 in which PTS is reflected.
- T2 corresponds to the point in time past by a delay Ltb from Tf1 in which PTS is reflected.
- T4 corresponds to the point in time past by a delay LTc from Tf3 in which PTS is reflected.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- each PES packet is generated and is transmitted so as to arrive at the reception side at least at the start time of the corresponding sample.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 1, disposed in a payload is transmitted so as to arrive at the reception side before time Tf1.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 2, disposed in a payload is transmitted so as to arrive at the reception side before time Tf2.
- Figs. 16(a) to 16(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.
- Fig. 16(a) illustrates the TTML document corresponding to the event 1.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- Fig. 16(b) illustrates the TTML document corresponding to the event 2.
- Fig. 16(c) illustrates the TTML document corresponding to an event 3.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 16(d) illustrates the TTML document corresponding to an event 4.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 16(e) illustrates the TTML document corresponding to an event 5.
- timing at which a PES packet having a TTML document in which region display-end-time information is filled is generated and is transmitted in this manner is limited.
- TTML "end time" corresponding to a "latency" amount from PTS in a packet transporting a fixed sample is updated to give a clear indication.
- a relatively long sample period is provided, there is a possibility that "end time” is difficult to estimate.
- transmitting a TTML document in which region display-end-time information is filled has difficulty in giving the region display-end-time information to the reception side. For example, regarding the region A, even when the end time T3 gets known after the timing Tf1, a TTML document in which region display-end-time information is filled, cannot be transmitted before the time T3. A similar situation occurs regarding the regions B and C.
- the value of presentation time stamp (PTS) to be inserted into the header of the PES packet indicates the region display end time. Accordingly, in this case, the region display end time is given with PTS.
- Fig. 17 illustrates an exemplary TTML layer and an exemplary container layer in that case.
- T3 is previously known.
- the fact that the end time "end time” is T3 is initially written in a TTML document in addition to the fact that "start time” is T1, in relation to the region A.
- a region B is present in the TTML layer.
- the region B is displayed from timing T2 to timing T5, partially overlapping with the region A described above, temporally.
- a start time "start time” is T2
- an end time “end time” is T5.
- T5 is previously known.
- the fact that the end time "end time” is T5 is initially written in a TTML document in addition to the fact that "start time” is T2, in relation to the region B.
- a region C is present in the TTML layer.
- the region C is displayed from timing T4 to timing T6, partially overlapping with the region B described above, temporally.
- a start time "start time” is 42 and an end time “end time” is T6.
- T6 is previously known.
- the fact that the end time "end time” is T6 is initially written in a TTML document in addition to the fact that "start time” is T4, in relation to the region C.
- the period thereof varies in accordance with a variation period of subtitle display (display of each region). Therefore, as illustrated in Fig. 17 , from T1 to T2, from T2 to T3, from T3 to T4, from T4 to T5, and from T5 to T6 each are defined as one sample.
- an event is provided for every sample. Accordingly, an event 1 includes the sample of from T1 to T2, an event 2 includes the sample of from T2 to T3, an event 3 includes the sample of from T3 to T4, an event 4 includes the sample of from T4 to T5, and an event 5 includes the sample of from T5 to T6.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- each PES packet is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample.
- a PES packet having a segment packet of a TTML document including the region information regarding the event 1, disposed in a payload is transmitted so as to arrive at the reception side before the time T1.
- a PES packet having a segment packet of a TTML document including the region information regarding the event 2, disposed in a payload is transmitted so as to arrive at the reception side before the time T2.
- Figs. 18(a) to 18(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.
- Fig. 18(a) illustrates the TTML document corresponding to the event 1.
- Fig. 18(b) illustrates the TTML document corresponding to the event 2.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- Fig. 18(c) illustrates the TTML document corresponding to the event 3.
- Fig. 18(d) illustrates the TTML document corresponding to the event 4.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 18(e) illustrates the TTML document corresponding to the event 5.
- FIG. 19 illustrates a TTML layer and a container layer in that case according to an embodiment of the claimed invention.
- Information regarding a region A is present in the TTML layer.
- the region A is displayed from timing T1 to timing T3 at the end, but T3 cannot get previously known. Therefore, initially, only the fact that "start time” is T1 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region A.
- a region B is present in the TTML layer.
- the region B is displayed from timing T2 to timing T5 at the end, partially overlapping with the region A described above, temporally, but T5 cannot get previously known. Therefore, initially, only the fact that "start time” is T2 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region B.
- a region C is present in the TTML layer.
- the region C is displayed from timing T4 to timing T6 at the end, partially overlapping with the region B described above, temporally, but T6 cannot get previously known. Therefore, initially, only the fact that "start time” is T4 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region C.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- Each PES packet is generated and is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 1, disposed in a payload is transmitted so as to arrive at the reception side before time Tf1.
- a PES packet having a segment packet of a TTML document including the region information regarding an event 2, disposed in a payload is transmitted so as to arrive at the reception side before time Tf2.
- from T1 to T2 from T2 to T3p, from T3p to T3, from T3 to T4, from T4 to T5p, from T5p to T5, from T5 to T6p, and from T6p to T6 each are defined as one sample.
- the sample of from T3p to T3, the sample of from T4 to T5p, and the sample of from T6p to T6 each are a sample for an update to a TTML document in which region display-end-time information is filled.
- the event 1 includes the sample of from T1 to T2
- the event 2 includes the sample of from T2 to T3p
- an event 3 includes the sample of from T3p to T3
- an event 4 includes the sample of from T3 to T4
- an event 5 includes the sample of from T4 to T5p
- an event 6 includes the sample of from T5p to T5
- an event 7 includes the sample of from T5 to T6p
- an event 8 includes the sample of from T6p to T6.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- Figs. 20(a) to 20(h) schematically illustrate the contents of the respective TTML documents of the events according to an embodiment of the claimed invention.
- Fig. 20(a) illustrates the TTML document corresponding to the event 1.
- Fig. 20(b) illustrates the TTML document corresponding to the event 2.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- Fig. 20(c) illustrates the TTML document corresponding to the event 3.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- the TTML document is filled with region-A display-end-time information.
- Fig. 20(d) illustrates the TTML document corresponding to the event 4.
- Fig. 20(e) illustrates the TTML document corresponding to the event 5.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 20(f) illustrates the TTML document corresponding to the event 6.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- the TTML document is filled with region-B display-end-time information.
- Fig. 20(g) illustrates the TTML document corresponding to the event 7.
- Fig. 20(h) illustrates the TTML document corresponding to the event 8.
- the TTML document includes only the information regarding the region C.
- the TTML document is filled with region-C display-end-time information.
- FIG. 21 illustrates an exemplary TTML layer and an exemplary container layer in that case.
- Information regarding a region A is present in the TTML layer.
- the region A is displayed from timing T1 to timing T3 at the end, but T3 cannot get previously known. Therefore, initially, only the fact that "start time” is T1 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region A.
- a region B is present in the TTML layer.
- the region B is displayed from timing T2 to timing T5 at the end, partially overlapping with the region A described above, temporally, but T5 cannot get previously known. Therefore, initially, only the fact that "start time” is T2 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region B.
- a region C is present in the TTML layer.
- the region C is displayed from timing T4 to timing T6 at the end, partially overlapping with the region B described above, temporally, but T6 cannot get previously known. Therefore, initially, only the fact that "start time” is T4 is written and a blank is rendered for an end time "end time” in a TTML document, in relation to the region C.
- from T1 to T2, from T2 to T3, from T3 to T4, from T4 to T5, and from T5 to T6 each are defined as one sample.
- an event is provided for every sample. Accordingly, an event 1 includes the sample of from T1 to T2, an event 2 includes the sample of from T2 to T3, an event 3 includes the sample of from T3 to T4, an event 4 includes the sample of from T4 to T5, and an event 5 includes the sample of from T5 to T6.
- the value of presentation time stamp (PTS) to be inserted into the header of each PES packet is the start time of the corresponding sample (Sample start time).
- each PES packet is transmitted so as to arrive at the reception side at least at the start time of the corresponding sample.
- a PES packet having a segment packet of a TTML document including the region information regarding the event 1, disposed in a payload is transmitted so as to arrive at the reception side before the time T1.
- a PES packet having a segment packet of a TTML document including the region information regarding the event 2, disposed in a payload is transmitted so as to arrive at the reception side before the time T2.
- Figs. 22(a) to 22(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.
- Fig. 22(a) illustrates the TTML document corresponding to the event 1.
- Fig. 22(b) illustrates the TTML document corresponding to the event 2.
- the TTML document includes the information regarding the region B together with the information regarding the region A.
- Fig. 22(c) illustrates the TTML document corresponding to the event 3.
- Fig. 22(d) illustrates the TTML document corresponding to the event 4.
- the TTML document includes the information regarding the region C together with the information regarding the region B.
- Fig. 22(e) illustrates the TTML document corresponding to the event 5.
- the value of presentation time stamp (PTS) to be inserted into the header of the PES packet indicates the region display end time. Accordingly, in this case, the region display end time is given with PTS.
- the value of presentation time stamp (PTS) to be inserted into the header of the PES packet indicates the transmission time of the PES packet, for example. Then, in this case, when the count indicates the end, the region display end time is given by addition of a time offset added to the count, to PTS.
- PTS presentation time stamp
- Fig. 23 illustrates an exemplary TTML layer and an exemplary container layer in that case.
- the TS formatter 116 renders the video stream generated by the video encoder 112, the audio stream generated by the audio encoder 113, and the subtitle stream generated by the subtitle encoder 115 into transport packetization and performs multiplexing, to acquire the transport stream TS as a multiplexed stream.
- the operation of the stream generation unit 110 illustrated in Fig. 2 will be briefly described.
- the image data DV is supplied to the video encoder 112.
- the video encoder 112 performs coding to the image data DV, to generate a video stream (PES stream) including video PES packets each having coded image data in a payload.
- the video stream is supplied to the TS formatter 116.
- the audio data DA is supplied to the audio encoder 113.
- the audio encoder 113 performs coding to the audio data DA, to generate an audio stream (PES stream) including audio PES packets each having coded audio data.
- the audio stream is supplied to the TS formatter 116.
- the text data (character code) DT as the subtitle information is supplied to the text format conversion unit 114.
- the text format conversion unit 114 acquires subtitle text information having display timing information, in a predetermined format, here, TTML.
- the TTML is supplied to the subtitle encoder 115.
- the subtitle encoder 115 generates a segment packet including a document in TTML acquired by the text format conversion unit 114 (timed text subtitling segment: TT subtitling segment). Then, the subtitle encoder 115 generates a subtitle PES packet having the segment packet disposed in a payload.
- the subtitle encoder 115 generates a subtitle PES packet for every sample. Furthermore, because region display end time is initially unknown for live subtitle delivery, the subtitle encoder 115 cannot insert the region display end time into the TTML document included in a subtitle PES packet, and thus renders a blank.
- the subtitle encoder 115 generates a new subtitle PES packet in order to give or clearly indicate region display-end-time information to the reception side.
- the subtitle encoder 115 generates a segment packet including download information for downloading a font file designated in TTML font designation information, and disposes the segment packet in the payload of the PES packet.
- the subtitle encoder 115 generates the subtitle stream (PES stream) including each subtitle PES packet described above.
- the subtitle stream is supplied to the TS formatter 116.
- the TS formatter 116 renders the respective streams generated by the encoders into transport packetization and performs multiplexing, to generates the transport stream TS as a multiplexed stream.
- Fig. 24 illustrates an exemplary configuration of the transport stream TS.
- a video PES packet "Video PES” that is a PES packet of the video stream identified with PID1
- an audio PES packet “Audio PES” that is a PES packet of the audio stream identified with PID2
- a subtitle PES packet "Subtitle PES” that is a PES packet of the subtitle stream identified with PID3 is present in the exemplary configuration.
- the PES packets each include a PES header (PES header) and a PES payload (PES payload).
- PES header PES header
- PES payload PES payload
- the time stamp of DTS/PTS is inserted in the PES header and a video coded stream (coded image data) is inserted in the PES payload.
- the audio PES packet the time stamp of PTS is inserted in the PES header and an audio coded stream (coded audio data) is inserted in the PES payload.
- the time stamp of PTS is inserted in the PES header and a segment packet (timed text subtitling segment: TT subtitling segment) having, for example, a TTML document, a control command, or font download information, is inserted in the PES payload.
- TT subtitling segment timed text subtitling segment having, for example, a TTML document, a control command, or font download information
- the transport stream TS includes a program map table (PMT) as program specific information (PSI).
- the PSI is information describing to which program each elementary stream included in the transport stream TS belongs.
- a program descriptor in which information relating to the entirety of the programs is written, is present in the PMT. Furthermore, elementary-stream loops having respective pieces of information related to the elementary streams, are present in the PMT.
- a video elementary-stream loop (Video ES loop), an audio elementary-stream loop (Audio ES loop), and a subtitle elementary-stream loop (Subtitle ES loop) are present in the exemplary configuration.
- Information regarding, for example, a packet identifier (PID) is disposed in each loop.
- Fig. 25 illustrates an exemplary configuration of the television receiver 200.
- the television receiver 200 has a reception unit 201, a TS analysis unit (demultiplexer) 202, a video decoder 203, a video superimposition unit 204, a panel drive circuit 205, and a display panel 206.
- the television receiver 200 has an audio decoder 207, an audio output circuit 208, a speaker 209, and a subtitle decoder 210.
- the television receiver 200 has a CPU 221, a flash ROM 222, a DRAM 223, an internal bus 224, a remote-control reception unit 225, and a remote-control transmitter 226.
- the CPU 221 controls the operation of each unit of the television receiver 200.
- the flash ROM 222 stores control software and retains data.
- the DRAM 223 forms a work area for the CPU 221.
- the CPU 221 develops the software and the data read from the flash ROM 222 onto the DRAM 223 and boots the software so as to control each unit of the television receiver 200.
- the remote-control reception unit 225 receives a remote-control signal (remote-control code) transmitted from the remote-control transmitter 226 and supplies the remote-control signal to the CPU 221.
- the CPU 221 controls each unit of the television receiver 200 on the basis of the remote-control code.
- the CPU 221, the flash ROM 222, and the DRAM 223 are connected to the internal bus 224.
- the reception unit 201 receives the transport stream TS superimposed and sent on the broadcast wave from the broadcast output system 100.
- the transport stream TS includes, as described above, the video stream, the audio stream, and the subtitle stream.
- the TS analysis unit 202 extracts the respective streams of video, audio, and subtitles from the transport stream TS.
- the audio decoder 207 performs decoding processing to the audio stream extracted by the TS analysis unit 202, to acquire the audio data.
- the audio output circuit 208 performs necessary processing, such as D/A conversion and amplification, to the audio data, and supplies the audio data to the speaker 209.
- the video decoder 203 performs decoding processing to the video stream extracted by the TS analysis unit 202, to acquire the image data.
- the subtitle decoder 210 performs decoding processing to the subtitle stream extracted by the TS analysis unit 202, to acquire bitmap data of each region to be superimposed onto the image data.
- the subtitle decoder 210 extracts, from a segment packet including a TTML document, the segment packet and acquires the bitmap data of each region on the basis of the TTML document.
- the subtitle decoder 210 extracts, from a segment packet including a control command for giving region display-end-time information, the segment packet and sends the control command thereof to the CPU 221.
- the CPU 221 acquires the region display end time, on the basis of the control command.
- the subtitle decoder 210 extracts, from a segment packet including font download information, the segment packet and sends the font download information to the CPU 221.
- the CPU 221 acquires, from the font download segment, at least information for downloading a font file designated in TTML-document font designation information.
- the video superimposition unit 204 superimposes the bitmap data of each region acquired from the subtitle decoder 210, onto the image data acquired by the video decoder 203. In this case, the video superimposition unit 204 superimposes the bitmap data of each region by the period from the display start time to the display end time of each region.
- the display start time is acquired from the region display-start-time information included in each TTML document.
- the display end time is acquired from the region display-end-time information included in each TTML document (refer to Figs. 13 and 17 ).
- the display end time is acquired from the region display-end-time information included in each updated TTML document when the first method is adopted (refer to Fig. 19 ) and is acquired from each control command when the second method is adopted (refer to Figs. 15 , 21 , and 23 ).
- the panel drive circuit 205 drives the display panel 206, on the basis of the image data for display acquired by the video superimposition unit 204.
- the display panel 206 includes, for example, a liquid crystal display (LCD), an organic electroluminescence (EL) display, or the like.
- the reception unit 201 receives the transport stream TS superimposed and sent on the broadcast wave from the broadcast output system 100.
- the transport stream TS includes the video stream, the audio stream, and the subtitle stream.
- the transport stream TS is supplied to the TS analysis unit 202.
- the TS analysis unit 202 extracts the respective streams of video, audio, and subtitles from the transport stream TS.
- the video stream (PES stream) extracted by the TS analysis unit 202 is supplied to the video decoder 203.
- the video decoder 203 performs decoding processing to the video stream, to acquire the image data.
- the image data is supplied to the video superimposition unit 204.
- the subtitle stream (PES stream) extracted by the TS analysis unit 202 is supplied to the subtitle decoder 210.
- the subtitle decoder 210 performs decoding processing to the subtitle stream, to acquire bitmap data of each region to be superimposed onto the image data.
- the subtitle decoder 210 extracts, from a segment packet including a control command for giving region display-end-time information, the segment packet and sends the control command to the CPU 221.
- the CPU 221 acquires the region display end time, on the basis of the control command.
- the subtitle decoder 210 extracts, from a segment packet including font download information, the segment packet and sends the font download information to the CPU 221.
- the CPU 221 acquires, from the font download segment, at least information for downloading a font file designated in TTML-document font designation information. The information is used as necessary in order to acquire the font file.
- the bitmap data of each region output from the subtitle decoder 210 is supplied to the video superimposition unit 204.
- the video superimposition unit 204 superimposes the bitmap data of each region output from the subtitle decoder 210, onto the image data acquired by the video decoder 203. In this case, the bitmap data of each region is superimposed by the period from the display start time to the display end time of each region.
- the display start time is acquired from the region display-start-time information included in each TTML document. Furthermore, for the pre-authored subtitle delivery, the display end time is acquired from the region display-end-time information included in each TTML document (refer to Figs. 13 and 17 ). Furthermore, for the live subtitle delivery, the display end time is acquired from the region display-end-time information included in each updated TTML document when the first method is adopted (refer to Fig. 19 ) and is acquired from each control command when the second method is adopted (refer to Figs. 15 , 21 , and 23 ).
- the image data for display acquired by the video superimposition unit 204 is supplied to the panel drive circuit 205.
- the panel drive circuit 205 drives the display panel 206, on the basis of the image data for display. This arrangement allows the display panel 206 to display the image on which each region is superimposed.
- the broadcast output system 100 in the transmission and reception system 10 illustrated in Fig. 1 generates a subtitle PES packet having a segment packet having a TTML document, disposed in a payload, for every sample, the period of each sample is varied, for example, in accordance with a variation period of subtitle display. Therefore, the transmission frequency of a subtitle PES packet can be inhibited, so that transport efficiency can improve.
- the broadcast output system 100 in the transmission and reception system 10 illustrated in Fig. 1 sends the region display-end-time information with the TTML layer (first method) or the container layer (second method) when the region display end time gets known. Therefore, the region display-end-time information can be given to the reception side, so that the region display end processing can be performed properly on the reception side.
- the broadcast output system 100 in the transmission and reception system 10 illustrated in Fig. 1 newly generates and transmits, for example, a PES packet having a segment packet having a control command for giving region display-end-time information, inserted in a payload, when the region display end time gets known (second method). Therefore, even in a case where the period of each sample is fixed, the region display-end-time information can be favorably given to the reception side, so that the region display end processing can be performed properly on the reception side.
- TTML is used as the subtitle text information having the display timing information, in the predetermined format
- the present technology is not limited to this, and thus it is considered that other timed text information having information similar to that of TTML is used.
- a TTML derived format may be used.
- the transmission and reception system 10 including the broadcast output system 100 and the television receiver 200 has been given in the embodiment described above, but the configuration of a transmission and reception system to be acquired by the application of the present technology is not limited to this.
- the part of the television receiver 200 may include a set-top box and a monitor connected through a digital interface, such as high-definition multimedia interface (HDMI).
- HDMI high-definition multimedia interface
- the container multiplexed stream
- the container is the MPEG-2 transport stream
- the present technology can be applied similarly even in a case where the multiplexed stream is, for example, an MMT stream or a DASH/ISOBMFF stream.
- varying the period of each sample in accordance with a variation period of subtitle display inhibits the transmission frequency of a subtitle PES packet including a TTML document, so that transport efficiency can improve (refer to Fig. 17 ). Furthermore, according to a main feature of the present technology, for live subtitle delivery, when region display end time gets known, transmission of region display-end-time information with a TTML layer or a container layer, allows region display end processing to be performed properly on the reception side (refer to Figs. 15 , 19 , 21 , and 23 ).
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Human Computer Interaction (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Description
- The present technology relates to a transmission device, a transmission method, a reception device, and a reception method, and particularly relates to, for example, a transmission device that transmits a document of subtitle text information.
- Conventionally, for example, management of transmitting subtitle information in bitmap data, has been performed in broadcasting, such as digital video broadcasting (DVB). In recent years, transmitting subtitle information in a character code of text, namely, on a text basis, has been proposed. In this case, font development corresponding to resolution is performed on the reception side.
- Furthermore, in a case where subtitle information is transmitted on a text basis, causing text information to have timing information has been proposed. As the text information, for example, Timed Text Markup Language (TTML) has been advocated by the World Wide Web Consortium (W3C) (refer to Patent Document 1).
-
Patent Document 2 further proposes a streaming device for streaming data in the MP4 format, including subtitle data stored in fragment boxes. The subtitle data contains TTML document instances which are identified by instance classification distinguishing information. -
Patent Document 3 proposes a method for adapting the display time of subtitle data taking into account the time necessary for processing the encoded subtitle data, which is dependent on the size of the encoded data, and the subtitle presentation time. -
Patent Document 4 proposes a reproduction that enables the correct display of subtitles after conversion from TS to PS, even during trick play modes. The beginning of a subtitle, divided into subtitle packets is the time stamp aligned with the beginning of an I-picture, divided into I-picture packets. -
Patent Document 5 proposes a content reproduction device that reproduces content items containing audio, video and subtitles organized in a specified data structure. Each content item is split into segment data items which are described in a XML file. Each segment data item corresponds to a plurality of segments of playback time periods. To each playback time period corresponds a group of subtitle data items that are reproduced within each of the time periods. -
Patent Document 6 proposes a reproduction device that controls the display of timed subtitles. A subtitle encoder generates a subtitle stream having the text information and the display timing information. The subtitles are in the TTML format. The display start time is indicated in the PTS of a PES packet and the text information and display timing is indicated in the PES payload. -
Patent Document 7 proposes TTML subtitle transmission management per regions and samples with start and end time. -
- Patent Document 1:
Japanese Patent Application Laid-Open No. 2012-169885 - Patent Document 2:
US 2009/303382 A1 - Patent Document 3:
WO 2016/098790 A1 - Patent Document 4:
JP 2008 193203 A - Patent Document 5:
US 2013/185760 A1 - Patent Document 6:
WO 2006/090908 A1 - Patent Document 7:
JP 2016 110645 A - An object of the present technology is to improve the transport efficiency of a document of subtitle text information. Alternatively, an object of the present technology is to allow region display end processing to be performed properly on the reception side in live delivery.
- According to a concept of the present technology, a transmission device for transmitting a plurality of subtitles for different regions comprises:
- a packet generation unit configured to generate for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, and wherein display of the subtitle is to be ended in the sample related to the second PES packet; and
- a transmission unit configured to transmit the PES packets as a subtitle stream,
wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles. - According to a concept of the present technology, a reception device for receiving a plurality of subtitles for different regions comprises:
- a reception unit configured to receive for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, and wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles; and
- a control unit configured to control subtitle display based on the TTML documents of subtitle text information disposed in the PES packets, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, wherein display of the subtitle is to be ended in the sample related to the second PES packet, and wherein the control unit is configured to end subtitle display based on the display timing information disposed in the second PES packet.
- According to a concept of the present technology, a transmission method for transmitting a plurality of subtitles for different regions comprises:
- generating for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, and wherein display of the subtitle is to be ended in the sample related to the second PES packet; and
- transmitting the PES packets as a subtitle stream, wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles.
- According to a concept of the present technology, a reception method for receiving a plurality of subtitles for different regions comprises:
- receiving for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, and wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles; and
- controlling subtitle display based on the TTML documents of subtitle text information disposed in the PES packets, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, wherein display of the subtitle is to be ended in the sample related to the second PES packet, and wherein the subtitle display is ended based on the display timing information disposed in the second PES packet.
- According to the present technology, the transport efficiency of a document of subtitle text information can improve. Alternatively, according to the present technology, region display end processing can be performed properly on the reception side in live delivery. Note that the effects described in the present specification are, but are not limited to, just exemplifications, and thus additional effects may be provided.
-
-
Fig. 1 is a block diagram of an exemplary configuration of a transmission and reception system as an embodiment. -
Fig. 2 is a block diagram of an exemplary configuration of a stream generation unit of a broadcast output system. -
Fig. 3 is a table for describing a TTML structure. -
Fig. 4 illustrates tables of an exemplary structure of a subtitle PES packet and an exemplary structure of "PES_data_byte_field()". -
Fig. 5 illustrates tables of exemplary definitions of subtitle segment types. -
Fig. 6 is a diagram of an exemplary structure of a subtitle stream (PES stream) including subtitle PES packets. -
Fig. 7 is a table of an exemplary structure of a TTML-document segment. -
Fig. 8 is a table of the descriptions of main information in the exemplary structure of the TTML-document segment. -
Fig. 9 is a table of an exemplary structure of a control-command segment. -
Fig. 10 is a table of the descriptions of main information in the exemplary structure of the control-command segment. -
Fig. 11 is a table of an exemplary structure of a font-download-information segment. -
Fig. 12 is a table of the descriptions of main information in the exemplary structure of the font-download-information segment. -
Fig. 13 is a chart not according to the claimed invention of an exemplary TTML layer and an exemplary container layer in pre-authored subtitle delivery with fixed samples. -
Fig. 14 illustrates schematic tables not according to the claimed invention of the contents of respective TTML documents corresponding to events. -
Fig. 15 is a chart not according to the claimed invention of an exemplary TTML layer and an exemplary container layer in live subtitle delivery with fixed samples. -
Fig. 16 illustrates schematic tables not according to the claimed invention of the contents of respective TTML documents corresponding to events. -
Fig. 17 is a chart not according to the claimed invention of an exemplary TTML layer and an exemplary container layer in pre-authored subtitle delivery with variable samples. -
Fig. 18 illustrates schematic tables not according to the claimed invention of the contents of respective TTML documents corresponding to events. -
Fig. 19 is a chart according to an embodiment of the claimed invention of a TTML layer and an exemplary container layer in live subtitle delivery with variable samples. -
Fig. 20 illustrates schematic tables according to an embodiment of the claimed invention of the contents of respective TTML documents corresponding to events. -
Fig. 21 is a chart not according to the claimed invention of an exemplary TTML layer and an exemplary container layer in live subtitle delivery with variable samples. -
Fig. 22 illustrates schematic tables not according to the claimed invention of the contents of respective TTML documents corresponding to events. -
Fig. 23 is a chart not according to the claimed invention of an exemplary TTML layer and an exemplary container layer in live subtitle delivery with variable samples. -
Fig. 24 is a diagram of an exemplary configuration of a transport stream TS. -
Fig. 25 is a block diagram of an exemplary configuration of a television receiver. -
Fig. 26 is a chart of another exemplary TTML layer and another exemplary container layer in the live subtitle delivery with the fixed samples. - A mode for carrying out the invention (hereinafter, referred to as an "embodiment") will be described below. Note that the descriptions will be given in the following order.
- 1. Embodiment
- 2. Modification
-
Fig. 1 illustrates an exemplary configuration of a transmission andreception system 10 as an embodiment. The transmission andreception system 10 includes abroadcast output system 100 and atelevision receiver 200. - The
broadcast output system 100 superimposes an MPEG-2 transport stream (hereinafter, simply referred to as a "transport stream") TS as a multiplexed stream, on a broadcast wave, to transmit the transport stream TS. - The transport stream TS has a subtitle stream together with a video stream and an audio stream. The video stream includes video PES packets each having coded image data disposed in a payload. The audio stream includes audio PES packets each having coded audio data disposed in a payload.
- The subtitle stream includes subtitle PES packets each having a segment packet disposed in a payload, according to the present embodiment, the segment packet having a document in TTML (TTML document) of subtitle text information having display timing information. Such a subtitle PES packet is generated for every sample.
- Here, each sample is, as conventionally known, a fixed sample in which the period thereof is fixed, or each sample is, as newly proposed, a variable sample in which the period thereof is variable in accordance with a variation period of subtitle display. Note that, in the present embodiment, subtitle display corresponding to each sample, is individually referred to as an event.
- For the variable samples, the transmission frequency of subtitle PES packets can be inhibited, so that transport efficiency can improve.
- For pre-authored subtitle delivery, because region display end time is known, the region display end time can be inserted into a TTML document. However, for live subtitle delivery, because region display end time is unknown at the beginning of a display start, it is impossible that the region display end time is inserted into a TTML document.
- In the present embodiment, when the region display end time becomes known, region display-end-time information is given to the reception side with a first method or a second method. This arrangement allows region display end processing on the reception side to be performed properly on the reception side in live delivery.
- In the first method, a PES packet having a segment packet having a TTML document in which region end-time information is filled, inserted in a payload, is newly generated and is transmitted. The first method is limited to, for the fixed samples, a sample unit thereof. In the second method, a PES packet having a segment packet having a control command for giving region display-end-time information, inserted in a payload, is newly generated and is transmitted.
- The
television receiver 200 receives the transport stream TS sent by the broadcast wave from thebroadcast output system 100. The transport stream TS has, as described above, the subtitle stream together with the video stream and the audio stream. The subtitle stream includes, as described above, the subtitle PES packets each having a segment packet having a TTML document, disposed in a payload. - The
television receiver 200 receives, in accordance with each sample, a subtitle PES packet for the sample before the beginning of the period of the sample. Thetelevision receiver 200 controls region display on the basis of the TTML document included in the subtitle PES packet. In a case where each sample is a variable sample, thetelevision receiver 200 has the reception frequency of PES packets lower than that in a case where each sample is a fixed sample, so that a processing load is reduced. - For the live subtitle delivery, with the first method adopted, the
television receiver 200 controls an end of the region display, on the basis of the TTML document in which region display-end-time information is filled, included in a subtitle PES packet received after the region display starts. Furthermore, in a similar case, with the second method adopted, thetelevision receiver 200 acquires the region display-end-time information and controls an end of the region display, on the basis of the control command for giving region display-end-time information, included in a subtitle PES packet received after the region display starts. This arrangement allows thetelevision receiver 200 to perform the region display end processing properly in the live delivery. -
Fig. 2 illustrates an exemplary configuration of astream generation unit 110 of thebroadcast output system 100. Thestream generation unit 110 has acontrol unit 111, avideo encoder 112, anaudio encoder 113, a textformat conversion unit 114, asubtitle encoder 115, and a TS formatter (multiplexer) 116. - The
control unit 111 includes, for example, a central processing unit (CPU), and controls the operation of each unit of thestream generation unit 110. Thevideo encoder 112 inputs image data DV and performs coding to the image data DV, to generate a video stream (PES stream) including video PES packets each having coded image data in a payload. Theaudio encoder 113 inputs audio data DA and performs coding to the audio data DA, to generate an audio stream (PES stream) including audio PES packets each having coded audio data. - The text
format conversion unit 114 inputs text data (character code) DT as subtitle information, and acquires subtitle text information having display timing information, in a predetermined format. For example, TTML or a TTML derived format is considered as the text information, but TTML is adopted in the present embodiment. -
Fig. 3 illustrates an exemplary structure of a document (file) in TTML. The TTML is described on an XML basis. Head and body are present in the TTML. Then, for example, metadata, styling, and layout are present in the head. - The metadata includes, for example, title information and copyright information. For example, a region identifier, and color information, font information, and position information in association with the region identifier are disposed in the styling. For example, background color information and alignment information are disposed in association with the region identifier in the layout. For example, subtitle text data to be displayed and timing information regarding a region display start and a region display end are disposed in association with the region identifier in the body.
- Referring back to
Fig. 2 , thesubtitle encoder 115 generates a segment packet including a document in TTML acquired by the text format conversion unit 114 (timed text subtitling segment: TT subtitling segment). Then, thesubtitle encoder 115 generates a subtitle stream (PES stream) including a subtitle PES packet having the segment packet disposed in a payload. - The
subtitle encoder 115 generates a subtitle PES packet for every sample. Furthermore, because the region display end time is unknown initially for the live subtitle delivery, thesubtitle encoder 115 cannot insert the region display end time into the TTML document included in a subtitle PES packet, and thus renders a blank. - Thereafter, when the region display end time becomes known, the
subtitle encoder 115 generates a new subtitle PES packet in order to give the region display-end-time information to the reception side. With the first method adopted, thesubtitle encoder 115 generates a segment packet including a TTML document in which the region end-time information is filled, and newly generates a subtitle PES packet having the segment packet disposed in a payload. With the second method adopted, thesubtitle encoder 115 generates a segment packet including a control command for giving the region display-end-time information, and newly generates a subtitle PES packet having the segment packet disposed in a payload. - Furthermore, the
subtitle encoder 115 generates a segment packet including download information for downloading a font file designated in TTML font designation information, and disposes the segment packet in the payload of the PES packet. -
Fig. 4(a) illustrates an exemplary structure of a subtitle PES packet (PES_packet). A fixed pattern of "0x000001" is disposed in the 24-bit field of "PES_startcode_prefix". The 8-bit field of "stream_id" indicates a stream identifier. For the subtitle PES packet, "10111101" that is a value indicating "private stream1", is provided. The 16-bit field of "PES_packet_length" indicates a byte length thereafter as the length (size) of the PES packet. - The field of "Optional_PES_header()" is present after "PES_packet_length". For example, the time stamp of PTS or DTS is disposed in the field. After the field, the field of "PES_packet_data_byte" is present. The field corresponds to a PES payload. "PES_data_byte_field()" for containing data is disposed in the field.
-
Fig. 4(b) illustrates an exemplary structure of "PES_data_byte_field()". The 8-bit field of "data_identifier" indicates an identifier that identifies the type of data in a container portion. Because conventional subtitles (in bitmap) are indicated with "0x20", identification of text can be made with a new value, for example, "0x21". - The 8-bit field of "subtitle_stream_id" indicates an identifier that identifies the type of a subtitle stream. A new value, for example, "0x01" is adopted for the subtitle stream that transports text information, so that a distinction can be made from the subtitle stream "0x00" that transports the conventional bitmap.
- After the field of "subtitle_stream_id", the field of "Subtitling_segments()" is present, following a pattern of "00001111". A segment packet is disposed in the field. After the field, the 8-bit field of "end_of_PES_data_field_marker" is present. The field is a marker indicating the end of the PES packet.
-
Fig. 5(a) illustrates exemplary definitions of subtitle segment types (subtitle segment_type). "0xA0" indicates a segment packet that transports a TTML document. "0xA1" indicates a segment packet that transports a compressed TTML document. "0xA2" indicates a segment packet that transports a control command. "0xA3" indicates a segment packet that transports font download information. As illustrated inFig. 5(b) , segments having different subtitle segment types in value, can be disposed in one PES payload. -
Fig. 6 illustrates an exemplary structure of a subtitle stream (PES stream) including subtitle PES packets. In the example, as the subtitle PES packets, for example, a subtitle PES packet including only a segment packet that transports a TTML document, a subtitle PES packet including only a segment packet that transports a control command, and furthermore a subtitle PES packet including both thereof are present. - An exemplary structure of a segment packet will be described.
Fig. 7 illustrates an exemplary structure of a TTML-document segment (TTL_document_segment()).Fig. 8 illustrates the descriptions of main information in the exemplary structure. The 8-bit field of "sync_byte" is a unique word indicating a segment start. The 8-bit field of "segment_type" is a segment type, and indicates the type of the segment. For the TTML-document segment, "segment_type" is "0xA0" (refer toFig. 5(a) ). - The 8-bit field of "number_of_regions" indicates the number of regions included in a TTML document. The 8-bit field of "region_id" is repeatedly present by the number of regions. The field of "region_id" indicates region identification information (id). The region identification information (id) corresponds to the region identification information (id) in the TTML document.
- The 16-bit field of "latency" indicates the delay in time from PTS until caption display (subtitle display) transported in byte data of "segment_data_field()" starts. The value is, for example, ten times the amount of actual time (by the second). The 16-bit field of "document_version_number" indicates information update of the TTML document. In a case where an update is made, the value is incremented by one.
- The 16-bit field of "segment_length" indicates a byte length thereafter as the length (size) of the TTML-document segment. After "segment_length", the field of "segment_data_field()" is present. The field corresponds to the payload of the segment (segment packet), and the entire TTML document is disposed in the field.
-
Fig. 9 illustrates an exemplary structure of a control-command segment (control_command_segment()).Fig. 10 illustrates the descriptions of main information in the exemplary structure. The 8-bit field of "sync_byte" is a unique word indicating a segment start. The 8-bit field of "segment_type" is a segment type, and indicates the type of the segment. For the control-command segment, "segment_type" is "0xA2" (refer toFig. 5(a) ). - The 8-bit field of "number_of_regions" indicates the number of regions. The 8-bit field of "region_id" and the 8-bit field of "command" are repeatedly present by the number of regions. The field of "region_id" indicates region identification information (id). The field of "latency" expresses the timing of command execution as a delay offset from PTS. The value is indicated with a tenfold value, and thus "1" and "16" express an execution delay of 0.1 seconds and an execution delay of 1.6 seconds, respectively. The field of "command" indicates a command. For example, "0x01" indicates an end of region display. "0x02" indicates that information relating to the region display is to be deleted from a buffer. "0x03" indicates that a region display state is to be indicated with a count (time count) at previously defined time intervals.
- When "command" is "0x03", the 16-bit field of "time count" is present. The field of "time count" indicates a time count (count value at fixed intervals). The time count is an ascended value from "0x0001", and a special value "0xFFFF" indicates a display end. The 8-bit field of "time_offset" indicates a time offset (time_offset) from the time stamp (timestamp) of a display end command.
-
Fig. 11 illustrates an exemplary structure of a font-download-information segment (font_download_information_segment()).Fig. 12 illustrates the descriptions of main information in the exemplary structure. The 8-bit field of "sync_byte" is a unique word indicating a segment start. The 8-bit field of "segment_type" is a segment type, and indicates the type of the segment. For the control-command segment, "segment_type" is "0xA3" (refer toFig. 5(a) ). - The 16-bit field of "original_network_id" indicates identification information regarding a network through which download data is to be transported. The 16-bit field of "transport_stream_id" indicates identification information regarding an individual transport stream. The 16-bit field of "service_id" indicates identification information regarding a service in which downloading is to be performed. For a common object to be downloaded between delivery media, there is a possibility that, instead of in the host transport stream, a font file is sent in another transport stream. In that case, information regarding "original_network_id", "transport_stream_id", and "service_id" can be designated as information for specifying a private section in a reference destination.
- The 8-bit field of "font_file_id" indicates an identification number assigned to the font file. The 24-bit field of "ISO_639_language_code" indicates a code having three characters allowing a language to be identified. For example, "jpn" indicates Japanese, and "eng" indicates English. The 8-bit field of "font_group_id" indicates identification information regarding a font group, and corresponds to a TTML generic family. The 8-bit field of "font_name_id" indicates an individual font name.
- The 8-bit field of "url_type" indicates the type of a server. For example, "0x01" indicates a font server (uncompressed URL) and "0x02" indicates a general server (uncompressed URL). "0x11" indicates a font server (compressed URL) and "0x12" indicates a general server (compressed URL). The 8-bit field of "url_string_length" indicates, in byte length, the length (size) of a character code portion indicating the character string of a URL, thereafter. The character code is disposed in the field of "char".
- Here, a TTML layer and a container layer will be further described. First, subtitle delivery produced at expense in time, namely, pre-authored subtitle delivery will be described with fixed samples.
Fig. 13 illustrates an exemplary TTML layer and an exemplary container layer in that case. - Information regarding a region A is present in the TTML layer. The region A is displayed from timing T1 to timing T3. In other words, regarding the region A, a start time "start time" corresponding to the point in time past by a delay LTa from Tf1 in which PTS is reflected, is T1 and an end time "end time" is T3. Because of the pre-authored subtitle delivery here, T3 is previously known. Thus, the fact that the end time "end time" is T3 is initially written in a TTML document in addition to the fact that "start time" is T1, in relation to the region A.
- Furthermore, information regarding a region B is present in the TTML layer. The region B is displayed from timing T2 to timing T5, partially overlapping with the region A described above, temporally. In other words, regarding the region B, a start time "start time" corresponding to the point in time past by a delay LTb from Tf1 in which PTS is reflected, is T2 and an end time "end time" is T5. Because of the pre-authored subtitle delivery here, T5 is previously known. Thus, the fact that the end time "end time" is T5 is initially written in a TTML document in addition to the fact that "start time" is T2, in relation to the region B.
- Moreover, information regarding a region C is present in the TTML layer. The region C is displayed from timing T4 to timing T6, partially overlapping with the region B described above, temporally. In other words, regarding the region C, a start time "start time" corresponding to the point in time past by a delay LTc from Tf3 in which PTS is reflected, is T4 and an end time "end time" is T6. Because of the pre-authored subtitle delivery here, T6 is previously known. Thus, the fact that the end time "end time" is T6 is initially written in a TTML document in addition to the fact that "start time" is T4, in relation to the region C.
- Here, an event is provided for every sample. In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
- Then, each PES packet is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample. For example, a PES packet having a segment packet of a TTML document including the region information regarding an
event 1, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf1. Furthermore, for example, a PES packet having a segment packet of a TTML document including the region information regarding anevent 2, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf2. A similar manner is made for the others. -
Figs. 14(a) to 14(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.Fig. 14(a) illustrates the TTML document corresponding to theevent 1. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end="T3"", and the start time and the end time of the region B are rendered in "begin="T2"end="T5"". -
Fig. 14(b) illustrates the TTML document corresponding to theevent 2. The TTML document includes only the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"". -
Fig. 14(c) illustrates the TTML document corresponding to anevent 3. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"", and the start time and the end time of the region C are rendered in "begin="T4"end="T6"". -
Fig. 14(d) illustrates the TTML document corresponding to anevent 4. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"", and the start time and the end time of the region C are rendered in "begin="T4"end="T6"". -
Fig. 14(e) illustrates the TTML document corresponding to anevent 5. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region C are rendered in "begin="T4"end="T6"". - Next, live subtitle delivery with fixed samples will be described.
Fig. 15 illustrates an exemplary TTML layer and an exemplary container layer in that case. - Information regarding a region A is present in the TTML layer. The region A is displayed from timing T1 to timing T3 at the end, but T3 cannot get previously known. Therefore, initially, only the fact that "start time" is T1 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region A. Note that T1 corresponds to the point in time past by a delay LTa from Tf1 in which PTS is reflected.
- Furthermore, information regarding a region B is present in the TTML layer. The region B is displayed from timing T2 to timing T5 at the end, partially overlapping with the region A described above, temporally, but T5 cannot get previously known. Therefore, initially, only the fact that "start time" is T2 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region B. Note that T2 corresponds to the point in time past by a delay Ltb from Tf1 in which PTS is reflected.
- Moreover, information regarding a region C is present in the TTML layer. The region C is displayed from timing T4 to timing T6 at the end, partially overlapping with the region B described above, temporally, but T6 cannot get previously known. Therefore, initially, only the fact that "start time" is T4 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region C. Note that T4 corresponds to the point in time past by a delay LTc from Tf3 in which PTS is reflected.
- Here, an event is provided for every sample. In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
- Then, each PES packet is generated and is transmitted so as to arrive at the reception side at least at the start time of the corresponding sample. For example, a PES packet having a segment packet of a TTML document including the region information regarding an
event 1, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf1. Furthermore, for example, a PES packet having a segment packet of a TTML document including the region information regarding anevent 2, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf2. A similar manner is made for the others. -
Figs. 16(a) to 16(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.Fig. 16(a) illustrates the TTML document corresponding to theevent 1. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end=" "", and the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 16(b) illustrates the TTML document corresponding to theevent 2. The TTML document includes only the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 16(c) illustrates the TTML document corresponding to anevent 3. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "", and the start time and the end time of the region C are rendered in "begin="T4"end=" "". -
Fig. 16(d) illustrates the TTML document corresponding to anevent 4. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "", and the start time and the end time of the region C are rendered in "begin="T4"end=" "". -
Fig. 16(e) illustrates the TTML document corresponding to anevent 5. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region C are rendered in "begin="T4"end=" "". - Here, in a case where the display end times of the regions A, B, and C get known, it is considered that a PES packet having a segment packet of a TTML document in which region display-end-time information is filled, disposed in a payload, is generated and is transmitted to give (clearly indicate) the region display-end-time information to the reception side. However, because a sample period is fixed, timing at which a PES packet having a TTML document in which region display-end-time information is filled is generated and is transmitted in this manner, is limited. Thus, for example, TTML "end time" corresponding to a "latency" amount from PTS in a packet transporting a fixed sample, is updated to give a clear indication. In a case where a relatively long sample period is provided, there is a possibility that "end time" is difficult to estimate.
- In some cases, transmitting a TTML document in which region display-end-time information is filled, in this manner, has difficulty in giving the region display-end-time information to the reception side. For example, regarding the region A, even when the end time T3 gets known after the timing Tf1, a TTML document in which region display-end-time information is filled, cannot be transmitted before the time T3. A similar situation occurs regarding the regions B and C.
- Thus, in a case where the end times of the regions A, B, and C get known in the example, a PES packet having a segment packet of a control command for giving region display-end-time information (command=0x01) (subtitle segment_type = 0xA2), disposed in a payload, is generated and is transmitted so as to arrive at the reception side before each of the times. In this case, the value of presentation time stamp (PTS) to be inserted into the header of the PES packet, indicates the region display end time. Accordingly, in this case, the region display end time is given with PTS.
- Next, pre-authored subtitle delivery with variable samples will be described.
Fig. 17 illustrates an exemplary TTML layer and an exemplary container layer in that case. - Information regarding a region A is present in the TTML layer. The region A is displayed from timing T1 to timing T3. In other words, regarding the region A, a start time "start time" is T1, and an end time "end time" is T3. Because of the pre-authored subtitle delivery here, T3 is previously known. Thus, the fact that the end time "end time" is T3 is initially written in a TTML document in addition to the fact that "start time" is T1, in relation to the region A.
- Furthermore, information regarding a region B is present in the TTML layer. The region B is displayed from timing T2 to timing T5, partially overlapping with the region A described above, temporally. In other words, regarding the region B, a start time "start time" is T2 and an end time "end time" is T5. Because of the pre-authored subtitle delivery here, T5 is previously known. Thus, the fact that the end time "end time" is T5 is initially written in a TTML document in addition to the fact that "start time" is T2, in relation to the region B.
- Moreover, information regarding a region C is present in the TTML layer. The region C is displayed from timing T4 to timing T6, partially overlapping with the region B described above, temporally. In other words, regarding the region C, a start time "start time" is 42 and an end time "end time" is T6. Because of the pre-authored subtitle delivery here, T6 is previously known. Thus, the fact that the end time "end time" is T6 is initially written in a TTML document in addition to the fact that "start time" is T4, in relation to the region C.
- For each variable sample, as described above, the period thereof varies in accordance with a variation period of subtitle display (display of each region). Therefore, as illustrated in
Fig. 17 , from T1 to T2, from T2 to T3, from T3 to T4, from T4 to T5, and from T5 to T6 each are defined as one sample. - Here, an event is provided for every sample. Accordingly, an
event 1 includes the sample of from T1 to T2, anevent 2 includes the sample of from T2 to T3, anevent 3 includes the sample of from T3 to T4, anevent 4 includes the sample of from T4 to T5, and anevent 5 includes the sample of from T5 to T6. - In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
- Then, each PES packet is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample. For example, a PES packet having a segment packet of a TTML document including the region information regarding the
event 1, disposed in a payload, is transmitted so as to arrive at the reception side before the time T1. Furthermore, for example, a PES packet having a segment packet of a TTML document including the region information regarding theevent 2, disposed in a payload, is transmitted so as to arrive at the reception side before the time T2. A similar manner is made for the others. -
Figs. 18(a) to 18(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.Fig. 18(a) illustrates the TTML document corresponding to theevent 1. The TTML document includes only the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end="T3"". -
Fig. 18(b) illustrates the TTML document corresponding to theevent 2. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end="T3"", and the start time and the end time of the region B are rendered in "begin="T2"end="T5"". -
Fig. 18(c) illustrates the TTML document corresponding to theevent 3. The TTML document includes only the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"". -
Fig. 18(d) illustrates the TTML document corresponding to theevent 4. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"", and the start time and the end time of the region C are rendered in "begin="T4"end="T6"". -
Fig. 18(e) illustrates the TTML document corresponding to theevent 5. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region C are rendered in "begin="T4"end="T6"". - Next, live subtitle delivery with variable samples will be described.
Fig. 19 illustrates a TTML layer and a container layer in that case according to an embodiment of the claimed invention. - Information regarding a region A is present in the TTML layer. The region A is displayed from timing T1 to timing T3 at the end, but T3 cannot get previously known. Therefore, initially, only the fact that "start time" is T1 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region A.
- Furthermore, information regarding a region B is present in the TTML layer. The region B is displayed from timing T2 to timing T5 at the end, partially overlapping with the region A described above, temporally, but T5 cannot get previously known. Therefore, initially, only the fact that "start time" is T2 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region B.
- Moreover, information regarding a region C is present in the TTML layer. The region C is displayed from timing T4 to timing T6 at the end, partially overlapping with the region B described above, temporally, but T6 cannot get previously known. Therefore, initially, only the fact that "start time" is T4 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region C.
- Here, an event is provided for every sample. In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
- Each PES packet is generated and is transmitted so as to arrive at the reception side at the latest before the start time of the corresponding sample. For example, a PES packet having a segment packet of a TTML document including the region information regarding an
event 1, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf1. Furthermore, for example, a PES packet having a segment packet of a TTML document including the region information regarding anevent 2, disposed in a payload, is transmitted so as to arrive at the reception side before time Tf2. A similar manner is made for the others. - In a case where the display end times of the regions A, B, and C get known with the variable samples in the example, a PES packet having a segment packet of a TTML document in which region display-end-time information is filled, disposed in a payload, is generated and is transmitted. That is, in a case where the display end times of the regions A, B, and C get known, the TTML document is updated.
- In this case, as illustrated in
Fig. 19 , from T1 to T2, from T2 to T3p, from T3p to T3, from T3 to T4, from T4 to T5p, from T5p to T5, from T5 to T6p, and from T6p to T6 each are defined as one sample. Here, the sample of from T3p to T3, the sample of from T4 to T5p, and the sample of from T6p to T6 each are a sample for an update to a TTML document in which region display-end-time information is filled. - Here, an event is provided for every sample. Accordingly, the
event 1 includes the sample of from T1 to T2, theevent 2 includes the sample of from T2 to T3p, anevent 3 includes the sample of from T3p to T3, anevent 4 includes the sample of from T3 to T4, anevent 5 includes the sample of from T4 to T5p, anevent 6 includes the sample of from T5p to T5, anevent 7 includes the sample of from T5 to T6p, and anevent 8 includes the sample of from T6p to T6. - In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
-
Figs. 20(a) to 20(h) schematically illustrate the contents of the respective TTML documents of the events according to an embodiment of the claimed invention.Fig. 20(a) illustrates the TTML document corresponding to theevent 1. The TTML document includes only the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end=" "". -
Fig. 20(b) illustrates the TTML document corresponding to theevent 2. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end=" "", and the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 20(c) illustrates the TTML document corresponding to theevent 3. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end="T3"", and the start time and the end time of the region B are rendered in "begin="T2"end=" "". In this case, as indicated in a broken-line rectangular box, the TTML document is filled with region-A display-end-time information. -
Fig. 20(d) illustrates the TTML document corresponding to theevent 4. The TTML document includes only the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 20(e) illustrates the TTML document corresponding to theevent 5. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "", and the start time and the end time of the region C are rendered in "begin="T4"end=" "". -
Fig. 20(f) illustrates the TTML document corresponding to theevent 6. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end="T5"", and the start time and the end time of the region C are rendered in "begin="T4"end=" "". In this case, as indicated in a broken-line rectangular box, the TTML document is filled with region-B display-end-time information. -
Fig. 20(g) illustrates the TTML document corresponding to theevent 7. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region B are rendered in "begin="T4"end=" "". -
Fig. 20(h) illustrates the TTML document corresponding to theevent 8. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region C are rendered in "begin="T4"end="T6"". In this case, as indicated in a broken-line rectangular box, the TTML document is filled with region-C display-end-time information. - Next, other live subtitle delivery with variable samples will be described.
Fig. 21 illustrates an exemplary TTML layer and an exemplary container layer in that case. - Information regarding a region A is present in the TTML layer. The region A is displayed from timing T1 to timing T3 at the end, but T3 cannot get previously known. Therefore, initially, only the fact that "start time" is T1 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region A.
- Furthermore, information regarding a region B is present in the TTML layer. The region B is displayed from timing T2 to timing T5 at the end, partially overlapping with the region A described above, temporally, but T5 cannot get previously known. Therefore, initially, only the fact that "start time" is T2 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region B.
- Moreover, information regarding a region C is present in the TTML layer. The region C is displayed from timing T4 to timing T6 at the end, partially overlapping with the region B described above, temporally, but T6 cannot get previously known. Therefore, initially, only the fact that "start time" is T4 is written and a blank is rendered for an end time "end time" in a TTML document, in relation to the region C.
- In this case, as illustrated in
Fig. 21 , from T1 to T2, from T2 to T3, from T3 to T4, from T4 to T5, and from T5 to T6 each are defined as one sample. Here, an event is provided for every sample. Accordingly, anevent 1 includes the sample of from T1 to T2, anevent 2 includes the sample of from T2 to T3, anevent 3 includes the sample of from T3 to T4, anevent 4 includes the sample of from T4 to T5, and anevent 5 includes the sample of from T5 to T6. - In accordance with each event, a PES packet (container packet) having a segment packet of a TTML document (subtitle segment_type = 0xA0) including region information regarding the event, disposed in a payload, is generated and is transmitted. In this case, the value of presentation time stamp (PTS) to be inserted into the header of each PES packet, is the start time of the corresponding sample (Sample start time).
- Then, each PES packet is transmitted so as to arrive at the reception side at least at the start time of the corresponding sample. For example, a PES packet having a segment packet of a TTML document including the region information regarding the
event 1, disposed in a payload, is transmitted so as to arrive at the reception side before the time T1. Furthermore, for example, a PES packet having a segment packet of a TTML document including the region information regarding theevent 2, disposed in a payload, is transmitted so as to arrive at the reception side before the time T2. A similar manner is made for the others. -
Figs. 22(a) to 22(e) schematically illustrate the contents of the respective TTML documents corresponding to the events.Fig. 22(a) illustrates the TTML document corresponding to theevent 1. The TTML document includes only the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end=" "". -
Fig. 22(b) illustrates the TTML document corresponding to theevent 2. The TTML document includes the information regarding the region B together with the information regarding the region A. In this case, the start time and the end time of the region A are rendered in "begin="T1"end=" "", and the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 22(c) illustrates the TTML document corresponding to theevent 3. The TTML document includes only the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "". -
Fig. 22(d) illustrates the TTML document corresponding to theevent 4. The TTML document includes the information regarding the region C together with the information regarding the region B. In this case, the start time and the end time of the region B are rendered in "begin="T2"end=" "", and the start time and the end time of the region C are rendered in "begin="T4"end=" "". -
Fig. 22(e) illustrates the TTML document corresponding to theevent 5. The TTML document includes only the information regarding the region C. In this case, the start time and the end time of the region C are rendered in "begin="T4"end=" "". - In a case where the display end times of the regions A, B, and C get known in the example, a PES packet having a segment packet of a control command for giving region display-end-time information (command=0x01) (subtitle segment_type = 0xA2), disposed in a payload, is generated and is transmitted so as to arrive at the reception side before each of the times. In this case, the value of presentation time stamp (PTS) to be inserted into the header of the PES packet, indicates the region display end time. Accordingly, in this case, the region display end time is given with PTS.
- Note that it is considered that region display-end-time information is given to the reception side with a control command (command=0x03) instead of with the control command (command=0x01) (refer to
Figs. 9 and10 ). In this case, a region display state is indicated with a count at previously defined time intervals, such as intervals of one second, and a PES packet having a segment packet of a control command including the count (command=0x03) (subtitle segment_type = 0xA2), disposed in a payload, is generated and is transmitted sequentially. - In this case, the value of presentation time stamp (PTS) to be inserted into the header of the PES packet indicates the transmission time of the PES packet, for example. Then, in this case, when the count indicates the end, the region display end time is given by addition of a time offset added to the count, to PTS.
-
Fig. 23 illustrates an exemplary TTML layer and an exemplary container layer in that case. For example, regarding a region A, PES packets having segment packets of respective control commands including "1" and "FFFF" as a count TC (command=0x03) (subtitle segment_type = 0xA2), disposed in payloads, each are generated and are transmitted sequentially at intervals of one second after time T1 passes. - In this case, because the display end time of the region A gets known as T3 after the transmission of the PES packet according to "1" as the count TC, "FFFF" indicating a display end as the next count TC, with a time offset "offset" added thereto, is transmitted, and PTS is PTS(2) at this time. Here, the relationship of T3 = PTS(2) + offset is satisfied.
- Furthermore, regarding a region B, PES packets having segment packets of respective control commands including "1", "2", "3", "4", "5", and "FFFF" as a count TC (command=0x03) (subtitle segment_type = 0xA2), disposed in payloads, each are generated and are transmitted sequentially at intervals of one second after time T2 passes.
- In this case, because the display end time of the region B gets known as T5 after the transmission of the PES packet according to "5" as the count TC, "FFFF" indicating an display end as the next count TC, with a time offset "offset" added thereto, is transmitted, and PTS is PTS(8) at this time. Here, the relationship of T5 = PTS(8) + offset is satisfied.
- Furthermore, regarding a region C, PES packets having segment packets of respective control commands including "1", "2", and "FFFF" as a count TC (command=0x03) (subtitle segment_type = 0xA2), disposed in payloads, each are generated and are transmitted sequentially at intervals of one second after time T4 passes.
- In this case, because the display end time of the region C gets known as T6 after the transmission of the PES packet according to "2" as the count TC, "FFFF" indicating an display end as the next count TC, with a time offset "offset" added thereto, is transmitted, and PTS is PTS(11) at this time. Here, the relationship of T6 = PTS(11) + offset is satisfied.
- Referring back to
Fig. 2 , theTS formatter 116 renders the video stream generated by thevideo encoder 112, the audio stream generated by theaudio encoder 113, and the subtitle stream generated by thesubtitle encoder 115 into transport packetization and performs multiplexing, to acquire the transport stream TS as a multiplexed stream. - The operation of the
stream generation unit 110 illustrated inFig. 2 , will be briefly described. The image data DV is supplied to thevideo encoder 112. Thevideo encoder 112 performs coding to the image data DV, to generate a video stream (PES stream) including video PES packets each having coded image data in a payload. The video stream is supplied to theTS formatter 116. - Furthermore, the audio data DA is supplied to the
audio encoder 113. Theaudio encoder 113 performs coding to the audio data DA, to generate an audio stream (PES stream) including audio PES packets each having coded audio data. The audio stream is supplied to theTS formatter 116. - Furthermore, the text data (character code) DT as the subtitle information is supplied to the text
format conversion unit 114. The textformat conversion unit 114 acquires subtitle text information having display timing information, in a predetermined format, here, TTML. The TTML is supplied to thesubtitle encoder 115. - The
subtitle encoder 115 generates a segment packet including a document in TTML acquired by the text format conversion unit 114 (timed text subtitling segment: TT subtitling segment). Then, thesubtitle encoder 115 generates a subtitle PES packet having the segment packet disposed in a payload. - The
subtitle encoder 115 generates a subtitle PES packet for every sample. Furthermore, because region display end time is initially unknown for live subtitle delivery, thesubtitle encoder 115 cannot insert the region display end time into the TTML document included in a subtitle PES packet, and thus renders a blank. - Thereafter, when the region display end time gets known, the
subtitle encoder 115 generates a new subtitle PES packet in order to give or clearly indicate region display-end-time information to the reception side. - With the first method adopted, the
subtitle encoder 115 generates a segment packet including a TTML document in which the region end-time information is filled (Subtitle segment_type=0xA0), and newly generates a subtitle PES packet having the segment packet disposed in a payload. - Meanwhile, with the second method adopted, the
subtitle encoder 115 generates a segment packet including a control command for giving the region display-end-time information (Subtitle segment_type=0xA2), and newly generates a subtitle PES packet including the segment packet disposed in a payload. - Furthermore, the
subtitle encoder 115 generates a segment packet including download information for downloading a font file designated in TTML font designation information, and disposes the segment packet in the payload of the PES packet. - The
subtitle encoder 115 generates the subtitle stream (PES stream) including each subtitle PES packet described above. The subtitle stream is supplied to theTS formatter 116. The TS formatter 116 renders the respective streams generated by the encoders into transport packetization and performs multiplexing, to generates the transport stream TS as a multiplexed stream. -
Fig. 24 illustrates an exemplary configuration of the transport stream TS. A video PES packet "Video PES" that is a PES packet of the video stream identified with PID1, is present in the exemplary configuration. Furthermore, an audio PES packet "Audio PES" that is a PES packet of the audio stream identified with PID2, is present in the exemplary configuration. Furthermore, a subtitle PES packet "Subtitle PES" that is a PES packet of the subtitle stream identified with PID3, is present in the exemplary configuration. - The PES packets each include a PES header (PES header) and a PES payload (PES payload). In the video PES packet, the time stamp of DTS/PTS is inserted in the PES header and a video coded stream (coded image data) is inserted in the PES payload. Furthermore, in the audio PES packet, the time stamp of PTS is inserted in the PES header and an audio coded stream (coded audio data) is inserted in the PES payload.
- Furthermore, in the subtitle PES packet, the time stamp of PTS is inserted in the PES header and a segment packet (timed text subtitling segment: TT subtitling segment) having, for example, a TTML document, a control command, or font download information, is inserted in the PES payload.
- Furthermore, the transport stream TS includes a program map table (PMT) as program specific information (PSI). The PSI is information describing to which program each elementary stream included in the transport stream TS belongs.
- A program descriptor (Program Descriptor) in which information relating to the entirety of the programs is written, is present in the PMT. Furthermore, elementary-stream loops having respective pieces of information related to the elementary streams, are present in the PMT. A video elementary-stream loop (Video ES loop), an audio elementary-stream loop (Audio ES loop), and a subtitle elementary-stream loop (Subtitle ES loop) are present in the exemplary configuration. Information regarding, for example, a packet identifier (PID) is disposed in each loop.
-
Fig. 25 illustrates an exemplary configuration of thetelevision receiver 200. Thetelevision receiver 200 has areception unit 201, a TS analysis unit (demultiplexer) 202, avideo decoder 203, avideo superimposition unit 204, apanel drive circuit 205, and adisplay panel 206. Furthermore, thetelevision receiver 200 has anaudio decoder 207, anaudio output circuit 208, aspeaker 209, and asubtitle decoder 210. Furthermore, thetelevision receiver 200 has aCPU 221, aflash ROM 222, aDRAM 223, aninternal bus 224, a remote-control reception unit 225, and a remote-control transmitter 226. - The
CPU 221 controls the operation of each unit of thetelevision receiver 200. Theflash ROM 222 stores control software and retains data. TheDRAM 223 forms a work area for theCPU 221. TheCPU 221 develops the software and the data read from theflash ROM 222 onto theDRAM 223 and boots the software so as to control each unit of thetelevision receiver 200. - The remote-
control reception unit 225 receives a remote-control signal (remote-control code) transmitted from the remote-control transmitter 226 and supplies the remote-control signal to theCPU 221. TheCPU 221 controls each unit of thetelevision receiver 200 on the basis of the remote-control code. TheCPU 221, theflash ROM 222, and theDRAM 223 are connected to theinternal bus 224. - The
reception unit 201 receives the transport stream TS superimposed and sent on the broadcast wave from thebroadcast output system 100. The transport stream TS includes, as described above, the video stream, the audio stream, and the subtitle stream. TheTS analysis unit 202 extracts the respective streams of video, audio, and subtitles from the transport stream TS. - The
audio decoder 207 performs decoding processing to the audio stream extracted by theTS analysis unit 202, to acquire the audio data. Theaudio output circuit 208 performs necessary processing, such as D/A conversion and amplification, to the audio data, and supplies the audio data to thespeaker 209. Thevideo decoder 203 performs decoding processing to the video stream extracted by theTS analysis unit 202, to acquire the image data. - The
subtitle decoder 210 performs decoding processing to the subtitle stream extracted by theTS analysis unit 202, to acquire bitmap data of each region to be superimposed onto the image data. Here, thesubtitle decoder 210 extracts, from a segment packet including a TTML document, the segment packet and acquires the bitmap data of each region on the basis of the TTML document. - Furthermore, the
subtitle decoder 210 extracts, from a segment packet including a control command for giving region display-end-time information, the segment packet and sends the control command thereof to theCPU 221. TheCPU 221 acquires the region display end time, on the basis of the control command. - Here, for a control command (command=0x01), the
CPU 221 acquires PTS inserted in the header of the PES packet as the region display end time. Furthermore, for a control command (command=0x03), theCPU 221 acquires, when the count indicates the end, the time offset that has been added to the count, added to PTS that has been inserted in the header of the PES packet, as the display end time. - Furthermore, the
subtitle decoder 210 extracts, from a segment packet including font download information, the segment packet and sends the font download information to theCPU 221. TheCPU 221 acquires, from the font download segment, at least information for downloading a font file designated in TTML-document font designation information. - The
video superimposition unit 204 superimposes the bitmap data of each region acquired from thesubtitle decoder 210, onto the image data acquired by thevideo decoder 203. In this case, thevideo superimposition unit 204 superimposes the bitmap data of each region by the period from the display start time to the display end time of each region. - Here, the display start time is acquired from the region display-start-time information included in each TTML document. For the pre-authored subtitle delivery, the display end time is acquired from the region display-end-time information included in each TTML document (refer to
Figs. 13 and17 ). - Furthermore, for the live subtitle delivery, the display end time is acquired from the region display-end-time information included in each updated TTML document when the first method is adopted (refer to
Fig. 19 ) and is acquired from each control command when the second method is adopted (refer toFigs. 15 ,21 , and23 ). - The
panel drive circuit 205 drives thedisplay panel 206, on the basis of the image data for display acquired by thevideo superimposition unit 204. Thedisplay panel 206 includes, for example, a liquid crystal display (LCD), an organic electroluminescence (EL) display, or the like. - The operation of the
television receiver 200 illustrated inFig. 25 , will be briefly described. Thereception unit 201 receives the transport stream TS superimposed and sent on the broadcast wave from thebroadcast output system 100. The transport stream TS includes the video stream, the audio stream, and the subtitle stream. - The transport stream TS is supplied to the
TS analysis unit 202. TheTS analysis unit 202 extracts the respective streams of video, audio, and subtitles from the transport stream TS. The video stream (PES stream) extracted by theTS analysis unit 202 is supplied to thevideo decoder 203. Thevideo decoder 203 performs decoding processing to the video stream, to acquire the image data. The image data is supplied to thevideo superimposition unit 204. - Furthermore, the subtitle stream (PES stream) extracted by the
TS analysis unit 202, is supplied to thesubtitle decoder 210. Thesubtitle decoder 210 performs decoding processing to the subtitle stream, to acquire bitmap data of each region to be superimposed onto the image data. - Furthermore, the
subtitle decoder 210 extracts, from a segment packet including a control command for giving region display-end-time information, the segment packet and sends the control command to theCPU 221. TheCPU 221 acquires the region display end time, on the basis of the control command. - Furthermore, the
subtitle decoder 210 extracts, from a segment packet including font download information, the segment packet and sends the font download information to theCPU 221. TheCPU 221 acquires, from the font download segment, at least information for downloading a font file designated in TTML-document font designation information. The information is used as necessary in order to acquire the font file. - The bitmap data of each region output from the
subtitle decoder 210, is supplied to thevideo superimposition unit 204. Thevideo superimposition unit 204 superimposes the bitmap data of each region output from thesubtitle decoder 210, onto the image data acquired by thevideo decoder 203. In this case, the bitmap data of each region is superimposed by the period from the display start time to the display end time of each region. - Here, the display start time is acquired from the region display-start-time information included in each TTML document. Furthermore, for the pre-authored subtitle delivery, the display end time is acquired from the region display-end-time information included in each TTML document (refer to
Figs. 13 and17 ). Furthermore, for the live subtitle delivery, the display end time is acquired from the region display-end-time information included in each updated TTML document when the first method is adopted (refer toFig. 19 ) and is acquired from each control command when the second method is adopted (refer toFigs. 15 ,21 , and23 ). - The image data for display acquired by the
video superimposition unit 204, is supplied to thepanel drive circuit 205. Thepanel drive circuit 205 drives thedisplay panel 206, on the basis of the image data for display. This arrangement allows thedisplay panel 206 to display the image on which each region is superimposed. - As described above, although the
broadcast output system 100 in the transmission andreception system 10 illustrated inFig. 1 generates a subtitle PES packet having a segment packet having a TTML document, disposed in a payload, for every sample, the period of each sample is varied, for example, in accordance with a variation period of subtitle display. Therefore, the transmission frequency of a subtitle PES packet can be inhibited, so that transport efficiency can improve. - Furthermore, for the live subtitle delivery, the
broadcast output system 100 in the transmission andreception system 10 illustrated inFig. 1 sends the region display-end-time information with the TTML layer (first method) or the container layer (second method) when the region display end time gets known. Therefore, the region display-end-time information can be given to the reception side, so that the region display end processing can be performed properly on the reception side. - Furthermore, for the live subtitle delivery, the
broadcast output system 100 in the transmission andreception system 10 illustrated inFig. 1 newly generates and transmits, for example, a PES packet having a segment packet having a control command for giving region display-end-time information, inserted in a payload, when the region display end time gets known (second method). Therefore, even in a case where the period of each sample is fixed, the region display-end-time information can be favorably given to the reception side, so that the region display end processing can be performed properly on the reception side. - Note that, the case where the command packet transport is not limited to the fixed samples, has been given in
Fig. 15 described above. However, similarly to the TTML document transport, it is assumed that the command packet transport is performed under limitation due to the fixed samples. In that case, as illustrated inFig. 26 , signalization of "latency(LTEn)" from PTS enables desired timing designation to command execution of a display end (refer toFig. 9 ). - Furthermore, the example in which TTML is used as the subtitle text information having the display timing information, in the predetermined format, has been given in the embodiment described above. However, the present technology is not limited to this, and thus it is considered that other timed text information having information similar to that of TTML is used. For example, a TTML derived format may be used.
- Furthermore, the transmission and
reception system 10 including thebroadcast output system 100 and thetelevision receiver 200 has been given in the embodiment described above, but the configuration of a transmission and reception system to be acquired by the application of the present technology is not limited to this. For example, the part of thetelevision receiver 200 may include a set-top box and a monitor connected through a digital interface, such as high-definition multimedia interface (HDMI). Note that "HDMI" is a registered trademark. - Furthermore, the example in which the container (multiplexed stream) is the MPEG-2 transport stream, has been given in the embodiment described above. Needless to say, the present technology can be applied similarly even in a case where the multiplexed stream is, for example, an MMT stream or a DASH/ISOBMFF stream.
- According to a main feature of the present technology, varying the period of each sample in accordance with a variation period of subtitle display inhibits the transmission frequency of a subtitle PES packet including a TTML document, so that transport efficiency can improve (refer to
Fig. 17 ). Furthermore, according to a main feature of the present technology, for live subtitle delivery, when region display end time gets known, transmission of region display-end-time information with a TTML layer or a container layer, allows region display end processing to be performed properly on the reception side (refer toFigs. 15 ,19 ,21 , and23 ). -
- 10
- Transmission and reception system
- 100
- Broadcast output system
- 110
- Stream generation unit
- 111
- Control unit
- 112
- Video encoder
- 113
- Audio encoder
- 114
- Text format conversion unit
- 115
- Subtitle encoder
- 116
- TS formatter
- 200
- Television receiver
- 201
- Reception unit
- 202
- TS analysis unit
- 203
- Video decoder
- 204
- Video superimposition unit
- 205
- Panel drive circuit
- 206
- Display panel
- 207
- Audio decoder
- 208
- Audio output circuit
- 209
- Speaker
- 210
- Subtitle decoder
- 221
- CPU
- 222
- Flash ROM
- 223
- DRAM
- 224
- Internal bus
- 225
- Remote-control reception unit
- 226
- Remote-control transmitter
Claims (8)
- A transmission device (100) for transmitting a plurality of subtitles for different regions, the transmission device (100) comprising:a packet generation unit (115) configured to generate for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, and wherein display of the subtitle is to be ended in the sample related to the second PES packet; anda transmission unit (116) configured to transmit the PES packets as a subtitle stream,wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles.
- The transmission device (100) according to claim 1, wherein each of the PES packets includes a respective header with a time stamp indicating the start time of the related sample.
- A reception device (200) for receiving a plurality of subtitles for different regions, the reception device (200) comprising:a reception unit (201) configured to receive for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, and wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles; anda control unit (210) configured to control subtitle display based on the TTML documents of subtitle text information disposed in the PES packets, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, wherein display of the subtitle is to be ended in the sample related to the second PES packet, and wherein the control unit (210) is configured to end subtitle display based on the display timing information disposed in the second PES packet.
- The reception device (200) according to claim 3, wherein each of the PES packets includes a respective header with a time stamp indicating the start time of the related sample.
- A transmission method for transmitting a plurality of subtitles for different regions, the method comprising:generating for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, and wherein display of the subtitle is to be ended in the sample related to the second PES packet; andtransmitting the PES packets as a subtitle stream,wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles.
- The transmission method according to claim 5, wherein each of the PES packets includes a respective header with a time stamp indicating the start time of the related sample.
- A reception method for receiving a plurality of subtitles for different regions, the method comprising:receiving for each of a plurality of consecutive samples of variable duration a respective Packetized Elementary Stream, PES, packet comprising a Timed Text Markup Language, TTML, document of subtitle text information in a payload, wherein each TTML document of subtitle text information comprises display timing information for the related sample, wherein each display timing information indicates at least a display start time for a subtitle of the plurality of subtitles that is to be displayed in the related sample, and wherein for each of the plurality of consecutive samples a start time of the respective sample coincides with the display start time for one of the plurality of subtitles or an end time of the respective sample coincides with a display end time for one of the plurality of subtitles; andcontrolling subtitle display based on the TTML documents of subtitle text information disposed in the PES packets, wherein the display timing information in a first PES packet of the PES packets does not indicate a display end time for the subtitle, wherein the display timing information in a succeeding second PES packet of the PES packets indicates the display end time for the subtitle, wherein display of the subtitle is to be ended in the sample related to the second PES packet, and wherein the subtitle display is ended based on the display timing information disposed in the second PES packet.
- The reception method according to claim 7, wherein each of the PES packets includes a respective header with a time stamp indicating the start time of the related sample.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP21204372.3A EP3979654A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016146812 | 2016-07-26 | ||
PCT/JP2017/026080 WO2018021107A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21204372.3A Division-Into EP3979654A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
EP21204372.3A Division EP3979654A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3493548A1 EP3493548A1 (en) | 2019-06-05 |
EP3493548A4 EP3493548A4 (en) | 2019-06-05 |
EP3493548B1 true EP3493548B1 (en) | 2022-01-26 |
Family
ID=61016167
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17834115.2A Active EP3493548B1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
EP21204372.3A Pending EP3979654A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21204372.3A Pending EP3979654A1 (en) | 2016-07-26 | 2017-07-19 | Transmission device, transmission method, reception device, and reception method |
Country Status (4)
Country | Link |
---|---|
EP (2) | EP3493548B1 (en) |
JP (1) | JPWO2018021107A1 (en) |
CN (1) | CN109479154B (en) |
WO (1) | WO2018021107A1 (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015159363A (en) * | 2014-02-21 | 2015-09-03 | 日本放送協会 | receiver and broadcasting system |
JP2016110645A (en) * | 2014-11-28 | 2016-06-20 | 日本放送協会 | Dividing device, analysis device, and program |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006238147A (en) * | 2005-02-25 | 2006-09-07 | Toshiba Corp | Content reproducing device, subtitle reproducing method, and program |
JP4779981B2 (en) * | 2007-02-01 | 2011-09-28 | 三菱電機株式会社 | DIGITAL VIDEO INFORMATION DATA GENERATION DEVICE, DIGITAL VIDEO INFORMATION RECORDING DEVICE, DIGITAL VIDEO INFORMATION REPRODUCING DEVICE, AND DIGITAL VIDEO INFORMATION DATA GENERATION METHOD |
JP4518194B2 (en) * | 2008-06-10 | 2010-08-04 | ソニー株式会社 | Generating apparatus, generating method, and program |
JP5685969B2 (en) | 2011-02-15 | 2015-03-18 | ソニー株式会社 | Display control method and display control apparatus |
EP2739047A4 (en) * | 2011-07-29 | 2015-03-25 | Sony Corp | Streaming distribution device and method, streaming receiving device and method, streaming system, program, and recording medium |
JP5672409B1 (en) * | 2014-10-31 | 2015-02-18 | ソニー株式会社 | Receiving apparatus and data processing method |
US10511867B2 (en) * | 2014-12-19 | 2019-12-17 | Sony Corporation | Transmission apparatus, transmission method, reception apparatus, and reception method |
-
2017
- 2017-07-19 JP JP2018529802A patent/JPWO2018021107A1/en active Pending
- 2017-07-19 CN CN201780044904.6A patent/CN109479154B/en active Active
- 2017-07-19 EP EP17834115.2A patent/EP3493548B1/en active Active
- 2017-07-19 EP EP21204372.3A patent/EP3979654A1/en active Pending
- 2017-07-19 WO PCT/JP2017/026080 patent/WO2018021107A1/en unknown
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015159363A (en) * | 2014-02-21 | 2015-09-03 | 日本放送協会 | receiver and broadcasting system |
JP2016110645A (en) * | 2014-11-28 | 2016-06-20 | 日本放送協会 | Dividing device, analysis device, and program |
Also Published As
Publication number | Publication date |
---|---|
JPWO2018021107A1 (en) | 2019-05-09 |
CN109479154B (en) | 2021-11-05 |
EP3493548A1 (en) | 2019-06-05 |
EP3493548A4 (en) | 2019-06-05 |
WO2018021107A1 (en) | 2018-02-01 |
CN109479154A (en) | 2019-03-15 |
EP3979654A1 (en) | 2022-04-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9131274B2 (en) | Method for transceiving media files and device for transmitting/receiving using same | |
JP7020406B2 (en) | Transmitter, transmitter, receiver and receiver | |
JP7176598B2 (en) | Sending method | |
RU2701172C2 (en) | Transmitting device, transmission method, receiving device and reception method | |
US10225600B2 (en) | Transmission device, transmission method, reproduction device and reproduction method | |
US10511802B2 (en) | Transmission device, transmission method, reproducing device and reproducing method | |
US11765330B2 (en) | Transmitter, transmission method, receiver, and reception method | |
US9426506B2 (en) | Apparatuses for providing and receiving augmented broadcasting service in hybrid broadcasting environment | |
EP3493548B1 (en) | Transmission device, transmission method, reception device, and reception method | |
US11290785B2 (en) | Transmission apparatus, transmission method, reception apparatus, and reception method for transmitting subtitle text information | |
EP3306941B1 (en) | Transmission device, transmission method, media processing device and media processing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20190226 |
|
A4 | Supplementary search report drawn up and despatched |
Effective date: 20190423 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20200813 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20210222 |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
INTC | Intention to grant announced (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20210604 |
|
RAP3 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: SONY GROUP CORPORATION |
|
GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTC | Intention to grant announced (deleted) | ||
INTG | Intention to grant announced |
Effective date: 20210914 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1466207 Country of ref document: AT Kind code of ref document: T Effective date: 20220215 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602017052875 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1466207 Country of ref document: AT Kind code of ref document: T Effective date: 20220126 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220526 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220426 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220426 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220427 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220526 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602017052875 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20221027 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20220731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220719 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220731 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230514 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20220719 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230725 Year of fee payment: 7 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230725 Year of fee payment: 7 Ref country code: DE Payment date: 20230726 Year of fee payment: 7 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20170719 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220126 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20240725 Year of fee payment: 8 |