JP2020025325A - Receiver, transmitter, and data processing method - Google Patents

Abstract

To provide a receiver capable of displaying captions at desired timing.SOLUTION: The receiver acquires a piece of caption information relevant to captions, which are transmitted on a broadcast wave of digital broadcasting, and control information which includes a piece of selection information for selecting a specific mode from plural modes for specifying a timing to display of the captions. The receiver controls the display of the captions in accordance with the caption information at a display timing corresponding to a specific mode based on a piece of selection information included in the control information. This technique is applicable to, for example, a television receiver conforming to ATSC 3.0.SELECTED DRAWING: Figure 1

Description

  The present technology relates to a receiving device, a transmitting device, and a data processing method, and particularly to a receiving device, a transmitting device, and a data processing method that can display subtitles at desired timing.

  As a method for superimposing and displaying subtitles on a video, TTML (Timed Text Markup Language), which is a markup language capable of specifying display timing, display position, and the like, is known (for example, see Patent Document 1). ). TTML is standardized by the W3C (World Wide Web Consortium).

JP 2012-169885 A

  By the way, a technical method for displaying subtitles using TTML or the like has not been established, and a proposal for displaying subtitles at desired timing has been requested.

  The present technology has been made in view of such a situation, and is to enable subtitles to be displayed at a desired timing.

  The receiving device according to the first aspect of the present technology includes a receiving unit that receives a broadcast wave of a digital broadcast, a plurality of subtitle information related to subtitles transmitted by the broadcast wave, and a plurality of subtitle information for specifying a display timing of the subtitle. An acquisition unit that acquires control information including selection information for selecting a specific mode among the modes, and a display timing according to the specific mode based on the selection information included in the control information. , A control unit that controls the display of the caption according to the caption information.

  The receiving device according to the first aspect of the present technology may be an independent device, or may be an internal block configuring one device. Further, the data processing method according to the first aspect of the present technology is a data processing method corresponding to the above-described receiving device according to the first aspect of the present technology.

  In the receiving device and the data processing method according to the first aspect of the present technology, the caption information related to captions transmitted by a broadcast wave of a digital broadcast and a plurality of modes for specifying the timing of display of the captions are included. Control information including selection information for selecting a specific mode is acquired, and based on the selection information included in the control information, at a display timing according to the specific mode, the subtitle information according to the subtitle information is obtained. The display of subtitles is controlled.

  A transmission device according to a second aspect of the present technology, a generation unit that generates control information including selection information for selecting a specific mode from among a plurality of modes for specifying a subtitle display timing, A transmission unit that transmits the control information in the form of a digital broadcast wave together with the subtitle information relating to the subtitles.

  The transmission device according to the second aspect of the present technology may be an independent device, or may be an internal block configuring one device. Further, a data processing method according to the second aspect of the present technology is a data processing method corresponding to the above-described transmission device according to the second aspect of the present technology.

  In the transmission device and the data processing method according to the second aspect of the present technology, in the transmission device and the data processing method, in the broadcast wave of the digital broadcast, identification of a plurality of modes for specifying the display timing of the subtitles, which is transmitted together with the subtitle information on the subtitles Control information including selection information for selecting the mode is generated.

  According to the first and second aspects of the present technology, captions can be displayed at desired timing.

  Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

FIG. 1 is a diagram illustrating a configuration of an embodiment of a transmission system to which the present technology is applied. FIG. 3 is a diagram illustrating a protocol stack according to the embodiment. FIG. 3 is a diagram showing the structure of an MPD file. FIG. 3 is a diagram illustrating a description example of an MPD file. FIG. 10 is a diagram illustrating a relationship between an AdaptationSet element and a Representation element when expressing a multiplexed stream. FIG. 4 is a diagram illustrating a relationship among an AdaptationSet element, a Representation element, and a SubRepresentation element when expressing a multiplexed stream. FIG. 4 is a diagram illustrating an example of attributes and elements that can be included in an AdaptationSet element. FIG. 3 is a diagram illustrating an example of attributes and elements that can be included in a Representation element. FIG. 3 is a diagram showing an MP4 file format. It is a figure showing an example of TTML processing mode. FIG. 8 is a diagram illustrating an example of a description of an MPD file when operation in mode 1 is performed. FIG. 11 is a diagram illustrating an example of subtitle display timing when operation in mode 1 is performed. FIG. 14 is a diagram illustrating an example of a description of an MPD file when operation in mode 2-1 is performed. It is a figure which shows the example of the display timing of a subtitle in case operation | movement in mode 2-1 is performed. It is a figure showing the example of description of the MPD file when operation in mode 2-2 is performed. It is a figure which shows the example of the display timing of a subtitle when operation | movement in mode 2-2 is performed. FIG. 13 is a diagram illustrating an example of a description of an MPD file when operation in mode 3 is performed. FIG. 10 is a diagram illustrating an example of subtitle display timing when operation in mode 3 is performed. FIG. 3 is a diagram illustrating a configuration example of an ATSC server. FIG. 3 is a diagram illustrating a detailed configuration example of an ATSC server. FIG. 3 is a diagram illustrating a detailed configuration example of an ATSC client. FIG. 3 is a diagram illustrating an example of a software configuration of an ATSC client. It is a flowchart explaining a transmission process. It is a flowchart explaining a component signaling process. It is a flowchart explaining a receiving process. It is a flowchart explaining a component signaling process. FIG. 14 is a diagram illustrating a configuration example of a computer.

  Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order.

1. 1. System configuration 2. Overview of this technology Specific operation example (1) Mode 1: TTML Time Only
(2) Mode 2: Sample Time Only
(2-1) Mode 2-1: Sample Time Only
(2-2) Mode 2-2: Sample Time Only But Till Next
(3) Mode 3: Asap
4. 4. Configuration of each device 5. Flow of processing executed in each device Modification 7 Computer configuration

<1. System Configuration>

  FIG. 1 is a diagram illustrating a configuration of an embodiment of a transmission system to which the present technology is applied. It should be noted that a system refers to a logical collection of a plurality of devices.

  In FIG. 1, a transmission system 1 includes an ATSC server 10 and an ATSC client 20. In this transmission system 1, data transmission conforming to digital broadcasting standards such as ATSC3.0 is performed. ATSC 3.0 is a next-generation ATSC (Advanced Television Systems Committee) standard that is currently being formulated.

  The ATSC server 10 is a transmitter corresponding to a digital broadcasting standard such as ATSC3.0, and includes, for example, a plurality of servers. The ATSC server 10 transmits (transmits) a stream of (components of) video, audio, subtitles, and the like, which constitute content such as a television program, as a digital broadcast signal via the transmission path 30.

  The ATSC client 20 is a receiver compatible with digital broadcasting standards such as ATSC3.0, and is, for example, a fixed receiver such as a television receiver or a set-top box, or a smartphone, a mobile phone, a tablet computer, or the like. It is a mobile receiver. Further, the ATSC client 20 may be a device mounted on an automobile such as an on-vehicle television, for example.

  The ATSC client 20 receives a digital broadcast signal transmitted (transmitted) from the ATSC server 10 via the transmission path 30, acquires and processes a stream of (a component of) video, audio, subtitles, and the like, Outputs video and audio of content such as television programs.

  In FIG. 1, as the transmission path 30, for example, a satellite line, a cable television network (wired line), or the like can be used in addition to terrestrial waves.

<2. Overview of this technology>

(Protocol stack)
By the way, in ATSC3.0, it has been decided to use not IP (UDP packets) for data transmission but IP (Internet Protocol) packets including UDP (User Datagram Protocol) packets, instead of TS (Transport Stream) packets. .

  In ATSC 3.0, ROUTE (Real-Time Object Delivery over Unidirectional Transport) and MMT (MPEG Media Transport) coexist as transport protocols, and video and video are transmitted using either transport protocol. A stream of (a component of) audio, subtitles, and the like is transmitted.

  Here, ROUTE is an extension of FLUTE (File Delivery over Unidirectional Transport), which is a protocol suitable for multicast transfer of a binary file in one direction. MMT is a transport method used on IP (Internet Protocol), and by setting an IP address and a URL (Uniform Resource Locator) by control information, data such as video and audio can be referred to. .

  Furthermore, in ATSC3.0, it is assumed that, as signaling, LLS (Link Layer Signaling) signaling information and SLS (Service Layer Signaling) signaling information are defined, and described in the LLS signaling information obtained in advance. The SLS signaling information for each service is obtained according to the information to be provided.

  Here, the LLS signaling information includes, for example, metadata such as an SLT (Service List Table). The SLT metadata includes information indicating the configuration of a stream or service in a broadcast network, such as information necessary for selecting a service (tuning information).

  The SLS signaling information includes, for example, metadata such as USD (User Service Description), LSID (LCT Session Instance Description), and MPD (Media Presentation Description). The USD metadata includes information such as the acquisition source of other metadata. LSID metadata is control information of the ROUTE protocol. The MPD metadata is control information for managing reproduction of a component stream. Note that metadata such as USD, LSID, and MPD are described in a markup language such as XML (Extensible Markup Language). The MPD metadata conforms to the MPEG-DASH (Dynamic Adaptive Streaming over HTTP) standard. Since the MPD metadata is provided as an XML format file, it will be described as an MPD file in the following description.

  FIG. 2 is a diagram illustrating a protocol stack according to the present embodiment.

  In FIG. 2, the lowest layer is a physical layer. The upper layer adjacent to the physical layer is the layer 2 (Layer 2), and the upper layer adjacent to the layer 2 is the IP layer. The upper layer adjacent to the IP layer is the UDP layer. That is, an IP packet (IP / UDP packet) including a UDP packet is arranged in the payload of the layer 2 generic packet, and is encapsulated. The physical layer frame (ATSC Physical Frame) is composed of a preamble and a data part.In the data part, parity for error correction is added to the BB frame obtained by encapsulating multiple generic packets. After that, data obtained by performing processing related to the physical layer such as interleaving and mapping is mapped.

  The upper layers adjacent to the UDP layer are ROUTE, MMT, and SLT. That is, the video, audio, and subtitle streams, the SLS signaling information stream, and the NRT content stream transmitted in the ROUTE session are stored in IP / UDP packets and transmitted. The NRT content is a content distributed by NRT (Non Real Time) broadcasting, and is reproduced after being temporarily stored in the storage of the ATSC client 20. Also, a file (for example, an application file) other than the NRT content may be transmitted in the ROUTE session.

  On the other hand, the video, audio, and subtitle streams and the SLS signaling information stream transmitted in the MMT session are stored in IP / UDP packets and transmitted. Further, the SLT metadata is stored in an IP / UDP packet and transmitted.

  Since the protocol stack as described above is employed, the ATSC client 20 uses the channel selection information included in the SLT metadata when selecting a service (channel) provided by the component stream transmitted in the ROUTE session. , SLS signaling information transmitted in the ROUTE session is acquired (S1-1, S1-2). Then, the ATSC client 20 connects to the stream of the component providing the selected service according to the metadata such as USD, LSID, and MPD (S1-3). Thus, the ATSC client 20 outputs video and audio of content (for example, a television program) according to the selected service.

  Also, when selecting a service provided by the component stream transmitted in the MMT session, the ATSC client 20 acquires SLS signaling information transmitted in the MMT session according to the tuning information included in the SLT metadata ( S2-1, S2-2). Then, the ATSC client 20 connects to the stream of the component providing the selected service according to the metadata such as USD, LSID, and MPD (S2-3). Thus, the ATSC client 20 outputs video and audio of content (for example, a television program) according to the selected service.

(MPD file structure)
Next, an MPD file transmitted as SLS signaling information will be described with reference to FIGS. FIG. 3 is a diagram showing the structure of the MPD file. In addition, the MPD file is described in a hierarchical structure in an XML format, as shown in the description example of FIG.

  Here, as shown in FIG. 3, in the MPD file, a Period element, an AdaptationSet element, a Representation element, and a SubRepresentation element are described in a hierarchical structure. The Period element is a unit that describes the configuration of content such as a TV program. An AdaptationSet element, a Representation element, or a SubRepresentation element is used for each stream of (components of) video, audio, subtitles, and the like, and can describe attributes of each stream.

  Specifically, the AdaptationSet element represents a stream encoded from various sources. Then, in order to allow the ATSC client 20 to select the stream according to, for example, a parametric such as a bit rate, a Representation element is arranged in an AdaptationSet element, and a plurality of options having different parameters such as a bit rate are displayed. Enumerated streams. Normally, the AdaptationSet element and the Representation element correspond to a single stream such as a video, audio, or subtitle stream.

  When the AdaptationSet element represents a stream in which a plurality of streams such as a video stream, an audio stream, and a subtitle stream are multiplexed, a Representation element is arranged in the AdaptationSet element, and parameters such as a bit rate are different. Enumerate multiplexed streams for multiple choices. That is, as shown in FIG. 5, for each Period element representing a time interval, a plurality of AdaptationSet elements representing a multiplexed stream are arranged, and a plurality of Representation elements arranged in those AdaptationSet elements form, for example, a bit. A plurality of multiplexed streams having different rates can be listed.

  In this case, by further arranging the SubRepresentation element under the Representation element, the attribute of the stream of each component constituting the multiplexed stream can be described. That is, as shown in FIG. 6, a plurality of AdaptationSet elements representing multiplexed streams are arranged for each Period element representing a time interval, and multiplexed streams having different bit rates, for example, are arranged in the AdaptationSet elements. Are represented, and the attributes of, for example, a video stream, an audio stream, and a subtitle stream can be described by the SubRepresentation elements arranged in the Representation elements.

  Note that the AdaptationSet element corresponds to a stream in which a plurality of streams are multiplexed in addition to a single stream such as a video stream and an audio stream. In the MPEG-DASH standard, the attributes and elements shown in FIG. 7 are defined as attributes and elements that can be included in such an AdaptationSet element. In addition, the Representation element lists a plurality of alternative streams having different parameters such as bit rates, for example, within the range of the AdaptationSet element which is a higher-order element (parent element). In the MPEG-DASH standard, the attributes and elements shown in FIG. 8 are defined as attributes and elements that can be included in such a Representation element. Note that the attributes and elements in FIG. 8 can be included in the SubRepresentation element.

(MP4 file format)
By the way, when ROUTE is used as the transport protocol, it is assumed that the MP4 file format is adopted as the streaming file format. The MP4 file format is a derivative of the ISO Base Media File Format defined in ISO / IEC 14496-12. The ISO base media file format has a tree structure called a box.

Here, the segment transmitted in the ROUTE session includes an initialization segment (Initialization Segment) and a media segment (Media Segment).
The initialization segment includes initialization information such as a data compression method. The media segment stores stream data of (components of) video, audio, subtitles, and the like.

  FIG. 9 is a diagram showing the structure of a media segment having the MP4 file format.

The media segment is composed of boxes of styp (segment type), sidx (segment index), ssix (subsegment index), moof (movie fragment), and mdat (madia data).
The styp box contains version information of the file format specification of the file in segment units. The sidx box contains index information in the segment. The ssix box contains index information for each sub-segment (level) in the segment. However, in FIG. 9, illustration of each box of stip, sidx, and ssix is omitted.

  The moof box contains control information of the data of the fragmented (component) stream. The moof box includes an mfhd (movie fragment header) box. The mfhd box includes a tfdt (track fragment decode time) box and a trun (track fragment run) box.

  The tfdt box contains BMDT (Base Media Decode Time) indicating the decoding start time of the sample. The trun box includes information indicating SampleCount representing the number of samples, SampleDuration representing the duration of the sample, and CompositionOffset representing the offset value.

  The mdat box stores fragmented (component) stream data (data body) as a sample (Sample). In this sample, the data of the stream (of the component) is stored in a basic unit to be processed.

  In the following description, among the samples, a sample of a TTML format file (TTML file) for displaying subtitles is also referred to as a TTML sample. Further, among the media segments, a media segment whose mdat box includes a TTML sample is also referred to as a TTML segment. Further, in the following description, a simple description of a segment means a media segment.

(TTML processing mode)
Also, when ROUTE is used as a transport protocol, it is assumed that a TTML file in TTML format is used for displaying closed captions (CC). However, a technical method for displaying subtitles specified in the TTML file has not been established, and a proposal for displaying subtitles at desired timing has been requested.

  Therefore, in the present technology, a plurality of modes are defined as the TTML processing mode for specifying the timing of displaying subtitles, and the ATSC server 10 selects a specific mode among the plurality of modes for the MPD file. The selection information for the ATSC client 20 based on the selection information included in the MPD file so that the subtitle specified in the TTML file can be displayed at a display timing corresponding to a specific mode. I do.

  FIG. 10 is a diagram illustrating an example of the TTML processing mode.

  Mode 1, mode 2, and mode 3 are defined as TTML processing modes. In the mode 2, two types, a mode 2-1 and a mode 2-2, are defined.

  Mode 1 is a mode in which captions are displayed at a timing according to time information specified in the TTML file. When mode 1 is set, “atsc: ttmlMode: ttmlTimeOnly” is specified as the selection information in the value of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the SupplementalProperty element in the MPD file.

  In the TTML file, a character string of a caption and time information such as a display start time and a display end time of the caption are specified by a p element in a body element.

  Mode 2 is a mode in which time information specified in the TTML file is ignored, and subtitles are displayed at timings according to the time information specified in the MP4 file format.

  Here, in the mode 2-1 of the mode 2, the display of the subtitles is started at the time corresponding to the BMDT stored in the moof box, and the display is performed only during the time corresponding to the SampleDuration stored in the moof box. To continue. When the mode 2-1 is set, “atsc: ttmlMode: sampleTimeOnly” is specified as the selection information in the value of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the SupplementalProperty element in the MPD file.

  In the mode 2-2, the subtitles are displayed at the time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the target TTML sample, and are displayed in the mdat box storing the next TTML sample. The display is continued until the time corresponding to the BMDT stored in the corresponding moof box. When the mode 2-2 is set, “atsc: ttmlMode: sampleTimeOnlyButTillNext” is specified in the MPD file as the selection information in the value of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the SupplementalProperty element.

  Mode 3 is a mode in which subtitles are displayed ignoring time information specified in the TTML file and time information specified in the MP4 file format. In this case, when the TTML file transmitted from the ATSC server 10 via the transmission path 30 is acquired, the ATSC client 20 immediately displays the caption specified by the TTML file. When mode 3 is set, “atsc: ttmlMode: asap” is specified as the selection information in the value of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the SupplementalProperty element in the MPD file.

<3. Specific operation example>

  Next, an operation example in the case where mode 1, mode 2-1, mode 2-2, and mode 3 are set as the TTML processing mode will be described in order.

(1) Mode 1: TTML Time Only

  First, an operation example when mode 1 is set as the TTML processing mode will be described with reference to FIGS. In this mode 1, only the time information specified in the TTML file is used.

(Example of MPD file description)
FIG. 11 is a diagram illustrating a description example of an MPD file in the case where operation in mode 1 is performed.

  In the MPD file of FIG. 11, a Role element is arranged in the AdaptationSet element under the Period element of the MPD element which is the root element, and "urn: mpeg: dash: role: 2011" is specified as a schemeIdUri attribute, and "Caption" indicating subtitles is specified as an attribute.

  In the AdaptationSet element, "atsc: ttmlMode: ttmlTimeOnly" is specified as the schemeIdUri attribute of the EssentialProperty element. That is, the attribute value of the schemeIdUri attribute of the EssentialProperty element indicates that the mode 1 is set as the TTML processing mode.

(Example of subtitle display timing)
FIG. 12 is a diagram illustrating an example of subtitle display timing when operation in mode 1 is performed.

  In FIG. 12, FIG. 12A schematically shows information on the time specified in the MPD file, and FIG. 12B schematically shows the structure of a segment (TTML segment). FIG. 12C shows a description example of a TTML file obtained from the segment (TTML segment) sample (TTML sample) shown in FIG. 12B. In FIG. 12, the direction of time is from left to right in the figure. Further, these relationships are the same in the drawings for explaining other modes described later.

  As shown in FIG. 12A, in the MPD file, the start time of streaming distribution according to the UTC (Coordinated Universal Time) time is specified in the availabilityStartTime attribute of the MPD element that is the root element. In the MPD file, Period (1), Period (2), ... are specified as Period elements, and the start time of each Period (period) is specified as a start attribute in those Period elements. You.

  In other words, in the MPD file, the start time of each Period is determined by the sum of the start time (time on the WallClock time axis) specified by the availabilityStartTime attribute and the time (time on the MPD time axis) specified by the Period element. Is represented. For example, the sum of the start time (MPD / @ availabilityStartTime) specified by the availabilityStartTime attribute and the start time of Period (2) (MPD / @ availabilityStartTime + MPD / Period (2 ) / @ start) determines the time at the beginning of Period (2).

  FIG. 12B schematically illustrates a segment in Period (2). This segment is a TTML segment in which the mdat box stores a TTML sample as a sample. From the TTML sample stored in the mdat box of the TTML segment, a TTML file is obtained. In the mode 1, time information (BMDT, SampleDuration, etc.) stored in the moof box is ignored.

  FIG. 12C shows a description example of the TTML file. In the TTML file, elements such as the styling element and the layout element in the head element specify the color, font, display position, and the like of characters displayed as subtitles. In the TTML file, a character string of a caption and time information such as a display start time and a display end time of the caption are specified by a p element in a body element.

  Specifically, of the two p elements described in the TTML file of FIG. 12C, a character string “text1” is described between the start tag and the end tag of the upper p element. In the upper p element, "t1" is specified as a begin attribute for setting a display start time, and "t2" is specified as an end attribute for setting a display end time.

  In the TTML file of FIG. 12C, a character string “text2” is described between the start tag and the end tag of the lower p element. In the lower p element, "t2" is specified as the begin attribute and "t3" is specified as the end attribute.

  Here, in the MPD file (FIG. 11) acquired at the time of this operation, since “atsc: ttmlMode: ttmlTimeOnly” is specified as the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element, mode 1 is set as the TTML processing mode. It will be set. When mode 1 is set, subtitles are displayed at a timing according to the time information described in the TTML file. Therefore, subtitles are displayed at timings according to the values of the begin attribute and end attribute of the p element. Will be displayed.

Specifically, as shown in FIG. 12, according to the sum (MPD / @ availabilityStartTime + MPD / Period (2) / @ start) of the start time specified by the availabilityStartTime attribute and the start time of Period (2). When the time t1 has elapsed with reference to the time at the beginning of Period (2), the display of the subtitle "text1" specified in the upper p element is started, and "text1" is displayed until the time t2 has elapsed. Is continued, and when the time t2 has elapsed, the display of the subtitle "text1" ends.
Also, when the time t2 has elapsed with reference to the time at the beginning of Period (2), the display of the subtitle “text2” specified in the lower p element starts, and “text2” continues until the time t3 has elapsed. The display of the subtitle "" is continued, and when the time t3 has elapsed, the display of the subtitle "text2" is terminated.

  Thereby, in the ATSC client 20, when a content such as a television program is being reproduced, the time is after the lapse of the time t1 and before the lapse of the time t2 with respect to the time at the beginning of Period (2). In this case, the subtitle “text1” is superimposed on the video, and when the time t2 has elapsed and before the time t3 has elapsed, the subtitle “text2” is superimposed on the video. Will be.

The operation example in the case where the mode 1 is set as the TTML processing mode has been described above.
This mode 1 uses the time information (values of the begin attribute and end attribute of the p element) specified in the TTML file to start displaying subtitles according to the time set in the begin attribute, By ending the display of the subtitles according to the time set in the subtitles, the subtitles can be displayed at a desired timing.

  For example, in the ATSC server 10, when a package of video, audio, and subtitles by studio recording or the like is produced as content, and the content is distributed to a plurality of ATSC clients 20 via the transmission path 30, a TTML file is used. It is considered appropriate to operate the subtitle specified by the p element at a timing relative to the time indicated by the begin attribute and the end attribute of the p element in the relative time from the beginning of the content. In order to realize such operation, by setting mode 1 as the TTML processing mode in the MPD file, the ATSC client 20 responds to the time indicated by the begin attribute and end attribute of the p element of the TTML file. At the timing, subtitles will be displayed.

(2) Mode 2: Sample Time Only

  Next, an operation example when mode 2 is set as the TTML processing mode will be described with reference to FIGS. In this mode 2, the time information specified in the MP4 file format (time information for each TTML sample) is used, ignoring the time information specified in the TTML file. Here, mode 2-1 and mode 2-2 will be sequentially described as mode 2.

(2-1) Mode 2-1: Sample Time Only

(Example of MPD file description)
FIG. 13 is a diagram illustrating a description example of an MPD file when the operation is performed in the mode 2-1.

  In the MPD file of FIG. 13, “atsc: ttmlMode: sampleTimeOnly” is specified as the schemeIdUri attribute of the EssentialProperty element in the AdaptationSet element under the Period element of the MPD element that is the root element. That is, the attribute value of the schemeIdUri attribute of the EssentialProperty element indicates that the mode 2-1 is set as the TTML processing mode.

(Example of subtitle display timing)
FIG. 14 is a diagram illustrating an example of subtitle display timing when the operation is performed in the mode 2-1.

  As shown in FIG. 14A, in the MPD file, the start time of streaming distribution is specified in the availabilityStartTime attribute of the MPD element. In the MPD file, Period (1), Period (2), ... are specified as Period elements, and the start time of each Period (period) is specified as a start attribute in those Period elements. You. Here, the sum of the start time specified by the availabilityStartTime attribute and the start time of Period (2) (MPD / @ availabilityStartTime + MPD / Period (2) / @ start) indicates that the start time of Period (2) is Desired.

FIG. 14B schematically shows a TTML segment (Segment) in Period (2).
From the TTML sample (sample) stored in the mdat box of this TTML segment, a TTML file (FIG. 14C) is obtained. In the MPD file (FIG. 13) acquired during this operation, the schemeIdUri attribute of the EssentialProperty element includes Since "atsc: ttmlMode: sampleTimeOnly" is specified and mode 2-1 is set as the TTML processing mode, the time information ("t1" of the begin attribute and "t1" of the end attribute) specified in the p element of the TTML file are set. "t3") is ignored.

  That is, in the mode 2-1, the time information specified in the TTML file is ignored, and the time information (time information for each TTML sample) stored in the moof box of the TTML segment is ignored. Here, it is assumed that one TTML sample is one TTML sample (TTML sample stored in the mdat box) with respect to one moof box (time information stored in the TTML segment).

  Specifically, as shown in FIG. 14, the sum of the start time specified by the availabilityStartTime attribute of the MPD file and the start time of Period (2) (MPD / @ availabilityStartTime + MPD / Period (2) / @ start When the time (BMDT × ts) corresponding to the BMDT stored in the moof box (moof / mfhd / tfdt) elapses with reference to the time at the beginning of Period (2) according to), the p element of the TTML file is used. The display of the subtitle specified by "text1" is started.

  However, BMDT (Base Media Decode Time) represents the time from the start time of Period (2) to the start of decoding TTML samples, that is, an offset. The ts multiplied by the BMDT represents a time scale, and is used to convert the value of the BMDT into a value on the MPD time axis.

  Then, during the time (SampleDuration × ts) corresponding to the SampleDuration stored in the moof box (moof / mfhd / trun), the display of the subtitle “text1” is continued, and the time corresponding to the SampleDuration has elapsed. The display of the subtitle "text1" is terminated.

  Here, SampleDuration represents the duration of the TTML sample. The ts multiplied by SampleDuration is used to convert the value of SampleDuration to a value on the MPD time axis.

  With this, in the ATSC client 20, when a content such as a television program is being played, a time corresponding to the BMDT stored in the moof box has elapsed with reference to the leading time of Period (2). Until the time corresponding to SampleDuration elapses, the subtitle “text1” is superimposed on the video.

  The operation example in the case where the mode 2-1 is set as the TTML processing mode has been described above. This mode 2-1 uses the time information (BMDT and TT for each TTML sample and SampleDuration) specified in the MP4 file format to start displaying subtitles at a time corresponding to the BMDT of the TTML sample. By continuing to display the subtitles only during the appropriate time, the subtitles can be displayed at a desired timing.

(2-2) Mode 2-2: Sample Time Only But Till Next

(Example of MPD file description)
FIG. 15 is a diagram illustrating a description example of the MPD file when the operation is performed in the mode 2-2.

  In the MPD file of FIG. 15, "atsc: ttmlMode: sampleTimeOnlyButTillNext" is specified as the schemeIdUri attribute of the EssentialProperty element in the AdaptationSet element under the Period element of the MPD element which is the root element. That is, the attribute value of the schemeIdUri attribute of the EssentialProperty element indicates that the mode 2-2 is set as the TTML processing mode.

(Example of subtitle display timing)
FIG. 16 is a diagram illustrating an example of subtitle display timing when the operation is performed in the mode 2-2.

  As shown in FIG. 16A, in the MPD file, the start time of streaming distribution is specified in the availabilityStartTime attribute of the MPD element. In the MPD file, Period (1), Period (2), ... are specified as Period elements, and the start time of each Period (period) is specified as a start attribute in those Period elements. You. Here, the sum of the start time specified by the availabilityStartTime attribute and the start time of Period (2) (MPD / @ availabilityStartTime + MPD / Period (2) / @ start) indicates that the start time of Period (2) is Desired.

FIG. 16B schematically illustrates a TTML segment (Segment) in Period (2).
From the TTML sample (sample) stored in the mdat box of this TTML segment, a TTML file (FIG. 16C) is obtained. In the MPD file (FIG. 15) obtained at the time of this operation, the schemeIdUri attribute of the EssentialProperty element includes Since "atsc: ttmlMode: sampleTimeOnlyButTillNext" is specified and mode 2-2 is set as the TTML processing mode, the time information (begin attribute "t1" and end attribute "t1") specified in the p element of the TTML file "t3") is ignored.

  That is, in the mode 2-2, the time information specified in the TTML file is ignored, and the time information (time information for each TTML sample) stored in the moof box of the TTML segment is ignored. Here, it is assumed that one TTML sample is one TTML sample (TTML sample stored in the mdat box) with respect to one moof box (time information stored in the TTML segment).

  Specifically, as shown in FIG. 16, the sum of the start time specified by the availabilityStartTime attribute of the MPD file and the start time of Period (2) (MPD / @ availabilityStartTime + MPD / Period (2) / @ start According to the BMDT stored in the moof box corresponding to the mdat box that stores the target TTML sample (TTML sample of the TTML file that specifies the target subtitle), based on the start time of Period (2) according to) When the elapsed time (BMDT × ts) has elapsed, the display of the subtitle “text1” specified in the p element of the TTML file is started.

  Then, until the time (BMDT × ts) corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the next TTML sample (TTML sample of the TTML file specifying the next subtitle) is elapsed , The display of the subtitle “text1” is continued, and when the time (BMDT × ts) corresponding to the BMDT has elapsed, the display of the subtitle “text1” ends.

  Thereby, in the ATSC client 20, when a content such as a television program is reproduced, the BMDT of the TTML segment (the moof box) including the target TTML sample is set based on the leading time of Period (2). After the elapse of the corresponding time, and until the elapse of the time corresponding to the BMDT of (the moof box of) the TTML segment including the next TTML sample, the subtitle "text1" is superimposed on the video. Will be.

  Although not shown in FIG. 16, when the next TTML sample (a TTML sample of the TTML file specifying the next subtitle) is obtained, the display of the subtitle “text1” is terminated and the next TTML sample is displayed. The display of the subtitle (for example, "text2") specified by the TTML sample (TTML file) is started.

  The operation example in the case where the mode 2-2 is set as the TTML processing mode has been described above. This mode 2-2 uses the time information (BMDT for each TTML sample) specified in the MP4 file format to start displaying subtitles at a time corresponding to the BMDT of the target TTML sample. By continuing the display of the caption until the time corresponding to the BMDT of the TTML sample, the caption can be displayed at a desired timing. The mode 2-1 and the mode 2-2 are common in that the timing of starting the display of subtitles is specified by a time corresponding to the BMDT, but the timing of ending the displayed subtitles is set to SampleDuration. The difference is in whether to specify the time according to the time or the time according to the BMDT.

  For example, when a content provider creates a video and audio package as content, and a broadcaster adds subtitles to the content later, or sets the display timing of the subtitles, etc. When it is more appropriate for operation to specify the subtitle display timing with the time information (time information for each TTML sample) specified in the MP4 file format, rather than specifying the subtitle display timing in the TTML file Is assumed. In order to realize such an operation, the mode 2-1 or the mode 2-2 is set as the TTML processing mode in the MPD file. (Subtitles are displayed at timings according to (time information for each TTML sample)).

(3) Mode 3: Asap

  Next, an operation example when mode 3 is set as the TTML processing mode will be described with reference to FIGS. In this mode 3, immediately after acquiring the TTML file (TTML sample), ignoring the time information specified in the TTML file and the time information (time information for each TTML sample) specified in the mp4 file format, , A process for displaying subtitles is performed.

(Example of MPD file description)
FIG. 17 is a diagram illustrating a description example of an MPD file in the case where operation in mode 3 is performed.

  In the MPD file of FIG. 17, “atsc: ttmlMode: asap” is specified as the schemeIdUri attribute of the EssentialProperty element in the AdaptationSet element under the Period element of the MPD element that is the root element. That is, the attribute value of the schemeIdUri attribute of the EssentialProperty element indicates that the mode 3 is set as the TTML processing mode.

(Example of subtitle display timing)
FIG. 18 is a diagram illustrating an example of subtitle display timing when operation in mode 3 is performed.

  As shown in FIG. 18A, in the MPD file, the start time of streaming distribution is specified in the availabilityStartTime attribute of the MPD element. In the MPD file, Period (1), Period (2), ... are specified as Period elements, and the start time of each Period (period) is specified as a start attribute in those Period elements. You.

  FIG. 18B schematically illustrates a TTML segment (Segment) of Period (2). From the TTML sample (sample) stored in the mdat box of this TTML segment, a TTML file (FIG. 18C) is obtained. In the MPD file (FIG. 17) obtained at the time of this operation, the schemeId Since "atsc: ttmlMode: asap" is specified and mode 3 is set as the TTML processing mode, the time information ("t1" of the begin attribute and "t3" of the end attribute) specified in the p element of the TTML file are set. ") Is ignored. When mode 3 is set as the TTML processing mode, the time information (BMDT and SampleDuration) stored in the moof box of the TTML segment is also ignored.

  That is, in mode 3, the time information specified in the TTML file and the time information for each TTML sample are both ignored, the TTML file (TTML sample) is obtained, and the process is immediately performed. Subtitles are displayed.

  Specifically, as shown in FIG. 18, when the TTML segment is obtained, the ATSC client 20 immediately processes the TTML file when the TTML file is obtained from the TTML sample of the TTML segment. , Ignoring the values of the begin attribute and end attribute of the p element, the display of the subtitle "text1" specified in the p element is started.

  Until the TTML segment including the next TTML file (TTML sample) is obtained, the display of the subtitle "text1" is continued, and the TTML segment including the next TTML file (TTML sample) is obtained. Then, the display of the subtitle "text1" ends.

  Thereby, in the ATSC client 20, when a content such as a television program is being reproduced, a TTML segment including a TTML sample of the target TTML file is obtained, and then a TTML segment including a TTML sample of the next TTML file is obtained. Until is acquired, the subtitle “text1” specified in the p attribute of the target TTML file is superimposed on the video.

  Although not shown in FIG. 18, when the next TTML file is obtained, the display of the subtitle “text1” is terminated, and the subtitle specified by the next TTML file (for example, “text2”) The display of ") starts.

The operation example in the case where the mode 3 is set as the TTML processing mode has been described above.
In mode 3, when the time information specified in the TTML file and the time information specified in the mp4 file format (time information for each TTML sample) are ignored and the TTML file (TTML sample) is obtained, Then, by displaying the subtitle specified in the TTML file, the subtitle can be displayed at a desired timing.

  For example, in the case of distributing live broadcast contents such as sports broadcasts to a plurality of ATSC clients 20 via the transmission path 30 in the ATSC server 10, it is premised that subtitles are delayed from live video. Since it is required to reduce the deviation as much as possible, rather than specifying the display time of the subtitles, it is appropriate to operate the ATSC client 20 to display the subtitles immediately after receiving the TTML file with best effort. It is believed that there is. In order to realize such an operation, by setting mode 3 as the TTML processing mode in the MPD file, the ATSC client 20 displays subtitles immediately after the TTML file is received. become.

<4. Configuration of each device>

  Next, a detailed configuration of each device included in the transmission system 1 of FIG. 1 will be described with reference to FIGS.

(Configuration of ATSC server)
FIG. 19 is a diagram showing a configuration example of the ATSC server 10 of FIG.

  In FIG. 19, the ATSC server 10 includes an AV server 101, a TTML server 102, a DASH server 103, and a broadcast server 104. For example, the AV server 101, the TTML server 102, and the DASH server 103 are provided by a provider such as a content provider or a broadcaster. Further, for example, the broadcast server 104 is provided by a business operator such as a broadcast business operator.

  The AV server 101 acquires (processes) video and audio data of the content, processes the data, and provides the data to the DASH server 103.

  The TTML server 102 generates and processes a TTML file as caption information to be superimposed on the video of the content, and provides it to the DASH server 103.

  The DASH server 103 generates an MPD file. The DASH server 103 processes video and audio data provided from the AV server 101 and a TTML file supplied from the TTML server 102 to generate a segment (segment data). The DASH server 103 provides the MPD file and the segment data to the broadcast server 104.

  The broadcast server 104 generates signaling information such as LLS signaling information and SLS signaling information. Further, the broadcast server 104 transmits segment data (including TTML files) and signaling information (including MPD files) as digital broadcast signals via the antenna 105.

(Detailed configuration of ATSC server)
FIG. 20 is a diagram showing a detailed configuration example of the ATSC server 10 of FIG.

  20, the ATSC server 10 includes a video data acquisition unit 111, a video encoder 112, an audio data acquisition unit 113, an audio encoder 114, a subtitle generation unit 115, a subtitle encoder 116, a signaling generation unit 117, a signaling processing unit 118, and a segment process. It comprises a unit 119, a multiplexer 120, and a transmission unit 121.

  Here, each block for realizing the function of the ATSC server 10 is included in the configuration of any one of the AV server 101 to the broadcast server 104 in FIG. 19. For example, the following configuration is adopted. be able to. That is, in FIG. 20, the video data acquisition unit 111, the video encoder 112, the audio data acquisition unit 113, and the audio encoder 114 are included in the configuration of the AV server 101.

Also, in FIG. 20, the caption generation unit 115 and the caption encoder 116 are included in the configuration of the TTML server 102. 20, the signaling generation unit 117, the signaling processing unit 118, and the segment processing unit 119 are included in the configuration of the DASH server 103.
Further, in FIG. 20, the signaling generation unit 117, the signaling processing unit 118, the multiplexer 120, and the transmission unit 121 are included in the configuration of the broadcast server 104.

  However, the configuration shown in FIG. 20 is an example, and for example, one server having both functions of the AV server 101 and the TTML server 102 may be configured, or both the DASH server 103 and the broadcast server 104 may be configured. The combination of blocks constituting each server is arbitrary, such as configuring one server having a function, and other configurations other than the configuration in FIG. 20 can be adopted.

  The video data acquisition unit 111 acquires content video data from an external server, camera, recording medium, or the like, and supplies the video data to the video encoder 112. The video encoder 112 encodes the video data supplied from the video data acquisition unit 111 according to a predetermined encoding method, and supplies the encoded video data to the segment processing unit 119.

The audio data acquisition unit 113 acquires audio data of the content from an external server, microphone, recording medium, or the like, and supplies the audio data to the audio encoder 114.
The audio encoder 114 encodes the audio data supplied from the audio data acquisition unit 113 in accordance with a predetermined encoding scheme, and supplies the encoded audio data to the segment processing unit 119.

  The caption generation unit 115 generates a TTML file in the TTML format as caption data and supplies the TTML file to the caption encoder 116. The subtitle encoder 116 encodes the subtitle data supplied from the subtitle generation unit 115 according to a predetermined encoding method, and supplies the encoded data to the segment processing unit 119.

  The signaling generation unit 117 generates signaling information and supplies it to the signaling processing unit 118. The signaling processing unit 118 processes the signaling information supplied from the signaling generation unit 117 and supplies the signaling information to the multiplexer 120. Here, for example, LLS signaling information such as SLT metadata, and SLS signaling information such as USD metadata, LSID metadata, and MPD metadata (MPD file) are generated and processed. However, the MPD file includes selection information for selecting the TTML processing mode.

  The segment processing unit 119, based on the video data supplied from the video encoder 112, the audio data supplied from the audio encoder 114, and the subtitle data supplied from the subtitle encoder 116, creates a segment ( Segment data) and supplies it to the multiplexer 120.

  The multiplexer 120 multiplexes the segment data supplied from the segment processing unit 119 and the signaling information supplied from the signaling processing unit 118, and supplies the resulting multiplexed stream to the transmission unit 121.

  The transmitting unit 121 transmits the multiplexed stream supplied from the multiplexer 120 as a broadcast wave of digital broadcasting (digital broadcasting signal) via the antenna 105.

  The ATSC server 10 is configured as described above.

(Detailed configuration of ATSC client)
FIG. 21 is a diagram showing a detailed configuration example of the ATSC client 20 of FIG.

  21, the ATSC client 20 includes a receiving unit 212, a demultiplexer 213, a control unit 214, a memory 215, an input unit 216, a video decoder 217, a video output unit 218, an audio decoder 219, an audio output unit 220, a subtitle decoder 221, It comprises a display unit 222 and a speaker 223. Although the configuration shown in FIG. 21 includes the display unit 222 and the speaker 223, the configuration may not include the display unit 222 and the speaker 223.

  The receiving unit 212 extracts and demodulates a signal corresponding to a user's tuning operation from a broadcast wave (digital broadcast signal) of a digital broadcast received via the antenna 211, and demodulates a multiplexed stream obtained as a result. It is supplied to the demultiplexer 213.

  The demultiplexer 213 separates the multiplexed stream supplied from the receiving unit 212 into audio, video, subtitle streams, and signaling information. The demultiplexer 213 supplies the video data to the video decoder 217, the audio data to the audio decoder 219, the subtitle data to the subtitle decoder 221, and the signaling information to the control unit 214. Note that the video, audio, and subtitle data are segments (segment data) conforming to the MP4 file format.

  The control unit 214 controls the operation of each unit of the ATSC client 20. Further, the control unit 214 controls the operation of each unit to reproduce the content based on the signaling information supplied from the demultiplexer 213.

  The memory 215 is a non-volatile memory such as an NVRAM (Non Volatile RAM), and records various data under the control of the control unit 214. The input unit 216 supplies an operation signal to the control unit 214 according to a user operation.

  The video decoder 217 decodes the video data supplied from the demultiplexer 213 according to a predetermined decoding method, and supplies the video data to the video output unit 218. The video output unit 218 outputs the video data supplied from the video decoder 217 to the display unit 222. As a result, the display unit 222 displays the video of the content according to the user's channel selection operation.

  The audio decoder 219 decodes the audio data supplied from the demultiplexer 213 according to a predetermined decoding method, and supplies the audio data to the audio output unit 220. The audio output unit 220 outputs the audio data supplied from the audio decoder 219 to the speaker 223. As a result, the audio of the content according to the user's channel selection operation is output from the speaker 223.

  The subtitle decoder 221 decodes the subtitle data supplied from the demultiplexer 213 according to a predetermined decoding method, and supplies the decoded data to the video output unit 218. The video output unit 218 causes the caption corresponding to the caption data supplied from the caption decoder 221 to be displayed while being superimposed on the video corresponding to the video data supplied from the video decoder 217. As a result, the display unit 222 displays subtitles superimposed on the video of the content according to the user's channel selection operation.

  The subtitle decoder 221 includes an MP4 parser 241 and a TTML parser 242. The MP4 parser 241 parses the segment data (TTML segment) from the demultiplexer 213, and supplies the resulting TTML file to the TTML parser 242. The TTML parser 242 parses the TTML file supplied from the MP4 parser 241, and supplies information for displaying the resulting subtitle to the video output unit 218.

(Software configuration of ATSC client)
FIG. 22 is a diagram illustrating a software configuration example of the ATSC client 20 of FIG.

  FIG. 22 shows the configuration of the ATSC client 20 shown in FIG. 21 as a software configuration. In FIG. 22, the ATSC client 20 includes a broadcast client middleware 251 and a DASH client 252.

  The broadcast client middleware 251 acquires various data such as segment data (including a TTML file) and signaling information (including an MPD file) transmitted from the ATSC server 10 and provides the DASH client 252 with the acquired data. I do.

  The DASH client 252 processes various data such as segment data (including a TTML file) and signaling information (including an MPD file) provided from the broadcast client middleware 251 to perform processing for reproducing content. For example, the DASH client 252 controls the display of subtitles specified in the TTML file at a display timing according to the TTML processing mode based on the MPD file.

  The ATSC client 20 is configured as described above.

<5. Process flow executed in each device>

  Next, with reference to flowcharts of FIGS. 23 to 26, a flow of processing executed by each device included in the transmission system 1 of FIG. 1 will be described.

(Transmission processing)
First, the transmission processing executed by the ATSC server 10 in FIG. 1 will be described with reference to the flowchart in FIG.

  In step S101, a component signaling process is performed. In this component signaling process, video and audio (stream) data processed by the AV server 101, subtitle data (TTML file) processed by the TTML server 102, and signaling information (MPD File) and signaling information (SLT metadata, USD metadata, LSID metadata, etc.) processed by the broadcast server 104, and component data and signaling information can be transmitted.

  The details of the component signaling processing in step S101 will be described later with reference to the flowchart in FIG.

  In step S102, a transmission process is performed by the broadcast server 104 (the transmission unit 121 and the like), and the video, audio, and subtitle component data and the signaling information processed in the process in step S101 are converted into digital broadcast signals as It is transmitted via the antenna 105. When the processing in step S102 ends, the transmission processing in FIG. 23 ends.

  The transmission processing executed by the ATSC server 10 has been described above.

(Component signaling processing)
Here, the details of the component signaling process corresponding to the process of step S101 in FIG. 23 will be described with reference to the flowchart in FIG. In FIG. 24, for simplification of the description, the processing performed by the AV server 101 is omitted, and the processing performed by the TTML server 102, the DASH server 103, and the broadcast server 104 will be mainly described.

  In step S111, (the caption generation unit 115 of) the TTML server 102 generates a TTML file.

  In step S112, the TTML server 102 (the subtitle encoder 116 thereof) stores the TTML file generated in the process of step S111 in the MP4 file format.

  In step S113, the TTML server 102 requests the DASH server 103 to generate a segment (TTML segment) of the TTML file stored in the MP4 file format.

  Although the processing performed by the AV server 101 is omitted here, the AV server 101 also stores video and audio data in the MP4 file format, and sends a segment generation request to the DASH server 103. Will be done.

  In step S121, the DASH server 103 acquires a segment generation request from the TTML server 102 (and the AV server 101).

  In step S122, (the signaling generation unit 117 of) the DASH server 103 generates an MPD file. Here, in the MPD file, as selection information for selecting the TTML processing mode, the value of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the schemeIdUri attribute of the SupplementalProperty element is set to mode 1, mode 2-1, mode 2-2, or mode. A character string for identifying 3 is specified.

  In step S123, the DASH server 103 (segment processing unit 119 thereof) uses the video and audio (stream) data from the AV server 101 and the TTML file from the TTML server 102 to generate a segment conforming to the MP4 file format. Generate (segment data).

  In step S124, the DASH server 103 transfers the MPD file generated in step S122 and the segment data generated in step S123 to the broadcast server 104.

  In step S131, the broadcast server 104 acquires the MPD file and the segment data from the DASH server 103.

  In step S132, (the signaling generation unit 117 of) the broadcast server 104 generates signaling information such as LLS signaling information and SLS signaling information.

  When the process in step S132 ends, the process returns to the process in step S101 in FIG. 23, and the subsequent processes are executed. That is, in the broadcast server 104 (the transmission unit 121 and the like), the process of step S102 (FIG. 23) described above is performed, and the segment data (including the TTML file) and the signaling information (including the MPD file) are transmitted to the digital broadcast signal. Sent as

  The component signaling processing executed by the ATSC server 10 has been described above. In this component signaling process, various processes are performed so that the ATSC client 20 can reproduce the content using the component data and the signaling information. Also, here, when a caption is superimposed on a video of a content, an MPD file including selection information for selecting a TTML processing mode is generated and transmitted together with the TTML file.

(Reception processing)
Next, the reception processing executed by the ATSC client 20 in FIG. 1 will be described with reference to the flowchart in FIG. The reception process in FIG. 25 is executed, for example, when the user performs a channel selection operation of a desired service.

  In step S201, reception processing is performed by the reception unit 212 and the like, and a digital broadcast signal transmitted from the ATSC server 10 via the transmission path 30 is received via the antenna 211.

  In step S202, a component signaling process is performed. In the component signaling processing, processing of video, audio, subtitle component data and signaling information obtained from the digital broadcast signal received in the processing of step S201 is performed, and the content according to the user's channel selection operation is performed. Is played.

  The details of the component signaling processing in step S202 will be described later with reference to the flowchart in FIG. When the processing in step S202 ends, the reception processing in FIG. 25 ends.

  The reception processing executed by the ATSC client 20 has been described above.

(Component signaling processing)
Here, the details of the component signaling process corresponding to the process of step S202 in FIG. 25 will be described with reference to the flowchart in FIG. FIG. 26 shows the processing performed by the broadcast client middleware 251 and the DASH client 252 of FIG.

  In step S211, the broadcast client middleware 251 acquires an MPD file. In step S212, the broadcast client middleware 251 acquires segment data.

  In step S213, the broadcast client middleware 251 transfers the MPD file obtained in step S211 and the segment data obtained in step S212 to the DASH client 252.

  In step S221, the DASH client 252 acquires the MPD file and the segment data transferred from the broadcast client middleware 251.

  In step S222, the DASH client 252 parses the MPD file obtained in the processing in step S221.

  In step S223, the DASH client 252 renders the video and audio segment data obtained in step S221 based on the result of parsing the MPD file in step S222. As a result, the video of the content is displayed on the display unit 222, and the sound is output from the speaker 223.

  In step S224, the DASH client 252 checks the value (attribute value) of the schemeIdUri attribute of the EssentialProperty element of the AdaptationSet element corresponding to the subtitle based on the result of parsing the MPD file in the process of step S222.

  In step S224, when it is determined that “ttmlTimeOnly” is specified as the value of the schemeIdUri attribute of the EssentialProperty element, mode 1 is set as the TTML processing mode, and the process proceeds to step S225. Steps S225 to S227 are executed.

  In step S225, the DASH client 252 parses the segment data (TTML segment) acquired in the processing in step S221. However, in mode 1, time information specified in the MP4 file format, that is, time information such as BMDT and SampleDuration stored in the moof box is ignored.

  In step S226, the DASH client 252 parses the TTML segment in the process of step S225, thereby parsing the TTML file obtained from the TTML sample stored in the mdat box. Here, in mode 1, the time information specified by the TTML file, that is, the time information specified by the begin attribute and end attribute of the p element in the body element is respected, and the subtitle is added at the time specified by the begin attribute. Is started, and the display of the subtitle is ended at the time specified by the end attribute.

  In step S227, the DASH client 252 performs a rendering process based on the result of parsing the TTML file in the process in step S226, and performs a rendering process between the display start time of the begin attribute and the display end time of the end attribute. To display subtitles as a character string specified by the p element.

  As described above, when the mode 1 is set as the TTML processing mode, the time information specified in the MP4 file format is ignored, and the time information specified in the TTML file is respected. To display subtitles.

  If it is determined in step S224 that "sampleTimeOnly" is specified as the value of the schemeIdUri attribute of the EssentialProperty element, the mode is set to the mode 2-1 as the TTML processing mode. And the processing of steps S228 to S230 is executed.

  In step S228, the DASH client 252 parses the segment data (TTML segment) acquired in the processing in step S221. Here, in the mode 2-1, the time information specified in the MP4 file format, that is, the time information such as the BMDT and the SampleDuration stored in the moof box is respected, and the subtitle is displayed at the time corresponding to the BMDT. After starting, the display is continued only for a time corresponding to the SampleDuration stored in the moof box.

  In step S229, the DASH client 252 parses the TTML segment in the process of step S228, thereby parsing the TTML file obtained from the TTML sample stored in the mdat box. However, in the mode 2-1, the time information specified by the TTML file, that is, the time information specified by the begin attribute and the end attribute of the p element is ignored.

  In step S230, the DASH client 252 performs a rendering process based on the parsed results of the processes of step S228 and step S229, and performs the p-time processing of the TTML file between the time corresponding to the BMDT and the time corresponding to the SampleDuration. Subtitles are displayed as character strings specified by elements.

  Further, in step S224, when it is determined that “sampleTimeOnlyButTillNext” is specified as the value of the schemeIdUri attribute of the EssentialProperty element, mode 2-2 is set as the TTML processing mode. And the processing of steps S231 to S233 is executed.

  In step S231, the DASH client 252 parses the segment data (TTML segment) acquired in the processing in step S221. Here, in the mode 2-2, the time information specified in the MP4 file format, that is, the time information of the BMDT stored in the moof box is respected, and the time corresponding to the BMDT corresponding to the target TTML sample is set. The display of the caption is started, and the display is continued until the time corresponding to the BMDT corresponding to the next TTML sample.

  In step S232, the DASH client 252 parses the TTML segment in the process of step S231 to parse the TTML file obtained from the TTML sample stored in the mdat box. However, in the mode 2-2, the time information specified by the TTML file, that is, the time information specified by the begin attribute and the end attribute of the p element is ignored.

  In step S233, the DASH client 252 performs a rendering process based on the parsed results of the processes of step S231 and step S232, and responds to the next TTML sample from the time corresponding to the BMDT corresponding to the target TTML sample. The subtitle as a character string specified by the p element of the TTML file obtained from the target TTML sample is displayed until the time corresponding to the BMDT to be performed.

  In order to recognize the time corresponding to the BMDT corresponding to the next TTML sample, it is necessary to return to the above-described processing of step S231 and parse the TTML segment. That is, by repeating the processing of steps S231 to S233, the time corresponding to the BMDT corresponding to the next TTML sample is recognized, and the subtitle specified in the TTML file obtained from the next TTML sample is displayed. Will be.

  As described above, when mode 2 (mode 2-1 and mode 2-2) is set as the TTML processing mode, the time information (time information for each TTML sample) specified in the MP4 file format is respected. By ignoring the time information specified in the TTML file, subtitles can be displayed at a desired timing.

  If it is determined in step S224 that “asap” is specified as the value of the schemeIdUri attribute of the EssentialProperty element, mode 3 is set as the TTML processing mode, and the process proceeds to step S234. Then, the processing of steps S234 to S236 is executed.

  In step S234, the DASH client 252 parses the segment data (TTML segment) acquired in the processing in step S221. However, in mode 3, time information specified in the MP4 file format, that is, time information such as BMDT and SampleDuration stored in the moof box is ignored.

  In step S235, the DASH client 252 parses the TTML segment in the process of step S234 to parse the TTML file obtained from the TTML sample stored in the mdat box. However, in mode 3, the time information specified in the TTML file, that is, the time information specified by the begin attribute and end attribute of the p element is ignored.

  In step S236, the DASH client 252 immediately renders the TTML file based on the result of parsing the TTML file in the processing in step S235, and displays subtitles as a character string specified by the p element of the TTML file. To do. The display of the captions displayed in this manner is terminated when the next TTML file (TTML sample) is obtained.

  As described above, when the mode 3 is set as the TTML processing mode, the time information specified in the MP4 file format (time information for each TTML sample) and the time information specified in the TTML file are ignored. By displaying the subtitles immediately, the subtitles can be displayed at a desired timing.

  The component signaling processing executed by the ATSC client 20 has been described above. In this component signaling process, a process for reproducing content is performed using component data and signaling information transmitted from the ATSC server 10. In addition, here, when the caption is superimposed on the video of the content, the MPD file including the selection information for selecting the TTML processing mode is acquired, so the TTML file is displayed at the display timing according to the TTML processing mode. Will be displayed.

<6. Modification>

  As described above, ATSC (for example, ATSC3.0), which is a system adopted in the United States and the like, has been described as a standard for digital broadcasting.However, this technology is based on ISDB (Integrated Services Digital Broadcasting) or DVB (Digital Video Broadcasting) which is a method adopted by European countries.

  Further, the name of the signaling information such as the SLT described above is an example, and another name may be used. Even if another name is used as the name of the signaling information, the name is merely changed formally, and the substantial content of the signaling information is not different. For example, SLT may be referred to as FIT (Fast Information Table).

  In the above description, the TTML file and the MPD file are described as being transmitted by the ATSC server 10 as digital broadcast signals. However, these files may be distributed from a server on the Internet. For example, a caption of a TTML file distributed via communication may be superimposed on a video of a content distributed via broadcasting. Also, video (audio) data may be adaptively streamed from a server on the Internet. However, this streaming distribution conforms to the MPEG-DASH standard.

  Further, in the above description, in order to specify the TTML processing mode, in the MPD file, the case where the attribute value of the SchemeIdUri attribute of the EssentialProperty element of the AdaptationSet element or the SupplementalProperty element is used, but other elements and attributes are used. Thus, the TTML processing mode may be designated. In the Representation element or SubRepresentation element, the TTML processing mode may be specified by the attribute value of the schemeIdUri attribute of the EssentialProperty element or the SupplementalProperty element. Further, if the TTML processing mode is recognized at the time of processing the TTML file, the TTML processing mode may be designated by signaling information other than the MPD file.

<7. Computer Configuration>

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in a computer. FIG. 27 is a diagram illustrating a configuration example of hardware of a computer that executes the series of processes described above by a program.

  In the computer 900, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are mutually connected by a bus 904. An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a recording unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

  The input unit 906 includes a keyboard, a mouse, a microphone, and the like. The output unit 907 includes a display, a speaker, and the like. The recording unit 908 includes a hard disk, a non-volatile memory, and the like. The communication unit 909 includes a network interface and the like. The drive 910 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer 900 configured as described above, the CPU 901 loads a program recorded in the ROM 902 or the recording unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executes the program, thereby performing the above-described operation. A series of processing is performed.

  The program executed by the computer 900 (CPU 901) can be provided by being recorded on, for example, a removable medium 911 as a package medium or the like. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer 900, the program can be installed in the recording unit 908 via the input / output interface 905 by attaching the removable medium 911 to the drive 910. Further, the program can be received by the communication unit 909 via a wired or wireless transmission medium and installed in the recording unit 908. In addition, the program can be installed in the ROM 902 or the recording unit 908 in advance.

  Here, in the present specification, processing performed by a computer according to a program does not necessarily need to be performed in a time series in the order described in the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or individually (for example, parallel processing or processing by an object). Further, the program may be processed by one computer (processor) or may be processed in a distributed manner by a plurality of computers.

  Note that embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

  Further, the present technology can have the following configurations.

(1)
A receiving unit for receiving broadcast waves of digital broadcasting,
An acquisition unit configured to acquire subtitle information related to subtitles transmitted by the broadcast wave and control information including selection information for selecting a specific mode among a plurality of modes for specifying a timing of displaying the subtitles; When,
A control unit that controls display of the caption according to the caption information at a display timing according to the specific mode based on the selection information included in the control information.
(2)
The subtitle information is a TTML file in TTML (Timed Text Markup Language) format, and the data is in conformity with the MP4 file format.
The control information is an MPD (Media Presentation Description) file in XML (Extensible Markup Language) format,
The receiving device according to (1), wherein the TTML file and the MPD file are transmitted in a ROUTE (Real-Time Object Delivery over Unidirectional Transport) session.
(3)
The plurality of modes include a first mode for displaying the subtitle at a timing according to time information specified in the TTML file,
The control unit, when the specific mode is the first mode, displays the subtitle specified in the TTML file at a timing according to time information specified in the TTML file (2) The receiving device according to the above.
(4)
The plurality of modes include a second mode for displaying the subtitle at a timing according to time information defined by the MP4 file format,
When the specific mode is the second mode, the control unit displays the subtitle specified by the TTML file at a timing according to time information defined by the MP4 file format. Or the receiving device according to (3).
(5)
The control unit starts displaying the subtitles at a time corresponding to a BMDT (BaseMediaDecodeTime) stored in a moof box defined by the MP4 file format, and a time corresponding to a SampleDuration stored in the moof box. The receiving device according to (4), wherein the display is continued only during the period.
(6)
The control unit starts the display of the subtitles at a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the data of the target subtitle, which is defined in the MP4 file format, and The display according to (4), wherein the display is continued until a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the caption data of the subtitle.
(7)
The plurality of modes include a time information specified in the TTML file and a third mode in which the subtitle is displayed ignoring time information defined in the MP4 file format,
The control unit, when the specific mode is the third mode, immediately displays the subtitle specified by the TTML file when the TTML file is obtained (2) to (4). The receiving device according to any one of the above.
(8)
The receiving device according to any one of (2) to (7), wherein the selection information is specified as extended information of the MPD file.
(9)
The receiving device according to (8), wherein the selection information is specified by a schemeIdUri attribute of an EssentialProperty element or a SupplementalProperty element in an AdaptationSet element arranged in a Period element of an MPD element.
(10)
Obtain control information including caption information related to subtitles, transmitted by digital broadcast waves, and selection information for selecting a specific mode among a plurality of modes for specifying the timing of display of the subtitles. ,
A data processing method, comprising: controlling display of the subtitles according to the subtitle information at a display timing according to the specific mode based on the selection information included in the control information.
(11)
A generation unit that generates control information including selection information for selecting a specific mode among a plurality of modes for specifying the timing of subtitle display;
A transmitting unit that transmits the control information together with the caption information related to the caption in a broadcast wave of a digital broadcast.
(12)
The subtitle information is a TTML file in the TTML format, and the data conforms to the MP4 file format.
The control information is an MPD file in XML format,
The transmission device according to (11), wherein the TTML file and the MPD file are transmitted in a ROUTE session.
(13)
The transmission device according to (12), wherein the plurality of modes include a first mode in which the caption is displayed at a timing according to time information specified in the TTML file.
(14)
The transmission device according to (12) or (13), wherein the plurality of modes include a second mode in which the subtitle is displayed at a timing according to time information defined by the MP4 file format.
(15)
The second mode starts displaying the subtitles at a time corresponding to the BMDT stored in the moof box defined by the MP4 file format, and sets a time corresponding to the SampleDuration stored in the moof box. The transmission device according to (14), wherein the mode is a mode in which display is continued only for a while.
(16)
The second mode starts displaying the subtitle at a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the data of the target subtitle, which is defined by the MP4 file format. The transmission device according to (14), wherein the display is continued until a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the data of the next subtitle.
(17)
The plurality of modes include a third mode in which the subtitle is displayed while ignoring time information specified in the TTML file and time information specified in the MP4 file format. (12) to (14) The transmission device according to any one of the above.
(18)
The transmission device according to any one of (12) to (17), wherein the selection information is specified as extended information of the MPD file.
(19)
The transmission device according to (18), wherein the selection information is specified by a schemeIdUri attribute of an EssentialProperty element or a SupplementalProperty element in an AdaptationSet element arranged in a Period element of an MPD element.
(20)
Generating control information including selection information for selecting a specific mode among a plurality of modes for specifying the display timing of the subtitle, which is transmitted together with the subtitle information about the subtitle in a broadcast wave of a digital broadcast; A data processing method including:

  Reference Signs List 1 transmission system, 10 ATSC server, 20 ATSC client, 30 transmission line, 101 AV server, 102 TTML server, 103 DASH server, 104 broadcast server, 111 video data acquisition unit, 112 video encoder, 113 audio data acquisition unit, 114 audio Encoder, 115 subtitle generation unit, 116 subtitle encoder, 117 signaling generation unit, 118 signaling processing unit, 119 segment processing unit, 120 multiplexer, 121 transmission unit, 212 reception unit, 213 demultiplexer, 214 control unit, 217 video decoder, 218 Video output unit, 219 audio decoder, 220 audio output unit, 221 subtitle decoder, 241 MP4 parser, 242 TTML parser, 2 1 broadcast client middleware, 252 DASH client, 900 computer, 901 CPU

A receiving device according to a first aspect of the present technology obtains a receiving unit that receives a broadcast wave of digital broadcasting, caption information related to subtitles transmitted by the broadcast wave, and control information for displaying the subtitles. an acquisition unit, based on the control information, the a control unit for controlling the display of the subtitles according to subtitle information, the control information, the plurality of modes for specifying the timing of the display of the subtitles Selection information for selecting a specific mode among them, including role information indicating that the control target is subtitles, the control unit, based on the selection information and the role information included in the control information, The receiving device controls display of the caption according to the caption information at a display timing according to the specific mode .

In the receiving device and the data processing method according to the first aspect of the present technology, caption information on subtitles transmitted by digital broadcast waves and control information for displaying the subtitles are acquired, and the control information includes based on the display of the subtitle in accordance with the subtitle information that are controlled. Further, the control information includes selection information for selecting a specific mode among a plurality of modes for specifying the timing of displaying the subtitle, and role information indicating that the control target is the subtitle. The display of the caption according to the caption information is controlled at a display timing according to the specific mode based on the selection information and the role information included in the control information.

The transmission device according to the second aspect of the present technology includes a generation unit that generates control information for displaying subtitles, and a transmission unit that transmits the control information , together with subtitle information related to the subtitles, in a digital broadcast wave. The control information includes selection information for selecting a specific mode among a plurality of modes for designating the display timing of the caption, and role information indicating that the control target is the caption It is a transmitting device.

In the transmitting apparatus and the data processing method of the second aspect of the present technology, the control information for displaying subtitles is created, with the subtitle information on the caption, wherein the control information is transmitted by broadcast wave of digital broadcasting . Further, the control information includes selection information for selecting a specific mode among a plurality of modes for specifying the timing of displaying the subtitle, and role information indicating that the control target is the subtitle. It is.

Claims (20)

A receiving unit for receiving broadcast waves of digital broadcasting,
An acquisition unit configured to acquire subtitle information related to subtitles transmitted by the broadcast wave and control information including selection information for selecting a specific mode among a plurality of modes for specifying a timing of displaying the subtitles; When,
A control unit that controls display of the caption according to the caption information at a display timing according to the specific mode based on the selection information included in the control information. The subtitle information is a TTML file in TTML (Timed Text Markup Language) format, and the data is in conformity with the MP4 file format.
The control information is an MPD (Media Presentation Description) file in XML (Extensible Markup Language) format,
The receiving device according to claim 1, wherein the TTML file and the MPD file are transmitted in a Real-Time Object Delivery over Unidirectional Transport (ROUTE) session. The plurality of modes include a first mode for displaying the subtitle at a timing according to time information specified in the TTML file,
The control unit, when the specific mode is the first mode, displays the subtitle specified in the TTML file at a timing according to time information specified in the TTML file. The receiving device according to the above. The plurality of modes include a second mode for displaying the subtitle at a timing according to time information defined by the MP4 file format,
The control unit, when the specific mode is the second mode, displays the subtitle specified by the TTML file at a timing according to time information defined by the MP4 file format. 3. The receiving device according to 2. The control unit starts displaying the subtitles at a time corresponding to a BMDT (BaseMediaDecodeTime) stored in a moof box defined by the MP4 file format, and a time corresponding to a SampleDuration stored in the moof box. The receiving device according to claim 4, wherein the display is continued only during the period. The control unit, specified in the MP4 file format, starts displaying the subtitles at a time corresponding to the BMDT stored in the moof box corresponding to the mdat box that stores target subtitle data, and The receiving device according to claim 4, wherein the display is continued until a time corresponding to the BMDT stored in the moof box corresponding to the mdat box that stores the data of the caption. The plurality of modes include a time information specified in the TTML file and a third mode in which the subtitle is displayed ignoring time information defined in the MP4 file format,
The reception unit according to claim 2, wherein, when the specific mode is the third mode, the control unit causes the caption specified by the TTML file to be displayed immediately upon acquiring the TTML file. apparatus. The receiving device according to claim 2, wherein the selection information is specified as extended information of the MPD file. The receiving device according to claim 8, wherein the selection information is specified by a schemeIdUri attribute of an EssentialProperty element or a SupplementalProperty element in an AdaptationSet element arranged in a Period element of an MPD element. Obtain control information including caption information related to subtitles, transmitted by digital broadcast waves, and selection information for selecting a specific mode among a plurality of modes for specifying the timing of display of the subtitles. ,
A data processing method comprising: controlling display of the subtitles according to the subtitle information at a display timing according to the specific mode based on the selection information included in the control information. A generation unit that generates control information including selection information for selecting a specific mode among a plurality of modes for specifying the timing of display of subtitles;
A transmitting unit that transmits the control information together with the caption information related to the caption in a broadcast wave of a digital broadcast. The subtitle information is a TTML file in the TTML format, and the data conforms to the MP4 file format.
The control information is an MPD file in XML format,
The transmission device according to claim 11, wherein the TTML file and the MPD file are transmitted in a ROUTE session. The transmission device according to claim 12, wherein the plurality of modes include a first mode in which the subtitle is displayed at a timing according to time information specified in the TTML file. The transmitting device according to claim 12, wherein the plurality of modes include a second mode in which the subtitle is displayed at a timing according to time information defined by the MP4 file format. The second mode starts displaying the subtitles at a time corresponding to the BMDT stored in the moof box defined by the MP4 file format, and sets a time corresponding to the SampleDuration stored in the moof box. The transmission device according to claim 14, wherein the mode is a mode in which display is continued only for a while. The second mode starts displaying the subtitles at a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing the data of the target subtitle, which is defined by the MP4 file format. The transmitting apparatus according to claim 14, wherein the mode is a mode in which display is continued until a time corresponding to the BMDT stored in the moof box corresponding to the mdat box storing data of the next subtitle.   The method according to claim 12, wherein the plurality of modes include a third mode in which the subtitle is displayed ignoring time information specified in the TTML file and time information specified in the MP4 file format. Transmission device. The transmission device according to claim 12, wherein the selection information is specified as extended information of the MPD file. The transmitting device according to claim 18, wherein the selection information is specified by a schemeIdUri attribute of an EssentialProperty element or a SupplementalProperty element in an AdaptationSet element arranged in a Period element of an MPD element. Generating control information including selection information for selecting a specific mode among a plurality of modes for specifying the display timing of the subtitles, which is transmitted together with the subtitle information about the subtitles in a broadcast wave of a digital broadcast. A data processing method including:

Images