JP4849343B2 - Data generation method, recording apparatus and method, and program - Google Patents

Description

The present invention relates to a data generation method , a recording apparatus and method, and a program, and more particularly, to a data generation method , a recording apparatus and method, and a program that are suitable for reproducing stream data.

In order to view different contents simultaneously, after decoding video data of multiple contents, uncompressed image data is synthesized, then D / A conversion is performed and output to the video output terminal. There is a technique for displaying on a display device (for example, Patent Document 1).
Japanese Patent Laying-Open No. 2005-20242

  As described above, not only when video data of a plurality of contents are combined and displayed, but also when a single content includes a plurality of video data, a plurality of video data can be combined and output. It has been demanded.

  When a plurality of pieces of video data are combined and output, the display method of the display screen that is the sub-screen with respect to the main display screen does not depend on the playback device, for example, for each content or for predetermined playback of the content It has been difficult for the content creator or content distributor to set for each location.

  The present invention has been made in view of such a situation, and makes it possible to specify a display method of a display screen that is a sub-screen with respect to a main display screen.

The data generation method of the present invention is a data generation method for generating playback management data used for content playback in a playback device, and acquires a video of a main stream and a video of a substream, and corresponds to the main stream. Acquires the audio to be played back, and if there is audio to be played back corresponding to the substream, it is played back corresponding to the substream, which is synthesized with the audio played back corresponding to the main stream. First information indicating a position on the time axis for acquiring the audio to be played and reproducing the main stream; second information indicating a position on the time axis for reproducing a substream different from the main stream; an information about the display state of the image of the sub-stream is combined with the picture of the main stream, movies of the main stream When executed on the basis of the information indicating the brightness of the image of the sub-streams are combined, the image of the main stream, and the information about the display state containing information about the process for synthesizing the video of the sub stream Generating reproduction management data having a data structure including the same.

  The information indicating the brightness of the video of the substream is at least one threshold value of information indicating the brightness to be subjected to the transmission process, and the video portion in which the brightness of the video of the substream is greater than or less than the threshold value Can be subjected to a transparent process.

In the data generation method of the present invention, the main stream video and the sub stream video are acquired, the audio reproduced corresponding to the main stream is acquired, and the audio reproduced corresponding to the sub stream exists. In this case, the first information indicating the position on the time axis at which the audio to be reproduced corresponding to the sub-stream, which is synthesized with the audio to be reproduced corresponding to the main stream, is acquired and the main stream is reproduced is obtained. Second information indicating a position on the time axis for reproducing a substream different from the main stream, and information on a display state of the substream video to be synthesized with the mainstream video , the mainstream video The main stream video and the sub stream are executed based on information indicating the brightness of the sub stream video to be combined with Reproduction management data is generated with a data structure containing information about the display state containing information about the process of synthesizing and over arm video.

The recording apparatus of the present invention is a first acquisition for acquiring a main stream video and a substream video in a recording apparatus for recording data on a recording medium on which playback management data used for content playback is recorded in the playback apparatus. Means, a second acquisition means for acquiring audio to be played back corresponding to the main stream, and playback corresponding to the main stream if there is audio to be played back corresponding to the substream. Third acquisition means for acquiring the audio reproduced corresponding to the substream, synthesized with the audio to be reproduced, first information indicating a position on the time axis for reproducing the main stream, and the main Second information indicating a position on the time axis for reproducing a substream different from the stream, and the substream to be combined with the video of the main stream. An information about the display state of the image of streams, said executed based on the information indicating the brightness of the image of the sub-stream is combined with the picture of the main stream, and the video of the main stream, the sub-stream Generating means for generating reproduction management data having a data structure including information relating to the display state including information relating to processing for synthesizing video; and the acquired video and audio of the main stream, and the acquired sub-stream And recording means for recording the video and audio or the video of the substream and the generated reproduction management data on the recording medium.
The information indicating the brightness of the video of the substream is at least one threshold value of information indicating the brightness to be subjected to the transmission process, and the video portion in which the brightness of the video of the substream is greater than or less than the threshold value Can be subjected to a transparent process.

The recording method of the present invention is a recording method of a recording medium on which playback management data used for content playback in a playback device is recorded, and acquires a main stream video and a sub stream video, and corresponds to the main stream If there is audio to be reproduced corresponding to the substream, the audio corresponding to the substream is combined with the audio to be reproduced corresponding to the main stream. The first information indicating the position on the time axis for acquiring the audio to be reproduced and reproducing the main stream, and the second information indicating the position on the time axis for reproducing a substream different from the main stream When the a information about the display state of the image of the sub-stream is combined with the picture of the main stream, film of the mainstream When executed on the basis of the information indicating the brightness of the image of the sub-streams are combined, the image of the main stream, and the information about the display state containing information about the process for synthesizing the video of the sub stream Generating the reproduction management data having a data structure including the acquired video and audio of the main stream, the acquired video and audio of the substream or the video of the substream, and the generated reproduction management data Recording on a recording medium.

The program of the present invention is a program for causing a computer to execute a process of recording data on a recording medium on which playback management data used for content playback in a playback device is recorded, and a main stream video and a substream video And audio to be played back corresponding to the main stream, and if there is audio to be played back corresponding to the substream, the audio played back corresponding to the main stream and The first information indicating the position on the time axis for obtaining the synthesized audio to be reproduced corresponding to the substream and reproducing the main stream, and the time for reproducing the substream different from the main stream Second information indicating a position on the axis, and video of the substream combined with the video of the main stream An information about the display state, synthetic wherein is performed based on the information indicating the brightness of the image of the sub-stream is combined with the picture of the main stream, and the video of the main stream, the video of the sub stream Generating reproduction management data having a data structure including information regarding the display state including information regarding processing to be performed, and acquiring the video and audio of the main stream acquired, video and audio of the substream acquired, or the substream And recording the generated reproduction management data on the recording medium.

In the recording apparatus, recording method, and program of the present invention, a main stream video and a sub stream video are acquired, audio to be played back corresponding to the main stream is acquired, and played back corresponding to the sub stream If there is audio to be played back, the audio to be played back corresponding to the substream that is synthesized with the audio played back corresponding to the main stream is acquired, and the position on the time axis for playing back the main stream is determined. a first information indicating a second information indicating a position on the time axis for reproducing a different sub-stream from the main stream, an information about the display state of the image of the sub-stream to be combined with the picture of the main stream The main stream is executed based on information indicating the brightness of the sub-stream video to be combined with the main-stream video. And image beam, the sub-stream of the reproduction management data having a data structure containing information about the display state containing information about the process for synthesizing the images are generated, the video and audio of the acquired mainstream obtained sub Stream video and audio or sub-stream video and the generated reproduction management data are recorded on a recording medium.

  As described above, according to the present invention, the video of the main stream file and the video of the substream file can be combined, and in particular, information regarding the display state of the video of the substream file included in the playback management information is obtained. Based on this, the video of the main stream file and the video of the substream file can be synthesized. For example, the size and display position of the sub display screen in picture-in-picture display do not depend on the specifications of the playback device. The content creator or the content distributor can be determined as appropriate.

  Further, according to the present invention, it is possible to provide data having a data structure including reproduction management information that is information for managing reproduction of a main stream file including at least one stream. First information including a main reproduction path indicating a position on the time axis of the stream file, second information including a sub reproduction path of a substream file different from the main stream file, video of the main stream file, and Since it is possible to provide data having a data structure including information related to the display state of the video of the substream file to be combined, for example, the size and display position of the sub display screen in picture-in-picture display are the specifications of the playback device. Without depending on the content creator or content distributor, It can be determined Yichun.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing an example of an application format on a recording medium mounted on a playback apparatus 20 (described later with reference to FIG. 35) to which the present invention is applied. The recording medium may be a magnetic disk or a semiconductor memory in addition to the optical disk described later.

  The application format has two layers, PlayList (playlist) and Clip (clip), for managing AV (Audio Visual) streams. Here, a pair of one AV stream and Clip information as information accompanying the AV stream is considered as one object, and these are collectively referred to as a Clip. Hereinafter, the AV stream is also referred to as an AV stream file. Clip information is also referred to as a Clip information file.

  In general, a file used in a computer or the like is handled as a byte string, but the content of an AV stream file is expanded on the time axis, and an access point of a Clip is mainly designated by a PlayList with a time stamp. That is, PlayList and Clip are layers for AV stream management.

  When the access point in the Clip is indicated by the PlayList with a time stamp, the Clip Information file is used to find address information to start decoding in the AV stream file from the time stamp.

  The PlayList is a collection of AV stream playback sections. One playback section in a certain AV stream is called PlayItem, which is represented by a pair of an IN point (playback start point) and an OUT point (playback end point) of the playback section on the time axis. Accordingly, the PlayList is composed of one or a plurality of PlayItems as shown in FIG.

  In FIG. 1, the first PlayList from the left is composed of two PlayItems, and the two PlayItems refer to the first half and the second half of the AV stream included in the left Clip, respectively. Also, the second PlayList from the left is composed of one PlayItem, whereby the entire AV stream included in the right Clip is referenced. Further, the third PlayList from the left is composed of two PlayItems, and the two PlayItems refer to a portion of the AV stream included in the left Clip and a portion of the AV stream included in the right Clip, respectively. ing.

  For example, when the disc navigation program shown in FIG. 1 specifies the left PlayItem included in the first PlayList from the left as the information indicating the playback position at that time, it is included in the left Clip referred to by the PlayItem. The first half of the AV stream is played back. Thus, the PlayList is used as playback management information for managing playback of AV stream files.

  The disc navigation program has a function of controlling the playback order of the PlayList and interactive playback of the PlayList. The disc navigation program also has a function of displaying a menu screen for the user to instruct execution of various reproductions. This disc navigation program is described in a programming language such as Java (registered trademark), and is prepared on a recording medium.

  In this embodiment, a playback path created by a sequence of one or more PlayItems (by successive PlayItems) in the PlayList is called a main path (Main Path), and is parallel (in parallel) to the Main Path in the PlayList. ) And a reproduction path created by a sequence of one or more SubPlayItems (by subplay items that may be discontinuous or continuous) is referred to as a sub path. That is, the application format on the recording medium mounted on the playback device 20 (described later with reference to FIG. 35) includes a sub path (Sub Path) to be played back in association with the main path in the PlayList. Have.

  FIG. 2 is a diagram illustrating the structure of the main path and the sub path. The PlayList can have one main path and one or more sub paths. One main path is created by a sequence of one or more PlayItems, and one sub-path is created by a sequence of one or more SubPlayItems.

  In the case of the example of FIG. 2, the PlayList has one main path made up of three PlayItems and three sub paths. Each PlayItem constituting the main path is assigned ID (Identification) in order from the top. Specifically, the main path is made up of PlayItems with PlayItem_id = 0, PlayItem_id = 1, and PlayItem_id = 2. Subpaths are also assigned IDs in order from the top, Subpath_id = 0, Subpath_id = 1, and Subpath_id = 2. A subpath of Subpath_id = 0 includes one SubPlayItem, a subpath of Subpath_id = 1 includes two SubPlayItems, and a subpath of Subpath_id = 2 includes one SubPlayItem.

  The stream referred to by the SubPlayItem included in the subpath of Subpath_id = 0 is assumed to be, for example, a Japanese dubbing sound of a movie, and may be played in place of the audio stream of the AV stream file referenced by Mainpath. Also, the stream referred to by SubPlayItem included in the subpath of Subpath_id = 1 is assumed to be a director's cut of a movie, for example, and a comment such as a movie director is included only in a predetermined part of the AV stream file referred to by Main Path. The case is considered.

  A Clip AV stream file referred to by one PlayItem includes at least video stream data (main image data). In addition, the ClipAV stream file may or may not include one or more audio streams that are played back at the same timing (synchronously) as the video stream (main image data) included in the ClipAV stream file. Also good. Further, the ClipAV stream file may or may not include one or more bitmap subtitle streams that are played back at the same timing as the video stream included in the ClipAV stream file. In addition, the ClipAV stream file may or may not include one or more interactive graphics streams that are played back at the same timing as the video stream included in the ClipAV stream file. The video stream included in the ClipAV stream file and the audio stream, bitmap subtitle stream file, or interactive graphics stream reproduced at the same timing as the video stream are multiplexed. That is, a Clip AV stream file referred to by one PlayItem includes video stream data, zero or more audio streams to be reproduced in accordance with the video stream, zero or more bitmap subtitle stream data, and zero or more Interactive graphics stream data is multiplexed.

  That is, a Clip AV stream file referenced by one PlayItem includes a plurality of types of streams such as a video stream, an audio stream, a bitmap subtitle stream, or an interactive graphics stream.

  One SubPlayItem refers to audio stream data and subtitle data of a stream (different stream) different from the ClipAV stream file referred to by the PlayItem.

  When playing a PlayList that has only the main path, the user can select audio and subtitles only from the audio stream and sub-picture stream multiplexed in the clip referenced by the main path. Can not do it. On the other hand, when playing a PlayList having a main path and a sub path, in addition to the audio stream and sub picture stream multiplexed in the Clip AV stream file referenced by the main path, the clip audio stream referenced by the SubPlayItem, A sub-picture stream can be referenced.

  In this way, since a plurality of SubPaths are included in one PlayList and each SubPath refers to each SubPlayItem, an AV stream with high expandability and high flexibility can be realized. In other words, in addition to the ClipAV stream referred to by MainPath, it is possible to add SubPlayItem later.

  FIG. 3 is a diagram illustrating an example of a main path and a sub path. In FIG. 3, a playback path of audio that is played back at the same timing as the main path (synchronized with AV) is represented using sub paths.

  The PlayList in FIG. 3 includes one PlayItem with PlayItem_id = 0 as a main path and one SubPlayItem as a sub path. One PlayItem () with PlayItem_id = 0 in the main path refers to the main AV stream in FIG. SubPlayItem () includes the following data. First, SubPlayItem () includes Clip_Information_file_name for designating a Clip referred to by a Sub Path in the PlayList. In the case of the example in FIG. 3, an auxiliary audio stream (audio stream) with SubClip_entry_id = 0 is referred to by SubPlayItem. Also, SubPlayItem () includes SubPlayItem_IN_time and SubPlayItem_OUT_time for designating the playback section of the Sub Path in the Clip (Auxiliary audio stream here). Furthermore, SubPlayItem () includes sync_PlayItem_id and sync_start_PTS_of_PlayItem for specifying the time at which the Sub Path starts to be played on the main path time axis (playback time axis). In the example of FIG. 3, sync_PlayItem_id = 0 and sync_start_PTS_of_PlayItem = t1. Thereby, it is possible to specify the time t1 at which the Sub Path starts playback on the time axis (playback time axis) of PlayItem_id = 0 of the main path. That is, in the case of the example in FIG. 3, the reproduction start time t1 of the main path and the start time t1 of the sub path are the same time.

  Here, the Clip AV stream of the audio referred to by the Sub Path must not include STC discontinuities (system time base discontinuities). The audio sample clock of Clip used for the sub path is locked to the audio sample clock of the main path.

  In other words, in SubPlayItem (), information specifying the Clip referred to by the Sub Path, information specifying the playback section of the Sub Path, and information specifying the time at which the Sub Path starts playback on the time axis of the Main path It is included. Since the Clip AV stream used in Sub Path does not include STC, information included in SubPlayItem () (information specifying Clip referred to by Sub Path, information specifying Sub Path playback section, and time axis of Main path) Based on the above (information specifying the time at which the Sub Path starts playback), playback can be performed with reference to an audio stream of a ClipAV stream different from the ClipAV stream (main AV stream) referred to by the main path.

  Thus, PlayItem and SubPlayItem manage ClipAV stream files, respectively. Here, the ClipAV stream file (main AV stream) managed by PlayItem and the ClipAV stream file managed by SubPlayItem are different files.

  Similar to the example of FIG. 3, a subtitle stream playback path that is played back at the same timing as the main path can also be expressed using a sub path.

  FIG. 4 is a diagram illustrating another example of the main path and the sub path. In FIG. 4, the video and audio playback paths that are played back at the same timing as the main path (synchronized with AV) are represented using sub-paths.

  The stream file referred to by PlayItem-1 in the main path is the first half of Clip-0 primary (primary) video stream and primary audio stream, and PG (Presentation Graphics) stream and IG (Intaractive Graphics) stream. -2 refers to the main video stream file of Clip-0, the primary video stream and the primary audio stream, and the latter half of the PG stream and the IG stream.

  The stream files referred to by SubPlayItems of the sub path are the clip-1 secondary (2ndary) video stream and the secondary audio stream.

  For example, a primary video stream and a primary audio stream referred to by the main path, and a PG stream and an IG stream as one movie content (AV content), and a secondary video stream and a secondary audio referred to by the sub path are used. When the stream is a bonus track that the director has commented on for the movie, and the video and audio stream referenced in the sub path is mixed with the video and audio stream referenced in the main path. In addition, such a configuration is used.

  Also, even when the secondary stream is not synchronized with the primary stream (a playlist setting that is always played back simultaneously), the user gives a command to synthesize and display the bonus track while watching the movie. Similarly, when the video and audio of the primary stream referenced in the main path and the video and audio of the secondary stream referenced in the sub path are mixed and reproduced when input to the playback device (player) Such a configuration is used.

  In FIG. 4, two PlayItems are arranged as PlayItem_id = 0, 1 in the main path, and one SubPlayItem is arranged in the sub path (Subpath_id = 0). SubPlayItem (described later in FIG. 21) called by SubPath (described later in FIG. 19) includes SubPlayItem_IN_time and SubPlayItem_out_time for designating the playback path of Subpath.

  When the secondary video and secondary audio are superimposed on the primary video and primary audio described with reference to FIG. 4 and reproduced and output, the audio is synthesized and output, and the video is as shown in FIG. On the main display screen 1 on which the primary video is displayed, the secondary video is displayed superimposed on the sub display screen 2 having a predetermined size at a predetermined position.

  In FIG. 4, what is referred to by the main path is the primary stream and what is referred to by the sub path is the secondary stream. However, the primary stream and the secondary stream may be referred to by the main path, respectively. , May be referenced by a subpath. For example, both the primary stream and the secondary stream may be referred to by the main path, and both the primary stream and the secondary stream may be referred to by the sub path.

  That is, as shown in FIG. 5, such a configuration displays a primary video stream as a main screen, and a so-called PinP (picture-in-picture) that combines and displays a secondary stream as a sub-screen in the main screen. : Picture-in-Picture) display can be used.

  FIG. 6 is a diagram showing an example of a file system of a data file that can be played back by the playback device 20 (described later with reference to FIG. 35). FIG. 6 shows, as an example, a case where a data file that can be played back by the playback device 20 is supplied by a recording medium such as an optical disk, and this file system has a directory structure. Even when a data file that can be played back by the playback device 20 is recorded and supplied on a recording medium other than the optical disk, or when it is recorded and played back on an internal recording medium, the playback device 20 can play back the data file. The data file preferably has a file system as shown in FIG.

  In this file system, a directory with the name “BDMV” is prepared under “root”, and a file with the name “Index.bdmv” and a file “NavigationObject.bdmv” Contains a file with a name. Hereinafter, these files will be referred to as an Index file and a NavigationObject file, respectively, as appropriate. As appropriate, each file is referred to by adding “file” to “file name”, or each directory is referred to by adding “directory” to “directory name”.

  The Index file is a file in which the above-described index table is described, and includes information related to a menu for reproducing a data file that can be reproduced by the reproducing apparatus 20. The playback device 20 includes, for example, a playback menu that includes items such as all content included in a data file that can be played back by the playback device 20, playback of only a specific chapter, repeated playback, and display of an initial menu. The screen is displayed on the display device based on the Index file. The navigation object to be executed when each item is selected can be set in the index table of the index file. When one item is selected from the playback menu screen by the user, the playback device 20 stores the navigation object in the index file index table. Execute the set NavigationObject command.

  The NavigationObject file is a file containing NavigationObject. The NavigationObject includes a command for controlling playback of a PlayList included in a data file that can be played back by the playback device 20. For example, the playback device 20 selects one of the NavigationObjects included in the file system. Then, the content can be reproduced by executing.

  The BDMV directory also includes a directory with the name “BACKUP” (BACKUP directory), a directory with the name “PLAYLIST” (PLAYLIST directory), and a directory with the name “CLIPINF” (CLIPINF directory). , A directory (STREAM directory) in which the name “STREAM” is set, and a directory (AUXDATA directory) in which the name “AUXDATA” is set.

  In the BACKUP directory, files and data for backing up files and data that can be played back by the playback device 20 are recorded.

  A PlayList file is stored in the PLAYLIST directory. As shown in the figure, each PlayList file is given a name obtained by adding an extension “.mpls” to a file name consisting of a 5-digit number. The PlayList file will be described later with reference to FIG.

  A Clip Information file is stored in the CLIPINF directory. Each Clip Information file is named by adding the extension “.clpi” to a file name consisting of five digits as shown in the figure.

  In the STREAM directory, a Clip AV stream file and a substream file are stored. Each stream file is given a name obtained by adding an extension “.m2ts” to a file name consisting of five digits as shown in the figure.

  The AUXDATA directory is not included in the Clip AV stream file or substream file, but is referred to from the Clip AV stream file or substream file, or data used independently of the Clip AV stream file or substream file. And other files are stored. In the example of FIG. 6, the AUXDATA directory stores sound data such as subtitle font files named “11111.otf” and sound effects with the name “sound.bdmv”. ing.

  Further, when a data file that can be played back by the playback device 20 is distributed by an optical disk, for example, the content author such as a production company or a movie distribution company, or the title author that is the supply source of the recording medium is displayed. Author_id, which is an identifier assigned to each title author for identification, and disc_id, which is an identifier assigned to identify the type of optical disc produced by the title author indicated by author_id, are rewritten by the user or the like. It cannot be recorded as secure electronic data or physically by pits.

  In addition, when a data file that can be played back by the playback device 20 is recorded on a removable recording medium other than the optical disk, or downloaded via a network and recorded on a recording medium inside the playback device, an internal storage unit Also, the author_id and the id corresponding to the author_id are assigned so that they can be distinguished from each other, and it is preferable to have a directory structure similar to that shown in FIG. Further, even if the data file that can be played back by the playback device 20 is not assigned author_id and id corresponding to author_id, the name “Index.bdmv” is the same as in the case described with reference to FIG. , Files with the name “NavigationObject.bdmv”, files with the name “BACKUP”, files with the name “PLAYLIST”, “CLIPINF Among the file group set with the name "STREAM", the file group set with the name "STREAM", and the file group set with the name "AUXDATA", necessary file groups are included as appropriate.

  Next, the data structure (syntax) of each file described with reference to FIG. 6 for specifically realizing the structure of the main path and the sub path described with reference to FIGS. 2 to 4 will be described.

  FIG. 7 shows the data structure of a PlayList file having the extension “.mpls” stored in the PLAYLIST directory.

  type_indicator describes information indicating the type of the file, and this field must be encoded as “MPLS” in accordance with ISO646.

  version_number indicates four character characters indicating the version number of this xxxxx.mpls, and vesrion_number must be encoded as ISO “0089”.

  PlayList_start_address indicates the start address of PlayList () with the relative number of bytes from the start byte of the PlayList file as a unit.

  PlayListMark_start_address indicates the head address of PlayListMark () with the relative number of bytes from the head byte of the PlayList file as a unit.

  PlayListEntsnsionData_start_address indicates the start address of PlayListEntsnsionData () with the relative number of bytes from the start byte of the PlayList file as a unit.

  AppInfoPlayList () stores parameters related to playback control of the PlayList.

  PlayList () stores parameters related to the main path and sub path of the PlayList. Details of PlayList () will be described later with reference to FIG.

  PlayListMark () stores PlayList mark information. Also, private data can be inserted into PlayListExtensionData ().

  FIG. 8 shows the syntax of PlayListExtensionData ().

  Length indicates the number of bytes of PlayListExtensionData () from immediately after this length field to the end of PlayListExtensionData ().

  data_block_start_address indicates the head address of the first data_block () with the relative number of bytes from the head byte of PlayListExtensionData () as a unit.

  number_of_PL_ext_data_entries indicates the number of entries of PlayListExtensionData included in PlayListExtensionData (). In PlayListExtensionData (), two or more same IDs must not exist.

  In ID1 / ID2, information (identifier etc.) that can identify the type of information described in the data_block of this PlayListExtensionData () is described.

  PL_ext_data_start_address indicates the number of data_block where the PlayListExtensionData is started. The top data of PlayListExtensionData must be aligned to the top of data_block.

  PL_ext_data_length indicates the size of PlayListExtensionData in bytes.

  data_block is an area in which PlayListExtensionData is stored, and all data_blocks in PlayListExtensionData () must have the same size.

  In the data_block, information indicating the display position and size of the secondary display screen 2 on which the secondary video described with reference to FIG. 5 is displayed can be described. In data_block, metadata indicating the display setting of picture-in-picture can be described.

  FIG. 9 shows a first example of the syntax of pip_metadata (metadata indicating picture-in-picture display settings) described in data_block.

  The length indicates the number of bytes of pip_metadata () from immediately after this length field to the end of pip_metadata ().

  The field of pip_metadata_type indicates the type of pip_metadata. As shown in FIG. 10, in the case of pip_metadata_type = 0x01, the secondary video stream in which picture-in-picture display is specified by this pip_metadata () is synchronous, that is, synchronized with the time axis (playback time axis) of the PlayItem of the Main Path Therefore, the secondary stream is always reproduced and output in synchronization with the primary stream. In addition, when pip_metadata_type = 0x02, the secondary video stream for which picture-in-picture display is specified by this pip_metadata () is asynchronous, that is, is played back in synchronization with the time axis of the Sub Path SubPlayItem, but the Main Path PlayItem Therefore, the secondary stream is reproduced and output only when the secondary stream display is instructed by the user's operation input.

  That is, pip_metadata_type = 0x02 and "synchronize with SubPlayItem" means that the secondary video stream for which picture-in-picture display is specified by this pip_metadata () is asynchronous with the time axis of the main path, but the time of the Sub Path It means that the axis is synchronized. That is, pip_metadata () in the case of pip_metadata_type = 0x02 is the display position and size of the sub display screen 2 during playback of the SubPlayItem, based on the moment when the secondary video display is started by the user's action (operation input) This is an indication.

  ref_to_PlayItem / SubPath_id indicates the value of PlayItem_id of PlayItem to which pip_metadata is applied, or the value of SubPath_id of SubPlayItem to which pip_metadata is applied. That is, when the secondary video stream in which picture-in-picture display is specified by this pip_metadata () is synchronized with the main path (when pip_metadata_type = 0x01), it becomes ref_to_PlayItm, and picture-in-picture display is specified by this pip_metadata () When the secondary video stream is synchronized with Sub Path (when pip_metadata_type = 0x02), ref_to_SubPath_id is set.

  pip_metadata_time_stamp is the time stamp of the PlayItem to which pip_metadata is applied, and the video of the secondary video stream is displayed on the sub display screen 2 at the display position and size specified by pip_metadata () at the timing indicated by pip_metadata_time_stamp.

  That is, pip_metadata_time_stamp needs to indicate the presentation-time between In_time / Out_time of PlayItem referenced by ref_to_PlayItem / SubPath_id. When the secondary video stream for which picture-in-picture display is specified by pip_metadata () is synchronized with the Sub Path (that is, when pip_metadata_type = 0x02), at the timing indicated by pip_metadata_time_stamp in the time axis of SubPlayItem The video of the secondary video stream is displayed on the secondary display screen 2 at the display position and size specified by pip_metadata ().

  pip_entry_video_PID indicates the PID value of the secondary video used for picture-in-picture display.

  pip_horizontal_position indicates the X coordinate of the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the primary video frame (main display screen 1 in FIG. 5). Further, pip_vertical_position indicates the Y coordinate of the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the primary video frame (main display screen 1 in FIG. 5).

  Note that, for example, the upper left corner of the sub display screen 2 by pip_horizontal_position and pip_vertical_position, such as designation of the X and Y coordinates of the lower right corner of the sub display screen 2 or designation of any of a plurality of predetermined display positions Of course, the display position of the secondary display screen 2 on which the secondary video is displayed may be designated by a method other than the designation of the X and Y coordinates.

  pip_scale is a field in which information indicating the size of the secondary display screen 2 on which the secondary video is displayed is described, for example, the ratio of the size of the secondary video to the primary video, the size of the original image of the secondary video Information indicating the reduction ratio or the number of pixels constituting the sub display screen 2 in the vertical and horizontal directions is described.

  FIG. 10 shows the value of pip_metadata_type in FIG. 9 and its meaning.

  pip_metadata_type = 0x00 is a value whose meaning is not set for future expansion. In the case of pip_metadata_type = 0x01, this pip_metadata () is synchronous, that is, synchronized with the time axis of the PlayItem in the Main Path, so that the secondary stream is always reproduced and output in synchronization with the primary stream. In the case of pip_metadata_type = 0x02, this pip_metadata () is asynchronous, that is, synchronized with the time axis of SubPlayItem of Sub Path, so only when secondary stream display is commanded by user operation input The video of the secondary video stream is reproduced and output on the sub display screen 2. In addition to the above three values of pip_metadata_type, no meaning is set for future expansion.

  FIG. 11 shows a second example of the syntax of pip_metadata described in data_block, which is different from the case in FIG. In addition, the description regarding the data field having the same name and the same definition as in FIG. 9 is omitted as appropriate.

  number_of_pip_entries is a field indicating the number of picture-in-picture applications.

  Synchronous_PIP_metadata_flag indicates to which path a picture-in-picture application (that is, a secondary video stream) is synchronized. That is, when the value of Synchronous_PIP_metadata_flag is 0 ′, this picture-in-picture application is synchronized with SubPath, and ref_to_SubPath_id is registered in ref_to_PlayItem / SubPath_id. On the other hand, when the value of Synchronous_PIP_metadata_flag is 1 ′, this picture-in-picture application is synchronized with MainPath, and ref_to_PlayItem / SubPath_id registers ref_to_PlayItem.

  That is, Synchronous_PIP_metadata_flag in FIG. 11 is information for defining substantially the same content as pip_metadata_type (FIG. 10) in pip_metadata described with reference to FIG.

  number_of_pip_metadata_entries is a field indicating the total number of PinP Metadata for the video stream of ref_to_PlayItem / ref_to_SubPath_id.

  pip_metadata_type indicates the type of metadata of this picture-in-picture. For example, in addition to information indicating the position and size of the secondary display screen 2 on which the secondary video stream is displayed, for example, the original video of the secondary video stream Information relating to the rotation of the displayed video, the designation of the color, and the like can be described.

  That is, pip_metadata_type in FIG. 11 is information whose definition is different from that of pip_metadata_type in pip_metadata described with reference to FIG.

  FIG. 12 shows a third example of the syntax of pip_metadata described in data_block, which is different from those in FIGS. 9 and 11.

  Length is a field in which information indicating the number of bytes of pip_metadata () immediately after this length field to the end of pip_metadata () is described.

  number_of_metadata_block_entries is a field in which information indicating the number of metadata block entries included in pip_metadata () is described.

  In metadata_block_header [k] (), information including header information related to the metadata block is described.

  ref_to_PlayItem_id [k] is a field in which information indicating the value of PlayItem_id of PlayItem to which pip_metadata is applied is described.

  ref_to_secondary_video_stream_id [k] is the id of the secondary video to which pip_metadata is applied, and is a field in which information indicating the value of secondary_video_stream_id defined in STN_table (described later in FIG. 22) in PlayItem referenced by ref_to_PlayItem_id is described. It is.

  pip_timeline_type [k] is a field in which information indicating the type of time axis referred to by pip_metadata_time_stamp is described. The value of pip_timeline_type [k] and its meaning will be described later.

  When this flag is set to 1, is_luma_key is applied to the secondary video stream according to the values of lower_limit_luma_key and upper_limit_luma_key.

  lower_limit_luma_key is a field in which information indicating the lower limit of the luminance value of the secondary video for luma keying is described.

  upper_limit_luma_key is a field in which information indicating the upper limit of the luminance value of the secondary video for luma keying is described.

  Here, “luma keying” means that an image in which an unnecessary part is cut out by using a difference in brightness (luminance value) is superimposed on the video. That is, when the flag of is_luma_key is set to 1, by setting the image having the luminance value specified in the range from the lower limit to the upper limit of the luminance value defined by lower_limit_luma_key and upper_limit_luma_key to be transparent, The secondary video from which the image in the range specified by the brightness information is removed is combined with the primary video.

  This allows you to display the minimum amount of secondary video to prevent the primary video from becoming difficult to see due to unnecessary portions of the secondary video, and to make the composition of the secondary video and primary video more flexible. It can be carried out.

  Further, in this embodiment, it has been described that it can be set to be transparent by using two threshold values of lower_limit_luma_key and upper_limit_luma_key, but the present invention is not limited thereto, and by setting the threshold value to either one, the luminance value is For example, only a portion below upper_limit_luma_key or above lower_limit_luma_key may be set to be transparent.

  metadata_block_data_start_address [k] is a field in which information indicating the first address of the first metadata_block_data [k] () is described in units of the relative number of bytes from the first byte of pip_metadata (). Note that metadata_block_data_start_address [k] entered in pip_metadata () must be registered with the aim of the address value.

  In padding_word, a padding word is inserted according to the value of metadata_block_data_start_address [k].

  In metadata_block_data [k] (), information including data information of the metadata block is described.

  number_pip_metadata_entries [k] is a field in which information indicating the total number of pip_metadata in metadata_block_data is described.

  pip_metadata_time_stamp [i] is a field in which information including a time stamp of 45 kHz, for example, indicating the time to which pip_metadata is applied is described. Note that the minimum interval between two consecutive pip_metadata_time_stamps is 1 second. The meaning of pip_metadata_time_stamp corresponding to the value of pip_timeline_type will be described later.

  pip_composition_metadata () is a field in which information on the size and position of the primary video frame is described.

  pip_horizontal_position [i] indicates the X coordinate of the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the primary video frame (main display screen 1 in FIG. 5). pip_vertical_position [i] indicates the Y coordinate of the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the primary video frame (main display screen 1 in FIG. 5).

  As shown in FIG. 13, pip_scale [i] is a reduction ratio with respect to the original image size of the secondary video of 1 ×, 1/2 ×, 1/4 ×, 1.5 ×, or full Screen (secondary This is a field for describing information to be displayed on the entire screen of the main display screen 1 in FIG.

  Note that pip_composition_metadata () of pip_metadata_time_stamp [i] is valid during an interval from pip_metadata_time_stamp [i] to pip_metadata_time_stamp [i + 1]. However, the effective interval of the last pip_compositoin_metadata is from the last pip_metadata_time_stamp to the SubPath presentation end time indicated by ref_to_secondary_video_stream_id [k].

  Next, the value of pip_timeline_type and its meaning will be described with reference to FIG. 14, and the meaning of pip_metadata_time_stamp corresponding to the value of pip_timeline_type will be described with reference to FIGS.

  pip_timeline_type = 0 is a value whose meaning is not set for future expansion.

  pip_timeline_type = 1 means that the picture-in-picture presentation path is a synchronous type. At this time, the type of Subpath indicated by ref_to_secondary_video_stream_id must be 5 or 7 (details of the Subpath type will be described later with reference to FIG. 20).

  At this time, as shown in FIG. 15, pip_metadata_time_stamp refers to the PlayItem time axis of ref_to_PlayItem_id and indicates the playback section of the associated SubPlayItem. That is, the playback section of SubPlayItem is projected on the time axis of PlayItem referenced by ref_to_PlayItem_id [k]. pip_metadata_time_stamp [0] indicates the beginning of the playback section of SubPlayItem [0] in SubPath.

  pip_timeline_type = 2 means that the picture-in-picture presentation path is an asynchronous type. At this time, the type of the Subpath indicated by ref_to_secondary_video_stream_id must be 6. In this case, the SubPath of SubPath_type = 6 includes only one SubPlayItem.

  At this time, as shown in FIG. 16, pip_metadata_time_stamp refers to the subpath time axis and indicates the playback path of the SubPath SubPlayItem indicated by ref_to_secondary_video_stream_id [k]. pip_metadata_time_stamp [0] indicates the beginning of the playback section of SubPlayItem.

  pip_timeline_type = 3 means that the picture-in-picture presentation path is an asynchronous type. At this time, the type of the Subpath indicated by ref_to_secondary_video_stream_id must be 6.

  At this time, as shown in FIG. 17, pip_metadata_time_stamp refers to the PlayItem time axis of ref_to_PlayItem_id and indicates the playback section of the PlayItem of ref_to_PlayItem_id [k]. pip_metadata_time_stamp [0] indicates the head of the playback section of the PlayItem of ref_to_PlayItem_id [k].

  FIG. 18 is a diagram illustrating the syntax of PlayList ().

  length is a 32-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of PlayList (). That is, this field indicates the number of bytes from reserved_for_future_use to the end of the Playlist. After this length, 16-bit reserved_for_future_use is prepared. number_of_PlayItems is a 16-bit field indicating the number of PlayItems in the PlayList. For example, in the example of FIG. 2, the number of PlayItems is three. The value of PlayItem_id is assigned from 0 in the order in which PlayItem () appears in the PlayList. For example, as shown in FIGS. 2 and 4, PlayItem_id = 0, 1, and 2 are allocated.

  number_of_SubPaths is a 16-bit field indicating the number of SubPaths (number of entries) in the PlayList. For example, in the example of FIG. 2, the number of Sub Paths is three. The value of SubPath_id is assigned from 0 in the order in which SubPath () appears in the PlayList. For example, as shown in FIG. 2, Subpath_id = 0, 1, and 2 are allocated. In the subsequent for statement, PlayItems are referenced by the number of PlayItems, and Sub Paths are referenced by the number of Sub Paths.

  FIG. 19 is a diagram illustrating the syntax of SubPath ().

  length is a 32-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of Sub Path (). That is, this field indicates the number of bytes from reserved_for_future_use to the end of the Playlist. After this length, 8-bit reserved_for_future_use is prepared. SubPath_type is an 8-bit field indicating the application type of SubPath. SubPath_type is used, for example, when the type indicates whether the Sub Path is audio, bitmap subtitle, text subtitle, or the like. This SubPath_type will be described later with reference to FIG. 15-bit reserved_for_future_use is prepared after SubPath_type. is_repeat_SubPath is a 1-bit field that designates the SubPath playback method, and indicates whether the SubPath playback is repeated during the main path playback or the SubPath playback is performed only once. For example, it is used when the playback timing of the clip specified by the main AV stream and the sub path is different (for example, when the main path is used as a still image slide show and the sub path audio path is used as the main path BGM). . 8-bit reserved_for_future_use is prepared after Is_repeat_SubPath. number_of_SubPlayItems is an 8-bit field indicating the number of SubPlayItems (number of entries) in one SubPath. For example, number_of_SubPlayItems has one SubPlayItem with SubPath_id = 0 and two SubPlayItems with SubPath_id = 1 in FIG. In subsequent for statements, SubPlayItems are referenced by the number of SubPlayItems.

  FIG. 20 is a diagram illustrating an example of SubPath_type (subpath type). That is, the type of SubPath is defined as shown in FIG. 20, for example.

  In addition, in each subpath type, the description “Out-of-mux... Type” is a TS (transport stream) including an ES (elementary stream) referred to in the subpath, that is, a clip. The type (kind) of the sub path when the play item (one or more ES) referred to in the main path is different from the TS (Clip), that is, the ES referred to in the sub path is referred to in the main path. This indicates that the sub-path type is not multiplexed in the TS (Clip) including the play item. Hereinafter, this type is referred to as a main path non-multiplex type path.

  In FIG. 20, SubPath_type = 0, 1 is reserved. SubPath_type = 2 is an Audio presentation path of the Browsable slideshow. For example, SubPath_type = 2 indicates that in the playlist, the audio presentation path referred to by the sub path and the main path referred to by the play item are asynchronous.

  SubPath_type = 3 is an interactive graphics presentation menu (interactive graphics presentation menu). For example, SubPath_type = 3 indicates that in the playlist, the interactive graphics menu referenced by the sub path and the main path referenced by the play item are asynchronous.

  SubPath_type = 4 is a Text subtitle presentation path (a text subtitle presentation path). For example, SubPath_type = 4 indicates that in the playlist, the text subtitle presentation path referenced by the sub path and the main path referenced by the play item are synchronized.

  Here, when the ES referred to in the main path and the ES referred to in the sub path are asynchronous as in SubPath_type = 2, 3, the type (type) of the sub path is hereinafter referred to as an asynchronous path. . On the other hand, when the ES referenced by the main path and the ES referenced by the sub path are synchronized as in SubPath_type = 4, the type (type) of the sub path is hereinafter referred to as a synchronous path.

  SubPath_type = 5 is Out-of-mux and AV Synchronized type of one or more elementary streams path (Primary audio / PG / IG / Secondary audio path) .Out of mux and AV synchronized type of Picture-in-Picture presentation path which Contains one or more elementary streams paths. That is, SubPath_type = 5 is a main path TS non-multiplexed and synchronous path, and includes one or more ES (Primary audio / PG / IG / Secondary audio) paths, picture-in-picture presentation paths, and so on. Has been.

  Here, the picture-in-picture presentation path refers to a primary audio stream stream, a secondary video stream, and a secondary audio stream for a predetermined primary video stream (a video stream referred to in the main path) in the above-described picture-in-picture method. , And one or more paths (types of such sub paths) in the subtitle stream.

  SubPath_type = 6 is Out-of-mux and AV non-Synchronized type of Picture-in-Picture presentation path which contains one or more elementary streams paths. That is, SubPath_type = 6 is a main path TS non-multiplexed and asynchronous path, and is a picture-in-picture presentation path (one or more ES paths).

  SubPath_type = 7 is In-mux type and AV Synchronized type of Picture-in-Picture presentation path which contains one or more elementary streams paths.

  Here, the description “In-mux type” includes a TS (Clip) that includes an ES referenced by a sub path, and a TS (Clip) that includes a play item (one or more ESs) referenced by a main path. Is the type (type) of sub-paths that are the same, that is, the type (type) of sub-paths that are multiplexed in the TS (Clip) that contains the play item referenced in the main path. It shows that there is. Hereinafter, this type is referred to as a main path multiplexing type path.

  That is, SubPath_type = 7 is a main path TS multiplexing type and synchronous type path, which is a picture-in-picture presentation path (one or more ES paths).

  SubPath_type = 8 to 255 is reserved.

  FIG. 21 is a diagram illustrating the syntax of SubPlayItem (i).

  The length is a 16-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of Sub playItem ().

  In FIG. 21, SubPlayItem is divided into a case of referring to one Clip and a case of referring to a plurality of Clips.

  First, a case where SubPlayItem refers to one Clip will be described.

  SubPlayItem includes Clip_Information_file_name [0] for specifying Clip. In addition, Clip_codec_identifier [0] that specifies the codec method of Clip, reserved_for_future_use, is_multi_Clip_entries that is a flag indicating the presence or absence of multi-clip registration, and ref_to_STC_id [0] that is information about STC discontinuity points (system time base discontinuity points) including. When the is_multi_Clip_entries flag is set, the syntax when SubPlayItem refers to a plurality of Clips is referenced. In addition, SubPlayItem_IN_time and SubPlayItem_OUT_time for specifying the playback section of the Sub Path in the Clip are included. Furthermore, sync_PlayItem_id and sync_start_PTS_of_PlayItem are included to specify the time when the Sub Path starts to be played on the time axis of the main path. As described above, the sync_PlayItem_id and sync_start_PTS_of_PlayItem are used in the case of FIGS. 3 and 4 (when the playback timing of the file indicated by the main AV stream and the sub path is the same), and the file indicated by the main AV stream and the sub path. Are not used (for example, when the still image referred to by the main path and the audio referred to by the sub path are not synchronized as in the BGM of a slide show composed of still images). Also, SubPlayItem_IN_time, SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start_PTS_of_PlayItem are used in common in the Clip referred to by SubPlayItem.

  Next, a case where SubPlayItem refers to a plurality of Clips (if (is_multi_Clip_entries == 1b), that is, multi-clip registration is performed) will be described. Specifically, as shown in FIG. 4, a case where SubPlayItem refers to a plurality of Clips is shown.

  num_of_Clip_entries indicates the number of Clips, and Clip_Information_file_name [SubClip_entry_id] specifies Clips excluding Clip_Information_file_name [0]. That is, Clips such as Clip_Information_file_name [1] and Clip_Information_file_name [2] are specified excluding Clip_Information_file_name [0]. Further, SubPlayItem includes Clip_codec_identifier [SubClip_entry_id] that specifies the codec method of Clip, ref_to_STC_id [SubClip_entry_id] that is information on STC discontinuity points (system time base discontinuity points), and reserved_for_future_use.

  Note that SubPlayItem_IN_time, SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start_PTS_of_PlayItem are used in common among a plurality of clips. In the example of FIG. 4, SubPlayItem_IN_time, SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start_PTS_of_PlayItem are used in common between SubClip_entry_id = 0 and SubClip_entry_id = 1, and the Text based subtitle for the selected SubClip_entry_id is _ Playback is based on SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start_PTS_of_PlayItem.

  Here, the value of SubClip_entry_id is assigned from 1 in the order in which Clip_Information_file_name [SubClip_entry_id] in SubPlayItem appears. Also, SubClip_entry_id of Clip_Information_file_name [0] is 0.

  FIG. 22 is a diagram illustrating the syntax of PlayItem ().

  length is a 16-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of PlayItem (). Clip_Information_file_name [0] is a field for designating a Clip referred to by PlayItem. In the example of FIG. 3, the main AV stream is referred to by Clip_Information_file_name [0]. Also included is Clip_codec_identifier [0] that specifies the codec format of Clip, reserved_for_future_use, is_multi_angle, connection_condition, and ref_to_STC_id [0] that is information regarding STC discontinuity points (system time base discontinuity points). Furthermore, IN_time and OUT_time for specifying the playback section of PlayItem in Clip are included. In the example of FIG. 3, the playback range of the main ClipAV stream file is represented by IN_time and OUT_time. Also, UO_mask_table (), PlayItem_random_access_mode, and still_mode are included. The case where there are a plurality of is_multi_angles is not directly related to the present invention, and the description thereof will be omitted.

  STN_table () in PlayItem () refers to the PlayItem's operations such as audio switching and subtitle switching when the target PlayItem and one or more SubPaths to be played back are prepared. It provides a mechanism that allows the user to select from Clips and Clips referenced by one or more of these SubPaths. STN_table () provides a mechanism that allows selection of mixing reproduction of two audio streams.

  FIG. 23 is a diagram illustrating a first example of the syntax of STN_table (). STN_table () is set as an attribute of PlayItem.

  length is a 16-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of STN_table (). 16-bit reserved_for_future_use is prepared after the length. number_of_video_stream_entries indicates the number of streams to which video_stream_id entered (registered) in STN_table () is given. video_stream_id is information for identifying a video stream, and video_stream_number is a video stream number that is used for video switching and is visible to the user.

  number_of_audio_stream_entries indicates the number of streams of the first audio stream to which audio_stream_id entered in STN_table () is given. The audio_stream_id is information for identifying the audio stream, and the audio_stream_number is an audio stream number that can be seen by the user used for audio switching. number_of_audio_stream2_entries indicates the number of streams of the second audio stream to which audio_stream_id2 entered in STN_table () is given. The audio_stream_id2 is information for identifying an audio stream, and the audio_stream_number is an audio stream number that can be seen by the user used for audio switching. Specifically, the number_of_audio_stream_entries audio stream entered in STN_table () is an audio stream decoded by the 1st audio decoder 75-1 of the playback apparatus 20 shown in FIG. 35 described later, and number_of_audio_stream2_entries entered in STN_table (). The audio stream is an audio stream decoded by a 2nd audio decoder 75-2 of the playback apparatus 20 shown in FIG. Thus, in FIG. 23 and STN_table () in FIGS. 33 and 34 described later, audio streams to be decoded by the two audio decoders can be entered.

  In the following description, the number_of_audio_stream_entries audio stream decoded by the 1st audio decoder 75-1 of the playback apparatus 20 in FIG. 35, that is, the primary audio stream is referred to as audio stream # 1, and the 2nd audio decoder of the playback apparatus 20 in FIG. The audio stream of number_of_audio_stream2_entries decoded in 75-2, that is, the secondary audio stream is referred to as audio stream # 2. Also, it is assumed that the audio stream # 1 is an audio stream that has priority over the audio stream # 2.

  number_of_PG_txtST_stream_entries indicates the number of streams to which PG_txtST_stream_id entered in STN_table () is given. In this, a stream (PG, Presentation Graphics stream) obtained by run-length encoding bitmap subtitles such as a DVD sub-picture and a text subtitle file (txtST) are entered. PG_txtST_stream_id is information for identifying a subtitle stream, and PG_txtST_stream_number is a subtitle stream number (text subtitle stream number) that can be seen by the user used for subtitle switching.

  number_of_IG_stream_entries indicates the number of streams provided with IG_stream_id entered in STN_table (). In this, an interactive graphics stream is entered. IG_stream_id is information for identifying an interactive graphics stream, and IG_stream_number is a graphics stream number that is visible to the user used for graphics switching.

  Here, the syntax of stream_entry () will be described with reference to FIG.

  length is an 8-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of stream_entry (). type is an 8-bit field indicating the type of information necessary to uniquely specify the stream to which the above-described stream number is given.

  In type = 1, a 16-bit packet ID (PID) is used to identify one elementary stream from among a plurality of elementary streams multiplexed in a Clip (Main Clip) referenced by PlayItem. Is specified. ref_to_stream_PID_of_mainClip indicates this PID. That is, with type = 1, the stream is determined simply by specifying the PID in the main ClipAV stream file.

  In type = 2, when SubPath refers to multiple Clips at a time and each Clip multiplexes multiple elementary streams, multiple elementary streams of one Clip (SubClip) referenced by SubPath SubPath_id, Clip id, and packet ID (PID) of the SubPath are specified in order to identify one elementary stream from the list. ref_to_SubPath_id indicates this SubPath_id, ref_to_SubClip_entry_id indicates this Clip id, and ref_to_stream_PID_of_SubClip indicates this PID. This is used when a plurality of Clips are referred to in SubPlayItem and a plurality of elementary streams are referred to this Clip.

  In this way, by using type (two types of type = 1 and type = 2), when PlayItem and one or more SubPaths to be played back are prepared, this PlayItem refers to One elementary stream can be identified from Clips referenced by Clip and one or more SubPaths. Note that type = 1 indicates a clip (main clip) referred to by the main path, and type = 2 indicates a clip (sub clip) referred to by the sub path.

  Returning to the description of STN_table () in FIG. 23, video_stream_id is given from 0 to one video elementary stream that is identified for each stream_entry () in turn in the for loop of the video stream ID (video_stream_id). Note that a video stream number (video_stream_number) may be used instead of the video stream ID (video_stream_id). In this case, video_stream_number is given from 1 instead of 0. That is, video_stream_number is obtained by adding 1 to the value of video_stream_id. The video stream number is defined from 1 since it is a video stream number that is used for video switching and is visible to the user.

  Similarly, in the for loop of the audio stream ID (audio_stream_id), audio_stream_id is assigned from 0 to one audio elementary stream that is specified for each stream_entry () in order. As in the case of the video stream, an audio stream number (audio_stream_number) may be used instead of the audio stream ID (audio_stream_id). In this case, audio_stream_number is given from 1 instead of 0. That is, audio_stream_number is obtained by adding 1 to the value of audio_stream_id. The audio stream number is an audio stream number that can be seen by the user and is used for audio switching.

  Similarly, audio_stream_id2 is given from 0 to one audio elementary stream specified for each stream_entry () in order in the for loop of the audio stream ID2 (audio_stream_id2). As in the case of the video stream, audio stream number 2 (audio_stream_number2) may be used instead of audio stream ID2 (audio_stream_id2). In this case, audio_stream_number2 is given from 1 instead of 0. That is, audio_stream_number2 is obtained by adding 1 to the value of audio_stream_id2. The audio stream number 2 is an audio stream number 2 that can be seen by the user and used for audio switching.

  That is, in STN_table () of FIG. 23, an audio stream of number_of_audio_stream_entries (audio stream # 1) and an audio stream of number_of_audio_stream2_entries (audio stream # 2) are defined. In other words, since the audio stream # 1 and the audio stream # 2 can be entered using the STN_table (), the user can select two audio streams to be reproduced in synchronization.

  Similarly, PG_txtST_stream_id is assigned from 0 to one bitmap subtitle elementary stream or text subtitle identified in order for each stream_entry () in the for loop of the subtitle stream ID (PG_txtST_stream_id). As in the case of the video stream, a subtitle stream number (PG_txtST_stream_number) may be used instead of the subtitle stream ID (PG_txtST_stream_id). In this case, PG_txtST_stream_number is given from 1 instead of 0. That is, PG_txtST_stream_number is obtained by adding 1 to the value of PG_txtST_stream_id. The subtitle stream number is a subtitle stream number (text subtitle stream number) that can be seen by the user used for subtitle switching, and is defined from 1.

  Similarly, in the for loop of the graphics stream ID (IG_stream_id), IG_stream_id is assigned from 0 to one interactive graphics elementary stream that is specified for each stream_entry () in order. As in the case of the video stream, a graphics stream number (IG_stream_number) may be used instead of the graphics stream ID (IG_stream_id). In this case, IG_stream_number is given from 1 instead of 0. That is, IG_stream_number is obtained by adding 1 to the value of IG_stream_id. Since the graphics stream number is a graphics stream number that can be seen by the user used for switching graphics, it is defined from 1.

  Next, stream_attribute () of STN_table () in FIG. 23 will be described.

  In the for statement after reserved_for_future_use, the video stream is referenced by the amount of the video stream, the audio stream is referenced by the disc manufacturer who has set the Main Path and Sub Path for the audio stream, and the amount of PG textST stream. Only the PG textST stream is referred to, and the IG stream is referred to by the amount of the IG stream.

  The stream_attribute () in the for loop of the video stream ID (video_stream_id) gives stream attribute information of one video elementary stream specified for each stream_entry (). That is, in this stream_attribute (), stream attribute information of one video elementary stream specified for each stream_entry () is described.

  Similarly, stream_attribute () in the for loop of the audio stream ID (audio_stream_id) gives stream attribute information of one audio elementary stream specified for each stream_entry (). That is, in this stream_attribute (), stream attribute information of one audio elementary stream specified for each stream_entry () is described. For example, since there is one audio elementary stream specified by type = 1 or type = 2 of stream_entry () in FIG. 24, stream_attribute () gives stream attribute information of that one audio elementary stream.

  Similarly, stream_attribute () in the for loop of audio stream ID2 (audio_stream_id2) gives stream attribute information of one audio elementary stream specified for each stream_entry (). That is, in this stream_attribute (), stream attribute information of one audio elementary stream specified for each stream_entry () is described. For example, since there is one audio elementary stream specified by type = 1 or type = 2 of stream_entry () in FIG. 24, stream_attribute () gives stream attribute information of that one audio elementary stream.

  Similarly, stream_attribute () in the for loop of the subtitle stream ID (PG_txtST_stream_id) gives stream attribute information of one bitmap subtitle elementary stream or text subtitle elementary stream specified for each stream_entry (). That is, in this stream_attribute (), stream attribute information of one bitmap subtitle elementary stream specified for each stream_entry () is described.

  Similarly, stream_attribute () in the for loop of the graphics stream ID (IG_stream_id) gives stream attribute information of one interactive graphics elementary stream specified for each stream_entry (). That is, stream attribute information of one interactive graphics elementary stream specified for each stream_entry () is described in the stream_attribute ().

  Here, the syntax of stream_attribute () will be described with reference to FIG.

  The length is a 16-bit unsigned integer indicating the number of bytes from immediately after this length field to the end of stream_attribute ().

  stream_coding_type indicates the encoding type of the elementary stream as shown in FIG. As encoding types of elementary streams, MPEG-2 video stream, HDMV LPCM audio, Dolby AC-3 audio, dts audio, Presentation graphics stream, Interactive graphics stream, and Text subtitle stream are described.

  video_format indicates the video format of the video elementary stream as shown in FIG. 480i, 576i, 480p, 1080i, 720p, and 1080p are described as the video format of the video elementary stream.

  As shown in FIG. 28, frame_rate indicates the frame rate of the video elementary stream. As frame rates of the video elementary stream, 24000/1001, 24, 25, 30000/1001, 50, and 60000/1001 are described.

  The aspect_ratio indicates the aspect ratio information of the video elementary stream as shown in FIG. As aspect ratio information of the video elementary stream, 4: 3 display aspect ratio and 16: 9 display aspect ratio are described.

  audio_presentation_type indicates presentation type information of an audio elementary stream as shown in FIG. As the presentation type information of the audio elementary stream, single mono channel, dual mono channel, stereo (2-channel), and multi-channel are described.

  Sampling_frequency indicates the sampling frequency of the audio elementary stream as shown in FIG. As sampling frequencies of audio elementary streams, 48 kHz and 96 kHz are described.

  audio_language_code indicates the language code of the audio elementary stream (Japanese, Korean, Chinese, etc.).

  PG_language_code indicates the language code (Japanese, Korean, Chinese, etc.) of the bitmap subtitle elementary stream.

  IG_language_code, which indicates the language code (Japanese, Korean, Chinese, etc.) of the interactive graphics elementary stream.

  textST_language_code indicates the language code (Japanese, Korean, Chinese, etc.) of the text subtitle elementary stream.

  The character_code indicates the character code of the text subtitle elementary stream as shown in FIG. The character code of the text subtitle elementary stream includes Unicode V1.1 (ISO 10646-1), Shift JIS (Japanese), KSC 5601-1987 including KSC 5653 for Roman character (Korean), GB 18030-2000 (Chinese), GB2312 (Chinese) and BIG5 (Chinese) are described.

  Hereinafter, the syntax of stream_attribute () in FIG. 25 will be specifically described with reference to FIG. 25 and FIGS. 26 to 32.

  When the encoding type of the elementary stream (stream_coding_type in FIG. 25) is an MPEG-2 video stream (FIG. 26), stream_attribute () includes the video format (FIG. 27) and frame rate (FIG. 28) of the elementary stream. ), And aspect ratio information (FIG. 29).

  When the encoding type of the elementary stream (stream_coding_type in FIG. 25) is HDMV LPCM audio, Dolby AC-3 audio, or dts audio (FIG. 26), the stream_attribute () includes the presentation type information of the audio elementary stream. (FIG. 30), sampling frequency (FIG. 31), and language code.

  When the encoding type of the elementary stream (stream_coding_type in FIG. 25) is a Presentation graphics stream (FIG. 26), the stream_attribute () includes the language code of the bitmap subtitle elementary stream.

  When the encoding type of the elementary stream (stream_coding_type in FIG. 25) is Interactive graphics stream (FIG. 26), stream_attribute () includes the language code of the interactive graphics elementary stream.

  When the encoding type of the elementary stream (stream_coding_type in FIG. 25) is a Text subtitle stream (FIG. 26), stream_attribute () includes the character code (FIG. 32) and language code of the text subtitle elementary stream. .

  Note that the attribute information is not limited to this.

  As described above, when a PlayItem and one or more SubPaths to be played back are prepared, a stream_entry () is used to select a Clip referred to by the PlayItem and a Clip referred to by one or more SubPaths. The attribute information of one identified elementary stream can be known by stream_attribute ().

  By examining this attribute information (stream_attribute ()), the playback device can check whether or not it has the function of playing back the elementary stream. Further, the playback device can select the elementary stream corresponding to the initial information of the language setting of the playback device by examining the attribute information.

  For example, it is assumed that the playback apparatus has only a playback function for a bitmap subtitle elementary stream and does not have a playback function for a text subtitle elementary stream. When the user instructs the playback apparatus to switch the language, the playback apparatus sequentially selects and reproduces only the bitmap subtitle elementary stream from the for loop of the subtitle stream ID (PG_txtST_stream_id).

  Also, for example, assume that the initial information of the language setting of the playback device is Japanese. When the user instructs the playback apparatus to switch the audio, the playback apparatus sequentially selects only the audio elementary stream whose language code is Japanese from the for loop of the audio stream ID (Audio stream id). And play.

  Furthermore, for example, when playing back an AV stream (for example, a movie) made up of a video stream and an audio stream referred to by the main path, the user instructs the playback device to switch the audio, and the audio stream # 1 ( When the audio to be output in a normal movie) and the audio stream # 2 (comment by a director or performer) are designated (selected) as audio to be played back, the playback apparatus uses the audio stream # 1 and the audio stream # 2. Are mixed (superposed) and reproduced together with the video stream.

  Note that the audio stream # 1 and the audio stream # 2 may both be audio streams included in the Clip referred to by the main path, as can be seen by referring to the STN_table () in FIG. Also, one of the audio stream # 1 and the audio stream # 2 may be an audio stream included in a clip referred to by the main path, and the other may be an audio stream included in the clip referenced by the sub path. As described above, it is possible to select two audio streams superimposed on the main AV stream referred to by the main path, mix them, and reproduce them.

  In this way, STN_table () in PlayItem () can be used for operations such as audio switching and subtitle switching by the user when this PlayItem and one or more SubPaths to be played back are prepared. Since a mechanism is provided that allows the user to select from the clip referenced by this PlayItem and the clip referenced by one or more SubPaths, a stream different from the AV stream to be reproduced, Interactive operations can also be performed on data files.

  Also, since a plurality of SubPaths are used in one PlayList and each SubPath refers to a SubPlayItem, an AV stream with high expandability and high flexibility can be realized. That is, it can be configured such that SubPlayItem can be added later. For example, when there is a ClipAV stream file referred to by the Main Path and a PlayList associated with the ClipAV stream file, and this PlayList is rewritten to a PlayList added with a new Sub Path, the ClipAV referenced by the Main Path is based on the new PlayList. Along with the stream file, reproduction can be performed by referring to a ClipAV stream file different from the ClipAV stream file referred to by the Main Path. Thus, it can be set as the structure which has an expandability.

  Furthermore, STN_table () in PlayItem () is an audio stream # 1 decoded by the 1st audio decoder 75-1 of the playback apparatus 20 shown in FIG. 35 to be described later, and an audio stream decoded by the 2nd audio decoder 75-2. A system that can mix and play # 2 is provided. For example, when PlayItem () and one or more SubPaths to be played back are prepared, the clip audio stream referenced by PlayItem is defined as audio stream # 1, and the clip audio stream referenced by SubPath is defined as audio stream # 1. An audio stream # 2 is provided, and a mechanism capable of mixing and reproducing these is provided. Also, for example, two audio streams included in a clip (main clip) referred to by PlayItem are set as an audio stream # 1 and an audio stream # 2, respectively, and these can be mixed and played back. did. Accordingly, it is possible to perform superimposed reproduction of an audio stream (for example, a director's comment stream) that is different from the main audio stream to be reproduced, in which the main AV stream is recorded. Also, the two audio streams # 1 and # 2 superimposed on the main AV stream can be superimposed (mixed) and reproduced.

  In addition, a combination of a secondary video stream, a primary audio stream, a secondary audio stream, and a subtitle stream that are simultaneously reproduced in combination with the primary video stream can be defined by STN_table (). 33 and 34 are diagrams illustrating a second example of the syntax of STN_table ().

  FIG. 33 and FIG. 34 show examples of the syntax of STN_table () when defining a combination of a secondary video stream, a primary audio stream, a secondary audio stream, and a subtitle stream that is simultaneously played in combination with the primary video stream. FIG. In the figure, the same parts as those in FIG. 23 are repeated, and the description thereof is omitted.

  In the example of FIGS. 33 and 34, a combination of a secondary video stream, a primary audio stream, a secondary audio stream, and a subtitle stream that are simultaneously reproduced in combination with the primary video stream is defined as follows. That is, first, one or more secondary video streams that are simultaneously reproduced in combination with the primary video stream are defined. For each of the one or more secondary video streams, an audio stream (primary audio stream and secondary audio stream) and a subtitle stream that are played back simultaneously are defined.

  Specifically, in FIG. 33, number_of_video_stream2_entries indicates the number of streams to which video_stream_id2 entered (registered) in STN_table () is given. video_stream_id2 is information for identifying a secondary video stream, and video_stream_number2 is a secondary video stream number that is used for video switching and is visible to the user.

  In FIG. 34, one video elementary stream (video elementary stream to be a secondary video stream) specified for each stream_entry () in order in the for loop of video stream ID 2 (video_stream_id 2) has a video_stream_id 2 from 0 to 0. Given.

  At this time, number_of_Audio_combinations_for_video2 is given to video_stream_id2, and in the subsequent for statement, audio_stream_id and audio_stream_id2 are given by the number of number_of_Audio_combinations_for_video2. number_of_Audio_combinations_for_video2 and the subsequent for statement are information that defines a combination of audio streams that are played back simultaneously with a secondary video stream, that is, a combination of a primary audio stream specified by audio_stream_id and a secondary audio stream specified by audio_stream_id2. is there. The number of sets of audio streams (a set of primary audio streams and secondary audio streams) that can be combined with the secondary video stream specified by video_stream_id2 is number_of_Audio_combinations_for_video2. Then, as a set of audio streams that can be combined with the secondary video stream identified by video_stream_id2, audio_stream_id that identifies the primary audio stream and audio_stream_id2 that identifies the secondary audio stream are defined in a for statement after number_of_Audio_combinations_for_video2, respectively. .

  Also, number_of_Subtitle_combinations_for_video2 is given to the same video_stream_id2, and PG_textST_stream_id is given to the same video_stream_id2 by the number of number_of_Subtitle_combinations_for_video2. The number_of_Subtitle_combinations_for_video2 and the subsequent for sentence are information defining a combination of subtitle streams that are reproduced simultaneously with the secondary video stream, that is, a combination of subtitle streams specified by PG_textST_stream_id. The number of subtitle streams that can be combined with the secondary video stream specified by video_stream_id2 is number_of_Subtitle_combinations_for_video2. Then, PG_textST_stream_id for specifying a subtitle stream that can be combined with the secondary video stream specified by video_stream_id2 is defined in a for sentence after number_of_Subtitle_combinations_for_video2.

  Similarly to the case described with reference to FIG. 23, each number is used instead of each ID, for example, an audio stream number (audio_stream_number) is used instead of audio_stream_id, or an audio stream number 2 ( audio_stream_number2) may be used. The same applies to the video stream and the subtitle stream.

  According to FIG. 33 and FIG. 34, it is possible to define a secondary video stream that is played back simultaneously with the primary video stream using video_stream_id2, and further, using video_stream_id2, audio_stream_id, audio_stream_id2, and PG_textST_stream_id, A combination of a primary audio stream, a secondary audio stream, and a subtitle stream that are played back simultaneously with the secondary video stream can be defined. That is, it is possible to define a combination of a secondary video stream, a primary audio stream, a secondary audio stream, and a subtitle stream that are played back simultaneously with the primary video stream.

  Therefore, it is not necessary to define a contradictory combination as a combination of the secondary video stream, the primary audio stream, the secondary audio stream, and the subtitle stream that is played back simultaneously with the primary video stream. It is possible to easily select a combination of streams that can be reproduced simultaneously.

  Also, STN_table () in PlayItem () has a Subpath_type of 5 to 7, that is, a picture-in-picture presentation, when this PlayItem and one or more SubPaths to be played back in association with this PlayItem are prepared. By defining a path (one or more paths of a secondary video stream, a secondary audio stream, and a presentation graphic stream), as described with reference to FIG. On the other hand, picture-in-picture display can be performed by displaying the secondary video stream.

  Since the picture-in-picture display setting is described in pip_metadata as described with reference to FIG. 9, FIG. 11, or FIG. 12, the size of the sub display screen 2 described with reference to FIG. The sheath display position can be arbitrarily set by the content creator or the content provider without depending on the playback device. In addition, the picture-in-picture display setting is not described in the data itself of the video stream (secondary video stream) displayed in picture-in-picture, but in the pip_metadata described with reference to FIG. 9, FIG. 11, or FIG. be written. That is, when it is desired to change the size or display position of the sub display screen 2 described with reference to FIG. 5, it is not necessary to correct the video stream (secondary video stream) displayed in picture-in-picture, and the description of pip_metadata is corrected. What should I do?

  Further, the display setting of the picture-in-picture is described in pip_metadata as described with reference to FIG. 9, FIG. 11, or FIG. 12, so that the sub display screen 2 described with reference to FIG. The size of the displayed video does not depend on the size of the image that is the source of the secondary video stream, and can be arbitrarily set by the content creator or the content provider.

  Next, a playback apparatus to which the present invention is applied will be described. FIG. 35 is a block diagram illustrating a configuration example of the playback device 20 to which the present invention has been applied. The playback device 20 is a playback device 20 that plays the PlayList having the main path and the sub path described above.

  The playback device 20 is provided with a storage drive 31, a switch 32, an AV decoder unit 33, and a controller 34.

  In the example of FIG. 35, first, the controller 34 reads out the PlayList file via the storage drive 31, and based on the information of the PlayList file, from the recording medium such as HDD, Blu-ray disc, or DVD via the storage drive 31 Read AV stream and AV data. The user can instruct the controller 34 to switch between audio and subtitles using the user interface. Further, the controller 34 is supplied with initial information of the language setting of the playback device 20 from a storage unit (not shown).

  The PlayList file includes STN_table () in addition to Main Path and Sub Path information. The controller 34 refers to a main Clip AV stream file (hereinafter referred to as a main Clip) referred to by a PlayItem included in the PlayList file, a sub Clip AV stream file (hereinafter referred to as a sub Clip) referred to by a SubPlayItem, and a SubPlayItem referred to. Text subtitle data is read from a recording medium or the like via the storage drive 31. Here, the main clip referred to by PlayItem and the sub Clip referred to by SubPlayItem may be recorded on different recording media. For example, the main clip may be recorded on a recording medium, and the corresponding sub clip may be supplied via a network (not shown) and stored in the HDD. In addition, the controller 34 selects and plays an elementary stream corresponding to the playback function of itself (playback device 20), or only the elementary stream corresponding to the initial information of the language setting of the playback device 20. Select and control to play.

  Further, the controller 34 refers to information (or identifier) described in ID1 / ID2 of PlayListExtensionData () of the PlayList file, and information (pip_metadata) related to picture-in-picture display is described in the data_block of this PlayListExtensionData (). If it is detected that the secondary video is displayed, the secondary video display setting displayed on the secondary display screen 2 described with reference to FIG. 5 is acquired with reference to pip_metadata described with reference to FIG. The video plane generating unit 92 of the unit 33 controls the synthesis of the primary video stream and the secondary video stream.

  The AV decoder unit 33 includes buffers 51 to 54, PID filter 55, PID filter 56, switches 57 to 59, PID filter 60, background decoder 71, 1st video decoder 72-1, 2nd video decoder 72-2, presentation graphic. Decoder 73, interactive graphics decoder 74, 1st audio decoder 75-1, 2nd audio decoder 75-2, Text-ST composition 76, switch 77, background plane generator 91, video plane generator 92, presentation graphic spray A generation unit 93, an interactive graphics plane generation unit 94, a buffer 95, a video data processing unit 96, a mixing processing unit 97, and a mixing processing unit 101.

  The 1st video decoder 72-1 is for decoding the primary video stream, and the 2nd video decoder 72-2 is for decoding the secondary video stream. The 1st audio decoder 75-1 is for decoding the audio stream # 1 (primary audio stream), and the 2nd audio decoder 75-2 is for decoding the audio stream # 2 (secondary audio stream). Is. Specifically, in STN_table () of FIG. 23 and FIGS. 33 and 34, the 1st video decoder 72-1 is for decoding the video stream given by video_stream_id, and the video stream given by video_stream_id 2 is The one for decoding is the 2nd video decoder 72-2, the one for decoding the audio stream given by audio_stream_id is the 1st audio decoder 75-1, and the one for decoding the audio stream given by audio_stream_id2 This is a 2nd audio decoder 75-2.

  As described above, the playback device 20 includes two video decoders (1st video decoder 72-1, 2nd video decoder 72-2) for decoding two video streams, and for decoding two audio streams. Two audio decoders (1st audio decoder 75-1 and 2nd audio decoder 75-2) are provided. In the following, when the 1st video decoder 72-1 and the 2nd video decoder 72-2 are not individually distinguished, they are referred to as the video decoder 72, and the 1st audio decoder 75-1 and the 2nd audio decoder 75-2 are individually distinguished. Otherwise, it is referred to as an audio decoder 75.

  The file data read by the controller 34 is demodulated by a demodulation / ECC decoding unit (not shown), and error correction is performed on the demodulated multiplexed stream. The switch 32 selects the demodulated and error-corrected data for each stream type based on the control from the controller 34 and supplies the selected data to the corresponding buffers 51 to 54. Specifically, the switch 32 supplies background image data to the buffer 51, supplies main clip data to the buffer 52, and supplies sub-clip data to the buffer 53 based on control from the controller 34. The switch 32 is switched to supply the Text-ST data to the buffer 54. The buffer 51 buffers background image data, the buffer 52 buffers main clip data, the buffer 53 buffers sub-clip data, and the buffer 54 buffers Text-ST data. To do.

  The main clip is a stream (for example, a transport stream) obtained by multiplexing one or more streams in addition to video among video, audio, bitmap subtitle (Presentation Graphics stream), and interactive graphics. The sub clip is a stream obtained by multiplexing one or more streams of video, bitmap subtitles, interactive graphics, and audio. Note that the data of the text subtitle data file (Text-ST) may or may not be in the form of a multiplexed stream such as a transport stream.

  When reading the main clip, sub-clip, and text subtitle data from the storage drive 31 (recording medium), each file may be read alternately in a time-division manner, or the sub-clip and text subtitle data are stored in the main All data may be preloaded into the buffer (buffer 53 or buffer 54) before reading from the clip.

  The playback device 20 reads the data of these files from the recording medium via the storage drive 31, and plays back video, bitmap subtitles, interactive graphics, and audio.

  Specifically, the stream data read from the buffer 52 which is the main clip read buffer is output to the PID (packet ID) filter 55 in the subsequent stage at a predetermined timing. The PID filter 55 distributes the input main clip to the decoders of the elementary streams in the subsequent stage according to the PID (packet ID) and outputs the result. That is, the PID filter 55 supplies the video stream to the PID filter 60 for supplying either the 1st video decoder 72-1 or the 2nd video decoder 72-2, and the presentation graphics stream to the presentation graphics decoder 73. The interactive graphics stream is supplied to the switch 57 serving as the supply source, the interactive stream is supplied to the switch 58 serving as the supply source to the interactive graphics decoder 74, and the audio stream is supplied to the 1st audio decoder 75-1 and the 2nd audio decoder 75-2. It supplies to the switch 59 used as a supply source.

  The presentation graphics stream is, for example, bitmap subtitle data, and the text subtitle data is, for example, text subtitle data.

  The stream data read from the buffer 53 serving as a sub-clip read buffer is output to the subsequent PID (packet ID) filter 56 at a predetermined timing. The PID filter 56 distributes the input sub clip to the decoder of each elementary stream in the subsequent stage according to the PID (packet ID) and outputs the result. That is, the PID filter 56 supplies the supplied video stream to the PID filter 60 for supplying either the 1st video decoder 72-1 or the 2nd video decoder 72-2, and the presentation graphics stream is supplied to the presentation graphics decoder. 73, the interactive graphics stream is supplied to the switch 58 serving as the supply source to the interactive graphics decoder 74, and the audio streams are supplied to the first audio decoder 75-1 and the second audio decoder 75-. 2 is supplied to the switch 59 which is a supply source to the power source 2.

  Data read from the buffer 51 that buffers background image data is supplied to the background decoder 71 at a predetermined timing. The background decoder 71 decodes the background image data and supplies the decoded background image data to the background plane generation unit 91.

  The switch 57 selects one of the presentation graphics stream included in the main clip supplied from the PID filter 55 and the presentation graphics stream included in the sub clip, and selects the selected presentation graphics stream. This is supplied to the presentation graphics decoder 73 in the subsequent stage. The presentation graphics decoder 73 decodes the presentation graphics stream, and supplies the decoded presentation graphics stream data to the switch 77 serving as a supply source to the presentation graphics plane generation unit 93.

  The switch 58 selects one of the interactive graphics stream included in the main clip supplied from the PID filter 55 and the interactive graphics stream included in the sub clip, and selects the selected interactive graphics stream. Is supplied to the interactive graphics stream decoder 74 in the subsequent stage. That is, the interactive graphics stream that is simultaneously input to the interactive graphics decoder 74 is a stream separated from either the main clip or the sub clip. The interactive graphics decoder 74 decodes the interactive graphics stream and supplies the decoded interactive graphics stream data to the interactive graphics plane generating unit 94.

  Further, the switch 59 selects one of the audio stream included in the main clip supplied from the PID filter 55 and the audio stream included in the sub clip, and selects the selected audio stream as the 1st audio in the subsequent stage. The data is supplied to the decoder 75-1 or the second audio decoder 75-2. Here, the audio stream simultaneously input to the 1st audio decoder 75-1 is a stream separated from either the main clip or the sub clip. Similarly, the audio stream input to the 2nd audio decoder 75-2 at the same time is a stream separated from either the main clip or the sub clip. For example, when the audio stream # 1 and the audio stream # 2 are included in the main clip, the PID filter 55 filters the audio stream # 1 and the audio stream # 2 based on the PID of the audio stream, and switches 59.

  For example, the switch 59 selects the switch so that the audio stream # 1 supplied from the PID filter 55 is supplied to the 1st audio decoder 75-1, and the audio stream # 2 supplied from the PID filter 55 is changed to the 2nd audio decoder. Select the switch to supply to 75-2.

  The PID filter 60 receives an input of the video stream included in the main clip supplied from the PID filter 55 or the video stream included in the sub clip supplied from the PID filter 56, and controls the controller 34. Based on this, it is determined whether the video stream is a primary video stream or a secondary video stream, the primary video stream is supplied to the 1st video decoder 72-1, and the secondary video stream is supplied to the 2nd video decoder 72-2.

  The video stream distributed by the PID filter 60 is supplied to the 1st video decoder 72-1 or 2nd video decoder 72-2 at the subsequent stage. The 1st video decoder 72-1 or the 2nd video decoder 72-2 decodes the supplied video stream and outputs the decoded video data to the video plane generation unit 92.

  When the video data is supplied from the 1st video decoder 72-1 and the 2nd video decoder 72-2, the video plane generation unit 92 combines the supplied video data based on the control of the controller 34 referring to pip_metadata. As described with reference to FIG. 5, a video plane constituted by the main display screen 1 and the sub display screen 2 is generated and supplied to the video data processing unit 96. In addition, when video data is supplied only from the 1st video decoder 72-1, the video plane generating unit 92 generates a video plane using the supplied video data, and supplies this to the video data processing unit 96. Note that combining two video data is also referred to as mixing or superimposing, for example.

  The 1st audio decoder 75-1 decodes the audio stream and supplies the decoded audio stream data to the mixing processing unit 101. The 2nd audio decoder 75-2 decodes the audio stream and supplies the decoded audio stream data to the mixing processing unit 101.

  Here, when audio stream # 1 and audio stream # 2 are superimposed and reproduced (when two audio streams are selected as audio streams to be reproduced by the user), decoding is performed by 1st audio decoder 75-1. The audio stream # 1 and the audio stream # 2 decoded by the 2nd audio decoder 75-2 are supplied to the mixing processing unit 101.

  The mixing processing unit 101 mixes (superimposes) the audio data from the 1st audio decoder 75-1 and the audio data from the 2nd audio decoder 75-2, and outputs the mixed data to the subsequent mixing processing unit 97. In the present embodiment, mixing (superimposing) the audio data output from the 1st audio decoder 75-1 and the audio data output from the 2nd audio decoder 75-2 is also referred to as synthesis. In other words, synthesis means to mix two audio data.

  The sound data selected by the switch 32 is supplied to the buffer 95 and buffered. The buffer 95 supplies the sound data to the mixing processing unit 97 at a predetermined timing. In this case, the sound data is sound effect data by menu selection or the like. The mixing processing unit 97 is the audio data mixed by the mixing processing unit 101 (audio data obtained by mixing the audio data output from the 1st audio decoder 75-1 and the audio data output from the 2nd audio decoder 75-2). The sound data supplied from the buffer 95 is mixed (superimposed or synthesized) and output as an audio signal.

  The data read from the buffer 54 that is a text subtitle read buffer is output to the text subtitle composition (decoder) 76 at the subsequent stage at a predetermined timing. The text subtitle composition 76 decodes the Text-ST data and supplies it to the switch 77.

  The switch 77 selects either the presentation graphics stream decoded by the presentation graphics decoder 73 or Text-ST (text subtitle data), and supplies the selected data to the presentation graphics plane generating unit 93. . That is, the subtitle image simultaneously supplied to the presentation graphics plane 93 is an output of either the presentation graphics decoder 73 or the text subtitle (Text-ST) composition 76. The presentation graphics stream that is simultaneously input to the presentation graphics decoder 73 is a stream separated from either the main clip or the sub clip (selected by the switch 57). Therefore, the subtitle image that is simultaneously output to the presentation graphics plane 93 is a decoding output of the presentation graphics stream from the main clip, the presentation graphics stream from the sub clip, or the text subtitle data.

  Based on the background image data supplied from the background decoder 71, for example, the background plane generation unit 91 generates a background plane that becomes a wallpaper image when the video image is displayed in a reduced size, and this is converted into the video data. This is supplied to the processing unit 96.

  The presentation graphics plane generation unit 93 generates, for example, a presentation graphics plane which is a rendering image based on the data (presentation graphics stream or text subtitle data) selected by the switch 77 and supplied, and this is converted into a video. The data is supplied to the data processor 96. The interactive graphics plane generating unit 94 generates an interactive graphics plane based on the data of the interactive graphics stream supplied from the interactive graphics decoder 74, and supplies this to the video data processing unit 96.

  The video data processing unit 96 includes a background plane from the background plane generation unit 91, a video plane from the video plane generation unit 92, a presentation graphics plane from the presentation graphics plane generation unit 93, and an interactive graphics plane generation unit. The interactive graphics plane from 94 is synthesized and output as a video signal. Also, the mixing processing unit 97 is the audio data from the mixing processing unit 101 (audio data obtained by mixing the audio data decoded by the 1st audio decoder 75-1 and the audio data decoded by the 2nd audio decoder 75-2). The sound data from the buffer 95 is mixed (synthesized or superimposed) and output as an audio signal.

  The switches 57 to 59 and the switch 77 are switched based on selection from the user via the user interface or the file side including the target data. For example, when the audio stream is included only in the sub clip AV stream file, the switch 59 switches the switch to the sub side.

  Next, playback processing in the playback apparatus 20 of FIG. 35 will be described with reference to the flowchart of FIG. Note that this process is started when a reproduction of a predetermined AV stream is instructed by the user via the user interface.

  In step S <b> 1, the controller 34 reads out a PlayList file recorded on a recording medium or an HDD (Hard Disk Drive) (not shown) via the storage drive 31. For example, the PlayList file (xxxxxx.mpls) described with reference to FIG. 7 is read.

  In step S2, the controller 34 determines whether or not secondary video exists in the stream to be played based on the read PlayList file.

  Specifically, the controller 34 executes the secondary video loop (for loop of video_stream_id2) of STN_table () described with reference to FIGS. 33 and 34 in the PlayItems in the read PlayList file, for example. Thus, it is determined whether or not the secondary video exists in the stream to be reproduced.

  If it is determined in step S2 that there is no secondary video, in step S3, a primary video playback process described later with reference to FIGS. 37 to 39 is executed.

  In step S4, the controller 34 determines whether or not the reproduction of the stream corresponding to the read PlayList file has been completed. If it is determined in step S4 that the reproduction of the stream is not finished, the process returns to step S3, and the subsequent processes are repeated. If it is determined that the reproduction of the stream is finished, the process is finished. The

  If it is determined in step S2 that a secondary video exists, the controller 34 reads pip_metadata in step S5. In addition, after confirming that secondary video is included as described above, only pip_metadata related to PlayItem may be stored in the memory, or pip_metadata as described with reference to FIGS. Since it is a part of Playlist, pip_metadata may be read simultaneously when reading the PlayList file in step S1.

  Specifically, for example, the controller 34 identifies the target pip_metadata based on the secondary_video_stream_id determined by the current PlayItem_id and STN_table (), and reads the identified pip_metadata from the PlayList file.

  In step S6, the controller 34 determines whether the secondary stream is synchronized with the primary stream (that is, synchronized with the time axis of the PlayItem of the main path) based on the read description of pip_metadata. .

  Specifically, for example, when the data structure of pip_metadata is the one described with reference to FIG. 9, the controller 34 determines that the secondary stream is the primary stream based on the value of pip_metadata_type described with reference to FIG. If the data structure of pip_metadata is the one described with reference to FIG. 11, the controller 34 refers to Synchronous_PIP_metadata_flag, and is the secondary stream synchronized with the primary stream? It can be determined whether or not.

  For example, when the data structure of pip_metadata is the one described with reference to FIG. 12, the controller 34 can determine whether the secondary stream is synchronized with the primary stream based on the Subpath_type.

  If the data structure of pip_metadata is the one described with reference to FIG. 9 or FIG. 11, whether or not the secondary stream is synchronized with the primary stream is not limited to pip_metadata but the data structure of pip_metadata in FIG. As with, it can also be determined by Subpath_type. That is, the controller 34 can determine whether or not the secondary stream is synchronized with the primary stream based on the Subpath_type.

  If it is determined in step S6 that the secondary stream is synchronized with the primary stream, the process proceeds to step S7, and the controller 34 determines that the display time of the primary stream is the secondary stream as described with reference to FIG. It is determined whether or not SubPlayItem_IN_time indicating the head of the playback section is reached.

  If it is determined in step S7 that the display time of the primary stream is not SubPlayItem_IN_time, in step S8, a primary video playback process described later with reference to FIGS. 37 to 39 is executed.

  On the other hand, when it is determined in step S7 that the display time of the primary stream is SubPlayItem_IN_time, primary and secondary video playback processing, which will be described later with reference to FIGS. 40 to 42, is executed in step S9.

  In step S10, the controller 34 determines whether or not the display time of the primary stream is SubPlayItem_OUT_time indicating the end of the playback section of the secondary stream. If it is determined in step S10 that it is not SubPlayItem_OUT_time, the process returns to step S9, and the subsequent processing is repeated. If it is determined that it is SubPlayItem_OUT_time, in step S11, the controller 34 corresponds to the read PlayList file. It is determined whether or not the reproduction of the stream to be completed is complete.

  If it is determined in step S11 that the reproduction of the stream is not complete, the process returns to step S7, and the subsequent processes are repeated. If it is determined that the reproduction of the stream is complete, the process is terminated. The

  On the other hand, when it is determined in step S6 that the secondary stream is not synchronized with the primary stream (that is, synchronized with the time axis of the SubPlayItem of the sub path), until the user gives an instruction to display the secondary video stream Since only the primary video stream is displayed, in step S12, a primary video playback process described later with reference to FIGS. 37 to 39 is executed.

  In step S13, the controller 34 determines whether or not an instruction to display a secondary video stream has been received from the user.

  If it is determined in step S13 that the secondary stream display instruction has not been received, in step S14, the controller 34 determines whether or not the reproduction of the stream corresponding to the read PlayList file has ended. . If it is determined in step S14 that the reproduction of the stream is not completed, the process returns to step S12, and the subsequent processes are repeated. If it is determined that the reproduction of the stream is completed, the process is terminated. The

  On the other hand, when it is determined in step S13 that an instruction to display a secondary video stream has been received, in step S15, primary and secondary video playback processing described later with reference to FIGS. 40 to 42 is executed.

  In step S <b> 16, the controller 34 determines whether a secondary video stream display end command has been received from the user. If it is determined in step S16 that a secondary stream display end command has been received, the process returns to step S12, and the subsequent processes are repeated.

  If it is determined in step S16 that the secondary video stream display end instruction has not been received, in step S17, the controller 34 determines whether or not the reproduction of the stream corresponding to the read PlayList file has ended. to decide. If it is determined in step S17 that the reproduction of the stream is not complete, the process returns to step S15, and the subsequent processes are repeated. If it is determined that the reproduction of the stream is complete, the process is terminated. The

  Through such processing, in the playback of the stream corresponding to the read PlayList file, whether or not the secondary stream exists, and if so, based on whether or not it is synchronized with the primary stream (main path) If the secondary stream is superimposed on the primary stream and always displayed, it is not synchronized with the primary stream (main path). If the secondary stream display is instructed by the user or the secondary stream display Whether or not to superimpose the secondary stream on the primary stream is determined based on whether or not the termination is instructed.

  Next, the primary video playback process executed in step S3, step S8, or step S12 of FIG. 36 will be described with reference to the flowcharts of FIGS.

  In step S41, the controller 34 reads out the main clip, the sub clip, and the text subtitle data (Text-ST data). Specifically, the controller 34 reads out the main clip based on the PlayItem described with reference to FIG. 22 included in the PlayList described with reference to FIG. In addition, the controller 34 reads out the sub clip and the text subtitle data based on the SubPlayItem described with reference to FIG. 21 that is referred to by the SubPath included in the PlayList.

  In step S42, the controller 34 controls the switch 32 so as to supply the read data (main clip, sub clip, and text subtitle data) to the corresponding buffers 51 to 54. Specifically, the controller 34 supplies background image data to the buffer 51, supplies main clip data to the buffer 52, supplies sub clip data to the buffer 53, and Text-ST data to the buffer 54. The switch 32 is switched so as to be supplied.

  In step S <b> 43, the switch 32 switches the switch 32 based on the control from the controller 34. As a result, the background image data is supplied to the buffer 51, the main clip data is supplied to the buffer 52, the sub clip data is supplied to the buffer 53, and the text subtitle data is supplied to the buffer 54.

  In step S44, each of the buffers 51 to 54 buffers the supplied data. Specifically, the buffer 51 buffers background image data, the buffer 52 buffers main clip data, the buffer 53 buffers sub-clip data, and the buffer 54 stores Text- Buffer ST data.

  In step S <b> 45, the buffer 51 outputs the background image data to the background decoder 71.

  In step S46, the buffer 52 outputs the stream data of the main clip to the PID filter 55.

  In step S47, the PID filter 55 distributes to the decoders of the elementary streams based on the PIDs attached to the TS packets constituting the main Clip AV stream file. Specifically, the PID filter 55 supplies the video stream to the 1st video decoder 72-1 through the PID filter 60, and supplies the presentation graphics stream to the switch 57 serving as the supply source to the presentation graphics decoder 73. The interactive graphics stream is supplied to the switch 58 serving as the supply source to the interactive graphics decoder 74, and the audio stream is supplied to the switch 59 serving as the supply source to the 1st audio decoder 75-1. That is, different PIDs are attached to the video stream, presentation graphics stream, interactive graphics stream, and audio stream, respectively.

  In step S <b> 48, the buffer 53 outputs the sub clip stream data to the PID filter 56.

  In step S49, the PID filter 56 distributes to the decoder of each elementary stream based on the PID. Specifically, the PID filter 56 supplies the supplied video stream to the 1st video decoder 72-1 through the PID filter 60, and the switch 57 serving as a supply source of the presentation graphics stream to the presentation graphics decoder 73. The interactive graphics stream is supplied to the switch 58 serving as the supply source to the interactive graphics decoder 74, and the audio stream is supplied to the switch 59 serving as the supply source to the 1st audio decoder 75-1 and the 2nd audio decoder 75-2. To supply.

  In step S50, the switches 57 to 59 and the PID filter 60 following the PID filter 55 and the PID filter 56 select either the main clip or the sub clip based on the control from the controller 34 via the user interface. Specifically, the switch 57 selects the presentation graphics stream of the main clip or the sub clip supplied from the PID filter 55 and supplies it to the presentation graphics decoder 73 at the subsequent stage. The switch 58 selects the interactive graphics stream of the main clip or the sub clip supplied from the PID filter 55 and supplies the selected interactive graphics stream to the subsequent interactive graphics stream decoder 74. Further, the switch 59 selects the audio stream of the main clip supplied from the PID filter 55 or the sub clip supplied from the PID filter 56 (in this case, the audio stream # 1 before the audio is switched). And supplied to the first audio decoder 75-1. When the user issues a command to switch the audio, the switch 59 supplies the main clip audio stream to the 2nd audio decoder 75-2 or the sub clip audio stream to the 1st audio decoder 75-. Although it may be supplied to the 1st and 2nd audio decoders 75-2, here, since the playback process before the sound is switched is described, the description thereof is omitted.

  In step S 51, the buffer 54 outputs the text subtitle data to the text subtitle composition 76.

  In step S <b> 52, the background decoder 71 decodes the background image data and outputs it to the background plane generation unit 91.

  In step S <b> 53, the 1st video decoder 72-1 decodes the video stream (that is, the supplied primary video stream), and outputs this to the video plane generating unit 92.

  In step S54, the presentation graphics decoder 73 decodes the presentation graphics stream selected and supplied by the switch 57, and outputs it to the subsequent switch 77.

  In step S55, the interactive graphics decoder 74 decodes the interactive graphics stream selected and supplied by the switch 58, and outputs this to the subsequent interactive graphics plane generating unit 94.

  In step S56, the 1st audio decoder 75-1 decodes the supplied audio stream (audio stream # 1) selected by the switch 59 and outputs it to the mixing processing unit 101 at the subsequent stage. In the reproduction process in a state where the user is not instructed to switch the audio, since the audio data is not output from the 2nd audio decoder 75-2, the mixing processing unit 101 is output from the 1st audio decoder 75-1. The audio data is supplied to the subsequent mixing processing unit 97 as it is.

  In step S57, the Text-ST composition 76 decodes the text subtitle data and outputs it to the subsequent switch 77.

  In step S58, the switch 77 selects either the presentation graphics decoder 73 or the data from the Text-ST composition 76. Specifically, the switch 77 selects any one of the presentation graphics stream decoded by the presentation graphics decoder 73 and Text-ST (text subtitle data), and the selected data is displayed in the presentation graphics sprayer. Supplied to the generator 93.

  In step S <b> 59, the background plane generation unit 91 generates a background plane based on the background image data supplied from the background decoder 71.

  In step S60, the video plane generating unit 92 generates a video plane based on the video data supplied from the 1st video decoder 72-1.

  In step S61, the presentation graphics plane generating unit 93 performs presentation based on the data from the presentation graphics decoder 73 or the data from the Text-ST composition 76 selected and supplied by the switch 77 in the process of step S58. Generate a graphics plane.

  In step S <b> 62, the interactive graphics plane generating unit 94 generates an interactive graphics plane based on the interactive graphics stream data supplied from the interactive graphics decoder 74.

  In step S63, the buffer 95 buffers the sound data selected and supplied in the process of step S43, and supplies it to the mixing processing unit 97 at a predetermined timing.

  In step S64, the video data processing unit 96 combines and outputs the data of each plane. Specifically, the data from the background plane generation unit 91, the video plane generation unit 92, the presentation graphics plane generation unit 93, and the interactive graphics plane generation unit 94 are combined and output as video data.

  In step S65, the mixing processing unit 97 mixes (synthesizes) and outputs the audio data (audio data output from the mixing processing unit 101) and sound data, and the process returns to step S3 in FIG. The process proceeds to S4 or returns to Step S8 in FIG. 36 and proceeds to Step S7 or returns to Step S12 in FIG. 36 and proceeds to Step S13.

  Through the processing described with reference to FIGS. 37 to 39, the main clip, sub-clip, and text subtitle data are referred to and reproduced by the main path and sub-path included in the PlayList. Here, the video to be displayed is only the video of the primary video stream, and the secondary display screen 2 described with reference to FIG. 5 is not displayed. Since the main path and sub path are provided in the PlayList, and a clip that is different from the clip AV stream file specified by the main path can be specified by the sub path, the clip is different from the main clip indicated by the PlayItem of the main path. The sub clip data and the main clip data can be reproduced together (at the same timing).

  In FIG. 37 to FIG. 39, the processes in steps S45 and S46 may be performed in reverse order or in parallel. In addition, the processes of step S47 and step S49 may be performed in reverse order or in parallel. Further, the processing of steps S52 to S57 may be performed in reverse order or in parallel. Further, the processing in steps S59 to S62 may be performed in reverse order or in parallel. Furthermore, the processing of step S64 and step S65 may be performed in reverse order or in parallel. That is, in FIG. 35, the processing of buffers 51 to 54 in the same vertical layer, the processing of switches 57 to 59, the processing of decoders 71 to 76, the processing of plane generation units 91 to 94, the video data processing unit 96, and the mixing processing unit The processes 97 may be executed in parallel, and the order thereof is not limited.

  Next, the primary and secondary video playback processing executed in step S9 or step S15 in FIG. 36 will be described with reference to the flowcharts in FIGS.

  In steps S101 to S106, basically the same processing as that in steps S41 to S46 described with reference to FIGS. 37 and 38 is executed.

  That is, the main clip, sub clip, and text subtitle data (Text-ST data) are read by the controller 34, and the read data (main clip, sub clip, and text subtitle data) are stored in the corresponding buffers 51 to 54. The switch 32 is controlled to supply. Based on the control from the controller 34, the switch 32 supplies the background image data to the buffer 51, the main clip data to the buffer 52, the sub clip data to the buffer 53, and the text subtitle data to The data is supplied to the buffer 54 and buffered in the buffers 51 to 54. The buffer 51 outputs the background image data to the background decoder 71. The buffer 52 outputs the main clip stream data to the PID filter 55.

  In step S107, the PID filter 55 distributes to the decoders of the elementary streams based on the PIDs attached to the TS packets constituting the main Clip AV stream file. Specifically, the PID filter 55 supplies the video stream to the PID filter 60, supplies the presentation graphics stream to the switch 57 serving as the supply source to the presentation graphics decoder 73, and converts the interactive graphics stream to the interactive graphics. The audio stream is supplied to the switch 59 which is a supply source to the decoder 74 and the audio stream is supplied to the switch 59 which is a supply source to the 1st audio decoder 75-1. That is, different PIDs are attached to the video stream, presentation graphics stream, interactive graphics stream, and audio stream, respectively. Based on the control of the controller 34, the PID filter 60 supplies the primary video stream to the 1st video decoder 72-1, and supplies the secondary video stream to the 2nd video decoder 72-2.

  In step S108, the buffer 53 outputs the sub clip stream data to the PID filter 56.

  In step S109, the PID filter 56 distributes to the decoder of each elementary stream based on the PID. Specifically, the PID filter 56 supplies the supplied video stream to the PID filter 60, supplies the presentation graphics stream to the switch 57 serving as a supply source to the presentation graphics decoder 73, and the interactive graphics stream is supplied. The video stream is supplied to the switch 58 serving as the supply source to the interactive graphics decoder 74, and the audio stream is supplied to the switch 59 serving as the supply source to the 1st audio decoder 75-1 and the 2nd audio decoder 75-2. Based on the control of the controller 34, the PID filter 60 supplies the primary video stream to the 1st video decoder 72-1, and supplies the secondary video stream to the 2nd video decoder 72-2.

  In steps S110 to S112, basically the same processing as that in steps S50 to S52 described with reference to FIGS. 37 and 38 is executed.

  That is, the switches 57 to 59 and the PID filter 60 select either the main clip or the sub clip based on the control from the controller 34. The buffer 54 outputs the text subtitle data to the text subtitle composition 76. The background decoder 71 decodes the background image data and outputs it to the background plane generation unit 91.

  In step S <b> 113, the 1st video decoder 72-1 decodes the supplied primary video stream and outputs it to the video plane generating unit 92.

  In step S <b> 114, the 2nd video decoder 72-2 decodes the supplied secondary video stream and outputs this to the video plane generating unit 92.

  In step S115, the presentation graphics decoder 73 decodes the presentation graphics stream selected and supplied by the switch 57, and outputs this to the subsequent switch 77.

  In step S116, the interactive graphics decoder 74 decodes the interactive graphics stream selected and supplied by the switch 58, and outputs this to the subsequent interactive graphics plane generating unit 94.

  In step S117, the 1st audio decoder 75-1 decodes the supplied primary audio stream selected by the switch 59 and outputs it to the mixing processing unit 101 at the subsequent stage.

  In step S118, the 2nd audio decoder 75-2 decodes the secondary audio stream selected and supplied by the switch 59, and outputs the decoded secondary audio stream to the subsequent mixing processing unit 101.

  In step S119, the Text-ST composition 76 decodes the displayed text subtitle data of primary or secondary, and outputs this to the switch 77 at the subsequent stage.

  In step S120, the switch 77 selects either the presentation graphics decoder 73 or the data from the Text-ST composition 76. Specifically, the switch 77 selects any one of the presentation graphics stream decoded by the presentation graphics decoder 73 and Text-ST (text subtitle data), and the selected data is displayed in the presentation graphics sprayer. Supplied to the generator 93.

  In step S <b> 121, the background plane generation unit 91 generates a background plane based on the background image data supplied from the background decoder 71.

  In step S122, the video plane generating unit 92 synthesizes the video data supplied from the 1st video decoder 72-1 and the 2nd video decoder 72-2 based on the control of the controller 34 referring to pip_metadata. A video plane constituted by the main display screen 1 and the sub display screen 2 as described above is generated and supplied to the video data processing unit 96.

  Specifically, the video plane generating unit 92 converts the secondary video stream positioned and scaled into the primary video based on the control of the controller 34 with reference to pip_metadata described with reference to FIG. 9, FIG. 11, or FIG. A video plane composed of the main display screen 1 and the sub display screen 2 as described with reference to FIG. 5 is generated by combining with the stream, and this is supplied to the video data processing unit 96.

  Here, the details of positioning and scaling will be described with reference to FIG.

  As shown in pip_metadata described with reference to FIG. 9, FIG. 11, or FIG. 12, pip_horizontal_position is, for example, in the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the main display screen 1 in FIG. For example, pip_vertical_position indicates the Y coordinate of the upper left corner of the secondary display screen 2 on which the secondary video is displayed on the main display screen 1 of FIG. Pip_scale indicates information related to the size of the sub display screen 2 on which the secondary video is displayed.

  That is, as shown in FIG. 43, on the Primary Video plane, the scaled secondary video adjusted to a predetermined size based on pip_scale is from the upper left corner of the primary video plane. A position that is pip_horizontal_position in the X-axis direction and pip_vertical_position in the Y-axis direction is arranged to be the upper left corner of the scaling secondary video.

  At this time, in the case of “is_luma_key = 1” in the pip_metadata described with reference to FIG. 12, the video plane generating unit 92 generates a video plane by combining the primary video stream and the secondary video stream by luma_keying. This is supplied to the video data processing unit 96.

  Here, the luma_keying composition is a technique in which an image in which an unnecessary part is cut out by using a difference in brightness (luminance value) is superimposed on a video as described with reference to pip_metadata in FIG. Next, the details of the luma_keying composition will be described with reference to FIGS. 44 and 45.

  FIG. 44 is a diagram showing Primary Video (secondary video) and Secondary Video (secondary video) before luma_keying composition.

  In the case of luma_keying composition of the primary video on the right side and the secondary video on the left side in the figure, as described using pip_metadata in FIG. The luminance value of the secondary video included in the range is made transparent, and the secondary video is synthesized with the primary video.

  That is, the secondary video from which the lower limit and the upper limit range of the luminance values defined by lower_limit_luma_key and upper_limit_luma_key are removed is combined with the primary video. Specifically, as shown in FIG. 44, the hatched area other than the parallelogram and the circle on the left secondary video becomes transparent because the brightness value is within the lower limit and upper limit range of the brightness value. The left secondary video processed in this way is superimposed on the right primary video and synthesized.

  FIG. 45 is a diagram showing the primary video and secondary video after the luma_keying composition.

  As shown in FIG. 45, the primary video and the secondary video after the luma_keying composition are transparent except for the parallelogram and the circle of the secondary video. Only the region will be combined with the primary video.

  In FIG. 45, for easy understanding, the secondary video area is represented by a dotted line, but this dotted line is not actually displayed.

  Returning to the flowchart of FIG. 42, in step S123, the presentation graphics plane generating unit 93 selects the data from the presentation graphics decoder 73 or the Text-ST composition 76 selected and supplied by the switch 77 in the process of step S58. A presentation graphics plane is generated based on the data.

  In step S124, the interactive graphics plane generating unit 94 generates an interactive graphics plane based on the data of the interactive graphics stream supplied from the interactive graphics decoder 74.

  In step S125, the buffer 95 buffers the sound data selected and supplied in step S43, and supplies the sound data to the mixing processing unit 97 at a predetermined timing.

  In step S126, the video data processing unit 96 combines and outputs the data of each plane. Specifically, the data from the background plane generation unit 91, the video plane generation unit 92, the presentation graphics plane generation unit 93, and the interactive graphics plane generation unit 94 are combined and output as video data.

  In step S127, the mixing processing unit 101 synthesizes the primary audio data output from the 1st audio decoder 75-1 and the secondary audio data output from the 2nd audio decoder 75-2, and supplies the synthesized audio data to the mixing processing unit 97. .

  In step S128, the mixing processing unit 97 mixes (synthesizes) and outputs the synthesized audio data and sound data output from the mixing processing unit 101, and the process returns to step S9 in FIG. Proceed to S10 or return to Step S15 in FIG. 36 and proceed to Step S16.

  40 to 42, the main clip, the sub clip, and the text subtitle data are referred to and reproduced by the main path and the sub path included in the PlayList. A main path and a sub path are provided in the PlayList, and a clip different from the Clip AV stream file specified by the main path can be specified by the sub path, and the display image of the secondary video stream is superimposed on the display image of the primary video stream. Can do.

  At this time, since the size and display position of the display image of the secondary video stream can be set, the display position and display size of the secondary display screen are predetermined depending on the playback device. In comparison, depending on the content or the timing at which it is displayed, the secondary video stream is displayed at a position or size that does not interfere with the display of the primary video stream, or depending on the content of the secondary video stream, For example, it is possible to display an important content in a conspicuous position, or to display a small content on the edge of the main display screen 1 if the content is not important.

  Further, the size and display position of the display image of the secondary video stream can be appropriately determined by the content producer or the content distributor.

  40 to 42, the processes in steps S105 and S106 may be executed in reverse order or in parallel. In addition, the processes in steps S107 and S109 may be performed in reverse order or in parallel. Further, the processing in steps S112 to S119 may be performed in reverse order or in parallel. In addition, the processes in steps S121 to S124 may be performed in reverse order or in parallel. Further, the processing in step S126 and the processing in steps S127 and S128 may be performed in reverse order or in parallel. That is, in FIG. 35, the processing of buffers 51 to 54 in the same vertical layer, the processing of switches 57 to 59, the processing of decoders 71 to 76, the processing of plane generation units 91 to 94, the video data processing unit 96, and the mixing processing unit The processes 97 may be executed in parallel, and the order thereof is not limited.

  The above processing is summarized as follows.

  The playback device 20 has a main path that is a main playback path indicating the position of a main ClipAV stream file including at least one stream, and a playback path of a sub ClipAV stream file that is played back by a playback path different from the main ClipAV stream file. A PlayList is obtained as playback management information including information on a certain subpath. Then, the playback device 20 accepts selection of a stream to be played based on STN_table () for selecting a stream to be played back included in the PlayList. STN_table () is for selecting a predetermined type of stream of the main ClipAV stream file (for example, a primary video stream) and another stream file that is played back in synchronization with the playback timing of the stream or asynchronously. Since it is information, selection of a stream to be played back can be accepted based on this STN_table ().

  Further, if the PlayList Subpath_type is In-Mux, the main path and the sub path can refer to the same Clip, and a sub path can be added, so that the stream can be extended. In addition, since a plurality of files can be referred to by one Sub Path (for example, FIG. 4), the user can select from a plurality of different streams.

  Further, in the Play item of the Main Path, the attached data (for example, an audio stream) multiplexed (included) in the AV stream file referenced by the Main Path and the attached data referenced by the Sub Path are defined. 23, and STN_table () of FIGS. 33 and 34 are provided as the table to be performed, so that a stream with higher expandability can be realized. In addition, Sub Path can be easily extended by entering STN_table ().

  In addition, since stream_attribute () of FIG. 25, which is stream attribute information, is provided in STN_table (), the playback apparatus 20 can determine whether or not the selected stream can be played back. Furthermore, by referring to stream_attribute (), it is possible to select and reproduce only a stream having a reproduction function.

  Furthermore, in STN_table () (FIG. 23, and FIGS. 33 and 34), two same types of stream files (here, primary video stream file and secondary video stream file, or primary audio stream file and secondary audio stream). File). Furthermore, the playback device 20 of FIG. 35 is provided with two video decoders (1st video decoder 72-1 and 2nd video decoder 72-2) and two audio decoders (1st audio decoder 75-1 and 2nd audio decoder 75-2). It was. A video plane generation unit 92 that combines (mixes) primary video data and secondary video data decoded by two video decoders is provided, and a mixing processing unit 101 that combines audio data decoded by two audio decoders is provided. I did it. As a result, two streams of the same type can be combined and reproduced simultaneously.

  That is, in the playback apparatus to which the present invention is applied, as shown in FIG. 5, the primary video stream is displayed on the main screen, and the secondary stream is combined and displayed as a child screen in the main screen. (Picture in picture) display is possible.

  At this time, the size and display position of the display image of the secondary video stream in picture-in-picture display can be set by pip_metadata. As a result, compared to the case where the display position and display size of the sub display screen are determined in advance depending on the playback device, the display of the primary video stream is obstructed depending on the content or the display timing. Depending on the content of the secondary video stream, for example, if the content is important, the secondary video stream may be displayed in a conspicuous position, or may not be important. For example, it is possible to display a small size at the end of the main display screen 1.

  Further, the size and display position of the display image of the secondary video stream can be appropriately determined by the content producer or the content distributor.

  In addition to the size and display position of the display image of the secondary video stream, information indicating the setting for displaying this picture-in-picture can be described in pip_metadata. In addition to the information indicating the position and size of the secondary display screen 2 on which the secondary video stream is displayed, for example, information relating to the rotation and color designation of the video displayed with respect to the original video of the secondary video stream is described. Can do.

  Here, information about display including the size and display position of the display image of the secondary video stream is not described in the secondary video stream, but is described in a playlist for controlling reproduction of the secondary video stream. That is, when it is desired to change only the display format without changing the secondary video stream (for example, when changing only the display position of the secondary video stream or the display size, etc.) Only the description of pip_metadata in the playlist needs to be updated without modifying the secondary video stream itself.

  Further, in the first content and the second content, the same secondary video stream is displayed by different display methods (for example, the same secondary video stream is large in the first content and small in the second content. It is very easy to display. That is, the information indicating the desired display format is described in the pip_metadata of the playlist in each of the first content and the second content without changing the data of the secondary video stream itself. In the content and the second content, the same secondary video stream can be displayed by different display methods desired by the content creator or the content distributor.

  In addition, since the display information including the size and display position of the display image of the secondary video stream is described in a playlist having detailed reproduction management information including time information, the display of the secondary video is detailed. It becomes possible to set.

  Further, as shown in FIG. 20, Sub Path refers to SubPath_type and Sub Path indicating the type of Sub Path (type such as audio or text subtitle and type indicating whether or not synchronized with main path). 21 includes the Clip_Information_file_name in FIG. 21 indicating the name of the sub clip, the SubPlayItem_IN_time in FIG. 21 indicating the in point of the clip referred to by the Sub Path, and the SubPlayItem_OUT_time in FIG. 21 indicating the out point. It can be accurately identified.

  Note that the data read by the storage drive 31 in FIG. 35 may be data recorded on a recording medium such as a DVD (Digital Versatile Disc), may be data recorded on a hard disk, Data downloaded via a network (not shown) may be used, or data obtained by combining these may be used. For example, it may be played back based on the PlayList and sub-Clip downloaded and recorded on the hard disk, and the main Clip AV stream file recorded on the DVD. Also, for example, when a PlayList and a main clip that record a clip AV stream file recorded on a DVD as a sub clip are recorded on the hard disk, the main clip and the sub clip are based on the play list recorded on the hard disk. Each may be read from the hard disk and DVD and played back.

  Note that the position and size of the secondary video can be changed by the function of the playback device 20 (player). However, in this case, the size may not be as intended by the author. On the other hand, if the size, position, etc. are managed in the playlist that is management information as in the present invention, even if it is desired to change the position after obtaining the ROM disk, for example, it is not necessary to change the actual clip. However, it is possible to change the position of the secondary video or the like as intended by the author simply by acquiring a playlist having a relatively small size.

  Next, with reference to FIGS. 46 and 47, a method for manufacturing the recording medium 21 on which data that can be reproduced by the reproducing apparatus 20 is recorded will be described by taking the case where the recording medium 21 is a disc-shaped recording medium as an example. .

  That is, as shown in FIG. 46, a master disc made of, for example, glass is prepared, and a recording material made of, for example, a photoresist is applied thereon. As a result, a recording master is produced.

  Then, as shown in FIG. 47, video data in a format that can be played back by the playback device 20 and encoded by the encoding device (video encoder) is stored in a temporary buffer in the software production processing unit, and is recorded by the audio encoder. The encoded audio data is stored in the temporary buffer, and data other than the stream (for example, Indexes, Playlist, PlayItem, etc.) encoded by the data encoder is further stored in the temporary buffer. Video data, audio data, and data other than the stream stored in each buffer are multiplexed together with a synchronization signal by a multiplexer (MPX), and an error correction code is added by an error correction code circuit (ECC). Is done. Then, predetermined modulation is applied by a modulation circuit (MOD), and software that is once recorded on a magnetic tape or the like according to a predetermined format and recorded on a recording medium 21 that can be played back by the playback device 20 is manufactured.

  This software is edited (premastered) as necessary, and a signal in a format to be recorded on the optical disc is generated. Then, as shown in FIG. 46, the laser beam is modulated corresponding to the recording signal, and this laser beam is irradiated onto the photoresist on the master. Thereby, the photoresist on the master is exposed in accordance with the recording signal.

  Then, this master is developed and pits appear on the master. The master plate thus prepared is subjected to a process such as electroforming to produce a metal master plate in which pits on the glass master plate are transferred. A metal stamper is further produced from this metal master, and this is used as a molding die.

  A material such as PMMA (acrylic) or PC (polycarbonate) is injected into this molding die by, for example, injection and fixed. Alternatively, 2P (ultraviolet curable resin) or the like is applied on the metal stamper and then cured by irradiation with ultraviolet rays. Thereby, the pits on the metal stamper can be transferred onto the replica made of resin.

  A reflective film is formed on the replica thus generated by vapor deposition or sputtering. Alternatively, a reflective film is formed on the generated replica by spin coating.

  Thereafter, the inner and outer diameters of the disk are processed, and necessary measures such as bonding two disks are performed. Further, a label is attached or a hub is attached and inserted into the cartridge. In this way, the recording medium 21 on which data reproducible by the reproducing apparatus 20 is recorded is completed.

  The series of processes described above can be executed by hardware or can be executed by software. In this case, the processing described above is executed by a personal computer 500 as shown in FIG.

  48, a CPU (Central Processing Unit) 501 performs various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. Execute. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via an internal bus 504. An input / output interface 505 is also connected to the internal bus 504.

  The input / output interface 505 includes an input unit 506 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), an output unit 507 including a speaker, and a storage configured by a hard disk. A communication unit 509 including a unit 508 and a modem, a terminal adapter, and the like is connected. A communication unit 509 performs communication processing via various networks including a telephone line and CATV.

  A drive 510 is also connected to the input / output interface 505 as necessary, and a removable medium 521 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted, and a computer program read therefrom is It is installed in the storage unit 508 as necessary.

  When a series of processing is executed by software, a program constituting the software is installed from a network or a program storage medium.

  As shown in FIG. 48, this program storage medium is configured only by a package medium including a removable medium 521 on which a program is recorded, which is distributed to provide the program to the user, separately from the computer. Rather, it is provided with a ROM 502 in which a program is recorded and a hard disk including a storage unit 508 provided to the user in a state of being preinstalled in the apparatus main body.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure which shows the example of the application format on the recording medium with which the reproducing | regenerating apparatus to which this invention is applied is mounted | worn. It is a figure explaining the structure of a main path and a sub path. It is a figure explaining the example of a main path and a sub path. It is a figure explaining another example of a main path and a sub path. It is a figure explaining a main display screen and a sub display screen. It is a figure which shows the example of the file system of the data file which can be reproduced | regenerated. It is a figure which shows the data structure of the PlayList file which has the extension ".mpls" stored in the PLAYLIST directory. It is a figure which shows the syntax of PlayListExtensionData (). It is a figure explaining the 1st example of the syntax of pip_metadata described in data_block. It is a figure explaining the value of pip_metadata_type of FIG. 9, and its meaning. It is a figure explaining the 2nd example of the syntax of pip_metadata described in data_block. It is a figure explaining the 3rd example of the syntax of pip_metadata described in data_block. It is a figure explaining the value of pip_scale of FIG. 12, and its meaning. It is a figure explaining the value of pip_timeline_type of FIG. 12, and its meaning. It is a figure explaining the meaning of pip_metadata_time_stamp when pip_timeline_type = 1 of FIG. It is a figure explaining the meaning of pip_metadata_time_stamp when pip_timeline_type = 2 of FIG. It is a figure explaining the meaning of pip_metadata_time_stamp when pip_timeline_type = 3 of FIG. It is a figure which shows the syntax of PlayList (). It is a figure which shows the syntax of SubPath (). It is a figure explaining SubPath_type. It is a figure which shows the syntax of SubPlayItem (i). It is a figure which shows the syntax of PlayItem (). It is a figure which shows the syntax of STN_table (). It is a figure which shows the syntax of stream_entry (). It is a figure which shows the syntax of stream_attribute (). It is a figure explaining stream_cording_type. It is a figure explaining video_format. It is a figure explaining frame_rate. It is a figure explaining aspect_ratio. It is a figure explaining audio_presentation_type. It is a figure explaining sampling_frequency. It is a figure explaining Character code. It is a figure which shows the syntax of STN_table (). It is a figure which shows the syntax of STN_table (). It is a block diagram which shows the structural example of the reproducing | regenerating apparatus to which this invention is applied. FIG. 36 is a flowchart for describing playback processing in the playback apparatus of FIG. 35. FIG. It is a flowchart explaining a primary video reproduction process. It is a flowchart explaining a primary video reproduction process. It is a flowchart explaining a primary video reproduction process. It is a flowchart explaining a primary and secondary video reproduction process. It is a flowchart explaining a primary and secondary video reproduction process. It is a flowchart explaining a primary and secondary video reproduction process. It is a figure explaining positioning and scaling. It is a figure explaining luma_keying composition. It is a figure explaining luma_keying composition. It is a figure for demonstrating manufacture of the recording medium which recorded the data which can be reproduce | regenerated with a reproducing | regenerating apparatus. It is a figure for demonstrating manufacture of the recording medium which recorded the data which can be reproduce | regenerated with a reproducing | regenerating apparatus. It is a figure which shows the structure of a personal computer.

Explanation of symbols

  1 main display screen, 2 sub display screen, 20 playback device, 31 storage drive, 32 switch, 33 AV decoder unit, 34 controller, 51 to 54 buffer, 55, 56 PID filter, 57 to 59 switch, 71 background decoder, 72-1 1st Video Decoder, 72-2 2nd Video Decoder, 73 Presentation Graphics Decoder, 74 Interactive Graphics Decoder, 75 Audio Decoder, 76 Text-ST Composition, 77 Switch, 91 Background Plane Generation Unit, 92 Video Plane Generating unit, 93 presentation graphics plane generating unit, 94 interactive graphics plane generating unit, 95 buffer 96 bidet Data processing unit, 97 a mixing processing unit, 101 mixing processor

Claims (6)

A data generation method for generating playback management data used for content playback in a playback device,
Get mainstream video and substream video,
Acquire audio to be played back corresponding to the main stream,
If there is audio played back corresponding to the substream, the audio played back corresponding to the substream is synthesized with the audio played back corresponding to the main stream,
First information indicating a position on the time axis for reproducing the main stream;
Second information indicating a position on a time axis for reproducing a substream different from the main stream;
Information relating to the display state of the sub-stream video to be combined with the main-stream video and executed based on information indicating the brightness of the sub-stream video to be combined with the main-stream video. A data generation method including a step of generating reproduction management data having a data structure including : information related to the display state including information related to processing for combining the video of the main stream and the video of the substream . The information indicating the brightness of the video of the substream is at least one threshold value of information indicating the brightness to be subjected to transmission processing, and the video portion where the brightness of the video of the substream is equal to or higher than the threshold value data generating method according to claim 1 for transmission processing on. In a recording apparatus for recording data on a recording medium on which reproduction management data used for content reproduction in the reproducing apparatus is recorded,
First acquisition means for acquiring a main stream video and a substream video;
Second acquisition means for acquiring audio played back corresponding to the main stream;
When there is audio to be played back corresponding to the substream, the audio to be played back corresponding to the substream, which is synthesized with the audio played back corresponding to the main stream, is acquired. Acquisition means of
First information indicating a position on the time axis for reproducing the main stream;
Second information indicating a position on a time axis for reproducing a substream different from the main stream;
Information relating to the display state of the sub-stream video to be combined with the main-stream video and executed based on information indicating the brightness of the sub-stream video to be combined with the main-stream video. Generating means for generating reproduction management data having a data structure including information relating to the display state including information relating to processing for combining the video of the main stream and the video of the substream ;
A recording apparatus comprising: a recording unit configured to record the acquired video and audio of the main stream, the acquired video and audio of the substream, or the video of the substream, and the generated reproduction management data on the recording medium. .   The information indicating the brightness of the video of the substream is at least one threshold value of information indicating the brightness to be subjected to the transmission process, and the video portion in which the brightness of the video of the substream is greater than or less than the threshold value Perform transparent processing on
  The recording apparatus according to claim 3. A recording medium recording method for recording playback management data used for content playback in a playback device,
Get mainstream video and substream video,
Acquire audio to be played back corresponding to the main stream,
If there is audio played back corresponding to the substream, the audio played back corresponding to the substream is synthesized with the audio played back corresponding to the main stream,
First information indicating a position on the time axis for reproducing the main stream;
Second information indicating a position on a time axis for reproducing a substream different from the main stream;
Information relating to the display state of the sub-stream video to be combined with the main-stream video and executed based on information indicating the brightness of the sub-stream video to be combined with the main-stream video. Generating reproduction management data having a data structure including information relating to the display state including information relating to processing for combining the video of the main stream and the video of the substream ;
Recording the obtained main stream video and audio, the obtained substream video and audio or the substream video, and the generated reproduction management data on the recording medium. Method. A program for causing a computer to execute processing for recording data on a recording medium on which playback management data used for content playback in a playback device is recorded,
Get mainstream video and substream video,
Acquire audio to be played back corresponding to the main stream,
If there is audio played back corresponding to the substream, the audio played back corresponding to the substream is synthesized with the audio played back corresponding to the main stream,
First information indicating a position on the time axis for reproducing the main stream;
Second information indicating a position on a time axis for reproducing a substream different from the main stream;
Information relating to the display state of the sub-stream video to be combined with the main-stream video and executed based on information indicating the brightness of the sub-stream video to be combined with the main-stream video. Generating reproduction management data having a data structure including information relating to the display state including information relating to processing for combining the video of the main stream and the video of the substream ;
A process including a step of recording the acquired video and audio of the main stream, the acquired video and audio of the substream or the video of the substream, and the generated reproduction management data on the recording medium. The program to be executed.

Images