JP6088968B2 - Fragment-based multimedia streaming service providing method and apparatus, and fragment-based multimedia streaming service receiving method and apparatus - Google Patents

Description

  The present invention relates to the generation and reception of data streams for transmitting multimedia content.

  In general, one multimedia content is composed of a combination of elements having different attributes such as audio, video, image, and metadata. Each of these data is encoded and recorded and utilized in a data structure for multimedia contents together with the relationship between the elements and information for decoding and reproducing each element.

  Due to the data structure of such multimedia contents, MPEG (Moving Picture Experts Group), which is an international standard organization, defines a basic file format that can be commonly applied to various applications, that is, an ISO-based media file format. . Using the ISO-based media file format, auxiliary data such as encoded elements of multimedia content and configuration information related to the elements are hierarchically stored in a plurality of containers. Basically, the ISO-based media file format defines a basic structure for efficiently storing each multimedia element in a single multimedia file.

  The present invention discloses a method and apparatus for providing a multimedia streaming service for transmitting 2D video or 3D video, and a method and apparatus for receiving a multimedia streaming service.

  According to an embodiment, a streaming service providing method for transmitting multimedia using a network includes dividing encoded data of a video including at least one of 2D video and 3D video into fragments. Each fragment includes at least one of three-dimensional signaling information including information for two-dimensional reproduction or three-dimensional reproduction of a current video and relation information for the video to refer to one or more other videos. Defining attribute information; and continuously transmitting a fragment composed of an auxiliary data area including attribute information about the fragment and a media data area including the encoded video data.

1 is a block diagram of a multimedia streaming service providing apparatus according to an embodiment. FIG. 1 is a block diagram of a multimedia streaming service receiver according to an embodiment. FIG. 6 is a diagram illustrating an mfsi box including a streaming file format transmitted in units of fragments and three-dimensional signaling information transmitted for each fragment according to an embodiment. 4 is a diagram illustrating a syntax of 3D signaling information included in an mfsi box and an scdi box according to an embodiment. 6 is a diagram illustrating an svfi box including a streaming file format transmitted in units of fragments and 3D signaling information transmitted for each fragment according to an embodiment. 3 is a diagram illustrating syntax of 3D signaling information included in an svfi box according to an embodiment. 3 is a diagram illustrating syntax of 3D signaling information included in an svfi box according to an embodiment. 3 is a diagram illustrating a syntax of 3D signaling information set for a file in a streaming file format transmitted in fragment units according to an embodiment; 3 is a diagram illustrating a syntax of 3D signaling information set for a sample in a streaming file format transmitted in fragment units according to an embodiment; 2 is a diagram illustrating two streaming file formats for transmitting a left-view video and a right-view video, respectively, according to an embodiment. 6 is a diagram illustrating reference information set for a stereoscopic video included in a file according to an exemplary embodiment. 10 is a diagram illustrating one piece of relation information set in a file when transmitting stereoscopic video using two streaming files according to an exemplary embodiment. 5 is a diagram illustrating a plurality of pieces of related information set in a file when transmitting stereoscopic video using two streaming files according to an exemplary embodiment. 6 is a diagram illustrating a syntax of track information including related information regarding a plurality of reference videos when transmitting stereoscopic video using two streaming files according to an exemplary embodiment. 6 is a diagram illustrating a syntax of relation information set in a file when transmitting stereoscopic video using two streaming files according to an exemplary embodiment. 6 is a diagram illustrating relation information set using an index file outside a file when transmitting stereoscopic video using two streaming files according to an exemplary embodiment. 3 is a diagram illustrating a streaming file format for transmitting left-view video and right-view video of a stereoscopic video according to an embodiment; 3 is a diagram illustrating a streaming file format for transmitting left-view video and right-view video of a stereoscopic video according to an embodiment; FIG. 6 is a diagram illustrating a method for determining a reference video using an index file outside a file when transmitting stereoscopic video using a single streaming file according to an exemplary embodiment. FIG. 6 is a diagram illustrating a method for determining a reference video using an index file outside a file when transmitting stereoscopic video using a single streaming file according to an exemplary embodiment. FIG. 6 is a diagram illustrating afgp box syntax including relationship information of multi-channel video groups according to an embodiment for an adaptive streaming service. FIG. FIG. 5 is a diagram illustrating a case where relationship information between multi-channel video groups is set using an index file outside a file for an adaptive streaming service according to an exemplary embodiment. 3 is a flowchart of a multimedia streaming service providing method according to an embodiment. 3 is a flowchart of a multimedia streaming service receiving method according to an embodiment.

  According to an embodiment, a streaming service providing method for transmitting multimedia using a network includes dividing encoded data of a video including at least one of 2D video and 3D video into fragments. For each fragment, at least one of three-dimensional signaling information including information for two-dimensional reproduction or three-dimensional reproduction of the current video, and relation information for the video to refer to one or more other videos. Determining attribute information including, continuously transmitting fragments composed of an auxiliary data area including attribute information about the fragment and a media data area including the encoded video data. .

  According to one embodiment, a streaming service receiving method for receiving multimedia using a network includes: receiving a plurality of fragments continuously; and a media data area for each of the fragments from a 2D video and a 3D video. Extracting the encoded data of the video including at least one in a form divided into fragments, and parsing the attribute information from the auxiliary data area for each of the fragments, Extracting and reading at least one of three-dimensional signaling information for three-dimensional reproduction, relation information for the video to refer to one or more other videos, and the attribute information for each fragment Based on the read information, the code of the fragment Comprising the steps of recovered data into 2D video or 3D video, the.

  According to an embodiment, a streaming service providing apparatus for transmitting multimedia using a network may include fragment video data that divides encoded data of video including at least one of 2D video and 3D video into fragments. At least one of generating information, 3D signaling information including information for 2D playback or 3D playback of the current video for each fragment, and relationship information for the video to refer to one or more other videos A fragment attribute information determination unit that determines attribute information including one, an auxiliary data area that includes attribute information about the fragment, and a media data area that includes the encoded video data are continuously generated. A streaming unit for transmission.

  According to an embodiment, a streaming service receiving apparatus that receives multimedia using a network includes a receiving unit that continuously receives a plurality of fragments, and a media data area for each of the two-dimensional video and three-dimensional video. A fragment video data extraction unit that extracts encoded data of a video including at least one of them in a form divided into fragments, and attribute information is parsed from an auxiliary data area for each of the fragments, A fragment attribute information extraction unit that extracts and reads at least one of two-dimensional reproduction information or three-dimensional signaling information for three-dimensional reproduction and relation information for the video to refer to one or more other videos; Read from the attribute information for each fragment Based on the information, and a restoration unit for restoring the encoded data of the fragment in a two-dimensional video or 3D video.

  The present invention includes a computer-readable recording medium having recorded thereon a program for implementing a multimedia streaming service providing method according to an embodiment on an arithmetic processor. A computer-readable recording medium on which a program for implementing a multimedia streaming service receiving method according to an embodiment is implemented by an arithmetic processor is included.

  FIG. 1 is a block diagram of an apparatus for providing a multimedia streaming service according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment uses a streaming file format transmitted in units of fragments according to an embodiment to use a 2D video, a 3D video, or a 2D video and a 3D video using a network. While transmitting the video data of the mixed video, it is possible to transmit the attribute information including at least one of the three-dimensional signaling information regarding the video data and the relationship information regarding the reference relationship between the plurality of videos for each file, fragment, and sample.

  The multimedia streaming service providing apparatus 100 according to an embodiment includes a fragment video data generation unit 110, a fragment attribute information determination unit 120, and a streaming unit 130.

  The fragment video data generation unit 110, the fragment attribute information determination unit 120, and the streaming unit 130 according to an exemplary embodiment may receive data from each other under the control of a central processing unit or a processor located inside or outside the multimedia streaming service providing apparatus 100. It is possible to operate organically while exchanging. Alternatively, at least one component of the fragment video data generation unit 110, the fragment attribute information determination unit 120, and the streaming unit 130 according to an embodiment is operated by a processor mounted on the fragment video data generation unit 110, and is organic with other components. It is controlled to operate.

  The fragment video data generator 110 according to an embodiment divides video data including at least one of 2D video data and 3D video data into fragment video data for transmission in fragment units.

  A fragment is a package unit for streaming data using a network. The fragment is composed of a media data area for transmitting media data and an auxiliary data area for transmitting attribute information such as information related to the content of media data in the media data area and structure related information. Streaming data is transmitted while one or more fragments are sequentially transmitted.

  For example, in the case of a streaming file in an ISO-based media file format, a fragment is composed of an mdat area containing media data and a moof area containing various attribute information related to the media data in the mdat area.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment may receive compressed and encoded video data and divide the data into fragments for streaming. Alternatively, the multimedia streaming service providing apparatus 100 according to an exemplary embodiment may receive uncompressed video data, encode the video data using an encoder mounted on the device, and divide the video data into fragment units. Also good.

  The fragment attribute information determination unit 120 according to an embodiment determines three-dimensional signaling information including information for video two-dimensional reproduction or three-dimensional reproduction for each fragment.

  The fragment attribute information determination unit 120 according to an embodiment determines three-dimensional signaling information transmitted through the auxiliary data area of a fragment. The 3D signaling information includes various information set depending on whether the current video is a 2D video or a 3D video. The three-dimensional signaling information includes various kinds of information necessary for the decoder to restore the video to perform two-dimensional reproduction or three-dimensional reproduction.

  When the 3D signaling information is transmitted for each fragment, 2D video / 3D video related information regarding the current video is transmitted for each fragment.

  The fragment attribute information determination unit 120 according to an embodiment determines relationship information for referring to each other between a plurality of videos including a current video. As related information, reference information and position information regarding other videos currently located in the file are transmitted. In addition, position information and index information for referring to one or more other videos located in an external file of the current file are transmitted as related information.

  The streaming unit 130 according to an exemplary embodiment continuously includes attribute information including at least one of the three-dimensional signaling information and the relationship information determined by the fragment attribute information determination unit 120 and a fragment including video data divided into fragments. Transmit. By continuously transmitting the fragments, video data and three-dimensional signaling information or related information are streamed.

  For each fragment, 3D signaling information and related information regarding the current fragment video is transmitted through the auxiliary data area, and encoded video data of the current fragment is transmitted through the media data area.

  The fragment attribute information determination unit 120 according to an embodiment determines three-dimensional configuration information and left-right order information regarding the current video for each fragment. The 3D configuration information according to an embodiment indicates a form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video of the current video. The left-right order information according to the embodiment indicates the order of the left viewpoint video and the right viewpoint video in the stereoscopic video.

  The fragment attribute information determination unit 120 according to an embodiment determines stereo mono section change count information indicating the number of changes between a monoscopic video section and a stereoscopic video section for the current video for each fragment.

  The fragment includes a monoscopic video section composed of a monoscopic video sequence or a stereoscopic video section composed of a stereoscopic video sequence. The fragment may also include a mixed video segment of a monoscopic video segment and a stereoscopic video segment.

  The fragment attribute information determination unit 120 according to an embodiment determines stereo confirmation information indicating whether the current section is a monoscopic video section or a stereoscopic video section.

  A sample is a data unit indicating each frame in a series of time continuous video frames or time continuous compressed audio sections. That is, in the multimedia streaming service providing apparatus 100 according to an embodiment, a fragment sample is used as a term similar to a video frame.

  In order to segment the monoscopic video segment or the stereoscopic video segment of the current fragment, the number of samples making up each segment is used. The fragment attribute information determination unit 120 according to an embodiment determines interval sample number information indicating the number of samples constituting the current interval.

  The fragment attribute information determination unit 120 according to an embodiment determines three-dimensional configuration information and left-right order information about samples constituting the current stereoscopic video section for each stereoscopic video section of the current fragment.

  The fragment attribute information determination unit 120 according to the embodiment determines at least one of telescopic camera information and stereoscopic display information for each stereoscopic video section of the current fragment.

  The stereoscopic camera information according to an embodiment includes information about the stereoscopic camera that has acquired the samples that make up the current stereoscopic video segment. The stereoscopic display information according to an embodiment includes information regarding a display standard for safely reproducing a sample in three dimensions.

  In the stereoscopic video section, the number of samples that make up each item is used to identify the item to which the stereoscopic camera display information applies. The fragment attribute information determination unit 120 according to an embodiment determines information indicating the number of items to which the stereoscopic camera display information is applied in the current stereoscopic video section of the current fragment.

  In addition, the fragment attribute information determination unit 120 according to an embodiment includes identification information of current stereoscopic camera display information referred to by each item, and information on a length of a section to which the current stereoscopic camera display information is applied. Define at least one of them. Information on the length of the section to which the current stereoscopic camera display information is applied is expressed by the number of samples constituting the current item to which the current stereoscopic camera display information is referred.

  The fragment attribute information determination unit 120 according to an embodiment may determine three-dimensional signaling information in units of files. The fragment attribute information determination unit 120 according to an embodiment determines three-dimensional configuration information and left-right order information regarding a stereoscopic video of the current file.

  The fragment attribute information determination unit 120 according to an embodiment includes a stereo mono which indicates the number of changes between a mono fragment composed of a monoscopic video section and a stereo fragment composed of a stereoscopic video section in a video of a current file. Define fragment change count information.

  The streaming unit 130 according to an exemplary embodiment transmits the three-dimensional configuration information of the current file, the left-right order information, and the stereo mono-fragment change count information, and the stereo mono section change count information and the section sample count information for each fragment of the current file. And at least one of the stereo confirmation information may be further transmitted.

  The fragment attribute information determination unit 120 according to an exemplary embodiment may determine identification information regarding each stereoscopic video section in one or more stereoscopic video sections of the current fragment.

  The fragment attribute information determination unit 120 according to an embodiment determines relationship information for referring to a plurality of videos when a reference relationship is formed between the plurality of videos including the current video. The relationship information according to an embodiment includes index information, position information, and the like of the reference video.

  The fragment attribute information determination unit 120 according to an exemplary embodiment determines position information of a reference video that is referred to for constructing a stereoscopic video regarding a video of a current file. The reference video location information according to an embodiment uses at least one of reference video identification information in the current file, remote location information other than the current file, URL (Uniform Resource Locator) information, and URN (Uniform Resource Name) information. Determined.

  The fragment attribute information determination unit 120 according to an embodiment determines reference video number information when a video of a current file refers to a plurality of videos. The fragment attribute information determination unit 120 according to an embodiment further determines at least one of reference index information for identifying a plurality of reference videos and position information regarding each reference video.

  The apparatus 100 for providing a multimedia streaming service according to an exemplary embodiment may provide a video that is differentially encoded according to video quality in order to securely transmit multimedia contents through a streaming service according to network conditions. The multimedia streaming service providing apparatus 100 according to an exemplary embodiment may differentially encode the same video according to video quality to generate a multi-channel video group including a plurality of channels of video.

  The fragment attribute information determination unit 120 according to an embodiment determines multi-channel number information indicating the number of videos of other channels that can replace the video of the current file. In addition, the fragment attribute information determination unit 120 according to an embodiment may determine reference information indicating a video position of another channel in the multi-channel video group.

  FIG. 2 is a block diagram of a multimedia streaming service receiver according to an embodiment.

  The multimedia streaming service receiver 200 according to an embodiment receives a fragment of streaming data using a network, and receives attribute information including at least one of three-dimensional signaling information and related information for each file, fragment, and sample. The corresponding video data is extracted to extract a 2D video, a 3D video, or a mixed video of 2D video and 3D video.

  The multimedia streaming service receiving apparatus 200 according to an embodiment includes a receiving unit 210, a video data extracting unit 220, a fragment attribute information extracting unit 230, and a restoring unit 240.

  The receiving unit 210, the video data extracting unit 220, the fragment attribute information extracting unit 230, and the restoring unit 240 according to an embodiment are controlled by a central processing unit or a processor located inside or outside the multimedia streaming service receiving device 200. , It operates organically by exchanging data with each other. Alternatively, at least one component of the reception unit 210, the video data extraction unit 220, the fragment attribute information extraction unit 230, and the restoration unit 240 according to an embodiment is operated by a processor mounted on itself, and other components And controlled to operate organically.

  The receiving unit 210 according to an embodiment continuously receives a plurality of fragments through network streaming.

  The video data extraction unit 220 according to an exemplary embodiment may include encoded data of a video including at least one of a 2D video and a 3D video from a media data area for each fragment received by the reception unit 210 in units of fragments. Extract in a divided form.

  The fragment attribute information extraction unit 230 according to an embodiment parses attribute information from the auxiliary data area for each fragment received by the reception unit 210. The fragment attribute information extraction unit 230 according to an embodiment includes information for reproducing a fragment video in two or three dimensions from the parsed attribute information, content-related characteristics, data structure, relationship information between a plurality of videos, and the like. Read attribute information.

  The restoration unit 240 according to an embodiment decodes the encoded video data of the fragment based on the three-dimensional signaling information or the relationship information read from the attribute information for each fragment by the fragment attribute information extraction unit 230. Restore to video or 3D video.

  The fragment attribute information extraction unit 230 according to an embodiment extracts at least one of the three-dimensional configuration information, the left-right order information, and the stereo mono section change count information set for the current video for each fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads a form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video of the current fragment from the three-dimensional configuration information of the current fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads the order of the left viewpoint video and the right viewpoint video in the stereoscopic video of the current fragment from the left-right order information of the current fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads the number of changes between the monoscopic video section and the stereoscopic video section of the current fragment from the stereo mono section change count information of the current fragment.

  The monoscopic video segment or the stereoscopic video segment of the current fragment is detected using the number of samples that make up each segment. The fragment attribute information extraction unit 230 according to an embodiment extracts at least one of stereo confirmation information and section sample number information for each fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads whether the current segment is a monoscopic video segment or a stereoscopic video segment from the stereo confirmation information of the current fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads the number of samples constituting the current section from the section sample number information of the current fragment.

  The fragment attribute information extraction unit 230 according to an embodiment extracts at least one of the three-dimensional configuration information and the left-right order information for each stereoscopic video section of the current fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads three-dimensional configuration information about samples constituting the current stereoscopic video section from the three-dimensional configuration information of the current stereoscopic video section of the fragment.

  The fragment attribute information extraction unit 230 according to an embodiment reads left / right order information about samples constituting the current stereoscopic video section from the left / right order information of the current stereoscopic video section of the fragment.

  The fragment attribute information extraction unit 230 according to an embodiment extracts stereoscopic camera display information for each stereoscopic video section of the current fragment video data.

  The fragment attribute information extraction unit 230 according to an embodiment obtains information about a stereoscopic camera that has acquired a sample constituting the current stereoscopic video section from the stereoscopic camera display information of the current stereoscopic video section, and a sample. Used to read information about display standards for safe 3D playback.

  The fragment attribute information extraction unit 230 according to an embodiment detects an item to which the stereoscopic camera display information is applied using the number of samples constituting each item in the stereoscopic video section.

  The fragment attribute information extraction unit 230 according to an embodiment includes information indicating the number of items to which the stereoscopic camera display information is applied in the current stereoscopic video section of the current fragment, and the current stereo referred to by each item. At least one of the identification information of the stereoscopic camera display information and the information about the length of the section to which the stereoscopic camera display information is currently applied is extracted and read.

  The fragment attribute information extraction unit 230 according to an exemplary embodiment uses the extracted information to detect an item to which the current stereoscopic camera display information is applied in the current stereoscopic video section.

  The fragment attribute information extraction unit 230 according to an exemplary embodiment extracts at least one of the three-dimensional configuration information regarding the video of the current file, left-right order information, and stereo mono-fragment change count information.

  The fragment attribute information extraction unit 230 according to an embodiment reads a form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video of the current file in the video of the current file from the three-dimensional configuration information of the current file. The fragment attribute information extraction unit 230 according to an embodiment reads the order of the left viewpoint video and the right viewpoint video constituting the stereoscopic video of the current file from the left / right order information regarding the video of the current file.

  The fragment attribute information extraction unit 230 according to an embodiment includes a mono fragment composed of a monoscopic video section and a stereoscopic video section of a video of the current file from the stereo mono fragment change count information of the current file. Read the number of changes with the stereo fragment.

  The fragment attribute information extraction unit 230 according to an exemplary embodiment extracts at least one of the 3D configuration information about the video of the current file, the left-right order information, and the stereo mono-fragment change count information. At least one of the stereo mono section change count information, the section sample number information, and the stereo confirmation information regarding the video may be extracted and read.

  In addition, the fragment attribute information extraction unit 230 according to an embodiment extracts and reads identification information about each stereoscopic video section in one or more stereoscopic video sections of the current fragment video data. A current stereoscopic video section is detected using the identification information regarding the read stereoscopic video section.

  The fragment attribute information extracting unit 230 according to an embodiment extracts and reads relation information indicating a reference relation between a plurality of videos when the video of the current file refers to one or more videos.

  The fragment attribute information extraction unit 230 according to an exemplary embodiment may include reference video position information, reference video identification information, remote information other than the current file, with respect to another video to be referred to in order to construct a stereoscopic video of the current file. At least one of position information, URL information, and URN information is extracted and read. The restoration unit 240 according to an exemplary embodiment may restore a stereoscopic video of the current file by detecting another video that is referred to using the read information.

  When the video data of the current file refers to a plurality of videos, the fragment attribute information extraction unit 230 according to an embodiment refers to the number of reference videos from the current file, the reference index information for identifying the plurality of reference videos, At least one of the positional information regarding the reference video is extracted and read. The restoration unit 240 according to an embodiment can restore the video of the current file in a desired form by selectively detecting the referenced video using the read information.

  The multimedia string service receiving apparatus 200 according to an embodiment includes a multi-channel including a plurality of channels of video encoded differentially according to video quality in order to securely receive multimedia content in a streaming service according to network conditions. An adaptive streaming service based on video groups is provided. The multimedia string service receiving apparatus 200 according to an exemplary embodiment receives video of a predetermined channel selectively according to a network state among videos differentiated into a plurality of channels according to video quality.

  The fragment attribute information extracting unit 230 according to an embodiment extracts at least one of multi-channel number information indicating the number of videos of other channels of the current file video and reference information indicating the video position of other channels. read. The restoration unit 240 according to an embodiment restores a video by combining the videos of other channels that are selectively received using the extracted multi-channel number information and reference information.

  The multimedia streaming service receiving apparatus 200 according to an embodiment may convert the 2D video or the 3D video restored by the restoration unit 240 into 2D or 2D based on information read from 3D signaling information or related information for each fragment. Play in 3D.

  FIG. 3 shows an mfsi box including a streaming file format transmitted in units of fragments and three-dimensional signaling information transmitted for each fragment according to an embodiment.

  A multimedia streaming service transmission apparatus 100 according to an embodiment includes a 2D video or 3D video encoded video data, and auxiliary data including various characteristic information about the transmitted video as a streaming file according to the embodiment. A data stream in the format 300 is recorded and transmitted.

  The streaming file format 300 according to an exemplary embodiment includes an ftyp box, a moov box 350, a sidx box, and a plurality of fragments 310, 312, 314. Each fragment 310, 320, 330 is composed of a moof box 320, 340, 360 and an mdat box 330, 350, 370.

  A streaming file format 300 according to an embodiment includes encoded video data of stereoscopic content. The ftyp box contains file type information of the current file in the streaming file format 300 according to one embodiment. The sidx box includes segment index information for searching each segment in the current file.

  The moov box 350 contains information about temporal and spatial information about the video of the current file, data structure, and the like. The moov box 350 of the streaming file format 300 according to an embodiment includes an svmi box 360 and a scdi box 370.

  The svmi box 360 includes various information such as stereoscopic visual type information of stereoscopic video transmitted through the current file in the streaming file format 300, mixed content of 2D / 3D content, and fragment related information. For example, fragment related information includes information regarding the number of fragments, the number of consecutive samples, and whether the current sample is stereoscopic data.

  The scdi box 370 includes various information for stereoscopic camera, display standard and visual safety regarding stereoscopic video transmitted through the current file in the streaming file format 300.

  The multimedia streaming service providing apparatus 100 according to an embodiment divides the encoded data of the current video into fragment units according to the streaming file format 300 for transmission. The mdat boxes 330, 350, and 370 of the respective fragments 310, 312, and 314 include video data divided into fragments. For example, the mdat box 330 includes divided data of the 2D video 335, the mdat box 350 includes divided data of the mixed video of the 2D video 352 and the 3D video 354, and the mdat box 350 includes It includes the split data of the mixed video of 3D video 372 and 2D video 374.

  In each of the fragments 310, 312, and 314, the moof boxes 320, 340, and 360 have temporal and spatial information and data structures related to videos inserted in the mdat boxes 350, 360, and 370 of the corresponding fragments 310, 312, and 314, respectively. Including information about

  For example, the moof box 340 includes 3D signaling information regarding the current video 352, 354 that is inserted into the mdat box 340 of the current fragment 312.

  For example, the moof box 340 includes an mfsi box 380 and a scdi box 390. The mfsi box 380 and the scdi box 390 according to one embodiment include three-dimensional signaling information regarding the current video 352, 354 that is inserted into the mdat box 340 of the current fragment 312.

  The multimedia streaming service receiving apparatus 200 according to an embodiment receives and parses a data stream according to the streaming file format 300 according to an embodiment, thereby receiving encoded video data of 2D video or 3D video, and receiving the data stream. Auxiliary data including various characteristic information regarding the recorded video is extracted.

  The multimedia streaming service receiver 200 according to an embodiment extracts 3D signaling information about a video from the mfsi box 380 and the scdi box 390 of the data stream according to the streaming file format 300 according to an embodiment, and performs 2D video or 3D. Restore the video.

  FIG. 4 illustrates the syntax of 3D signaling information included in the mfsi box and the scdi box according to an embodiment.

  The mfsi box according to one embodiment includes stereoscopic visual type information as 3D signaling information about the video inserted into the current fragment.

  The stereoscopic visual type information according to an embodiment includes three-dimensional configuration information 'stereoscopic_composition_type', left-right order information 'is_left_first', and stereo mono section change count information 'stereo_mono_change_count' regarding the current fragment video.

  The 3D configuration information 'stereoscopic_composition_type' according to an exemplary embodiment indicates a form in which a left-view image and a right-view image form a stereoscopic image in a current fragment video. The stereoscopic video type identified by the three-dimensional configuration information of the stereoscopic video according to an embodiment is a side-by-side image in which a left viewpoint video and a right viewpoint video are arranged in the left and right directions in a single stereoscopic video. Side (side-by-side) type, top and bottom (top-and-bottom) type arranged vertically, field (frame) sequential in which left viewpoint video and right viewpoint video are alternately transmitted one by one Type, and a type in which a left-view video sequence and a right-view video sequence are transmitted, respectively.

  The left-right order information ‘is_left_first’ according to an embodiment indicates an order in which the left-viewpoint image and the right-viewpoint image are arranged in the stereoscopic video of the current fragment. For example, by using the left / right order information 'is_left_first', the left viewpoint video and the right viewpoint video or the right viewpoint video and the left viewpoint video are displayed in the left / right area inside the stereoscopic video that is a side-by-side type. Arranged in order, or in the top / bottom area inside a stereoscopic video that is a top-and-bottom type, arranged in the order of left-view video and right-view video or right-view video and left-view video It is confirmed.

  The stereo mono section change count information 'stereo_mono_change_count' according to an embodiment includes a monoscopic video section and a stereoscopic video section when the current fragment includes a video composed of a monoscopic video section or a continuous section of a stereoscopic video section. Indicates the number of video segment changes.

  Stereo confirmation information 'stereo_flag' indicating whether the current fragment is a monoscopic video or a stereoscopic video is transmitted. The stereoscopic visual type information according to an embodiment includes stereo confirmation information 'stereo_flag' for indicating a monoscopic video section or a stereoscopic video section for each video section of the current fragment.

  Also, the stereoscopic visual type information according to an embodiment includes section sample number information “sample_count” indicating the number of samples constituting the current section for each video section of the current fragment.

  Also, the stereoscopic visual type information according to an embodiment includes three-dimensional configuration information ‘stereoscopic_composition_type’ information and left-right order information ‘is_left_first’ regarding the stereoscopic video section of the current fragment.

  Therefore, in the current fragment, for each stereoscopic video section, three-dimensional configuration information regarding the form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video, and the order in which the left viewpoint video and the right viewpoint video are arranged. And the left-right order information on is transmitted.

  As 3D signaling information according to an embodiment, a scdi box including at least one of stereoscopic camera information and stereoscopic display information is transmitted for each stereoscopic video section of the current fragment.

  Like the mfsi box syntax of FIG. 4, the scdi box according to an embodiment is included in the mfsi box according to an embodiment for transmission. As shown in FIG. 3, an mfsi box 380 and an scdi box 390 related to the current fragment video data are transmitted.

  The stereoscopic visual type information according to an exemplary embodiment includes 'scdi_item_count' information related to a stereoscopic video segment in the current fragment.

  'Scdi_item_count' information indicates the number of items to which the stereoscopic camera display information is applied. For example, an item to which stereoscopic camera display information is applied indicates a section including 3D video data. In this case, 'scdi_item_count' information indicates the number of items or samples that are 3D video data sections included in the current fragment.

  The scdi box according to an exemplary embodiment includes identification information 'scdi_ID' of current stereoscopic camera display information.

  The stereoscopic camera display information according to an embodiment may refer to the stereoscopic camera display information of the scdi box 370 of the moov box 350 or the stereoscopic camera display information of other scdi boxes of the moof boxes 320, 340, and 360. Therefore, the scdi box that is the reference object is searched using 'scdi_ID'.

  For each item or sample to which the stereoscopic camera display information is applied, 'scdi_ID' information and 'scdi_duration' information are transmitted. The 'scdi_duration' information indicates the length of the item or sample to which the current stereoscopic camera display information is applied. Accordingly, the current stereoscopic camera display information regarding the current fragment is transmitted in units of items or samples constituting the stereoscopic video segment.

  Therefore, based on the multimedia streaming file format 300 according to the embodiment of FIG. 3 and the syntax of the msfi box and scdi box of FIG. 4, video including monoscopic video or stereoscopic video is transmitted and received in fragments. , 3D signaling information about video is transmitted and received for each fragment.

  In addition, information indicating the number of samples constituting each section is transmitted / received to / from the monoscopic video section or the stereoscopic video section in the fragment. Also, stereoscopic camera display information regarding the fragment video is transmitted in units of items or samples.

  FIG. 5 illustrates an svfi box including a streaming file format transmitted in units of fragments and three-dimensional signaling information transmitted for each fragment according to an embodiment.

  A multimedia streaming service transmission apparatus 100 according to an embodiment includes a 2D video or 3D video encoded video data, and auxiliary data including various characteristic information about the transmitted video as a streaming file according to the embodiment. It is recorded in a data stream of format 500 and transmitted.

  The streaming file format 500 according to an exemplary embodiment includes a plurality of fragments 501, 502 and an mfra box including an ftyp box, a moov box 510, and a moof box 520, 540 and an mdat box 530, 550.

  The track box included in the moov box 510 includes information for searching for one track in the presentation. A presentation refers to one or more motion sequences that can be combined with audio, and a track refers to a collection of samples associated with each other, for example, a video sequence in a video.

  The tkhd box included in the trak box is a track header including all information related to the current track, and the mdia box includes declaration information regarding the medium of the current track. The minf box included in the mdia box includes any object that declares characteristic information about the current media, and the stbl box includes indexing information such as sample time, data, and position for searching the current track sample. The svmi box of the stbl box contains information about the stereoscopic video for the current media of the current track.

  The mvex box included in the moov box 510 includes information indicating that a video fragment is transmitted through the current file. The trex box included in the mvex box includes basic value information regarding the samples used in the fragments 501 and 502.

  Through the mdat boxes 530, 550 of the fragments 501, 502, encoded data of monoscopic video, stereoscopic video, or mixed video is transmitted. The mfhd box included in the moof boxes 530 and 550 includes information regarding the serial number for each fragment in the current file.

  The traf box included in the moof boxes 530 and 550 includes one or more track fragments constituting each track among the current fragments. Each track fragment includes one or more track runs that describe a series of samples. The run box included in the traf box includes information on the current track run.

  The mfra box contains random access point information to approach each fragment randomly. The trfa box includes position information and playback time information of a sample that can be randomly accessed.

  The multimedia streaming service providing apparatus 100 according to an embodiment transmits svfi boxes 525 and 545 including 3D signaling information transmitted for each fragment through a streaming file format 500 transmitted in units of fragments.

  The svfi boxes 525 and 545 according to an embodiment record three-dimensional signaling information regarding the videos 535 and 555 inserted in the mdat boxes 530 and 550 of the current fragments 501 and 502, respectively.

  For example, the multimedia streaming service providing apparatus 100 may insert the svfi box into the traf box, which is a lower box of the moof boxes 530 and 550, and transmit the result.

  The multimedia streaming service receiver 200 according to an embodiment receives and parses a data stream according to the streaming file format 500 according to an embodiment, thereby receiving encoded video data of 2D video or 3D video, and receiving the data stream. And auxiliary data including various characteristic information on the video data.

  The multimedia streaming service receiving apparatus 200 according to an embodiment extracts the 3D signaling information about the video data from the svfi boxes 525, 545, the svmi box, etc. of the data stream according to the streaming file format 500 according to the embodiment to extract the 2D video. Or restore the 3D video.

  6 and 7 illustrate the syntax of the 3D signaling information included in the svfi box according to an embodiment.

  Through the svfi box according to an embodiment, 3D signaling information about the video transmitted through the current fragment is transmitted and received.

  The three-dimensional signaling information of the svfi box shown in FIGS. 6 and 7 includes stereo mono section change count information 'stereo_mono_change_count' regarding the current fragment video. Through the stereo mono section change count information 'stereo_mono_change_count', a monoscopic video section and a stereoscopic video section are searched from the current fragment. The three-dimensional signaling information of the svfi box according to an embodiment includes section sample number information ‘sample_count’ and stereo confirmation information ‘stereo_flag’ with respect to a stereoscopic video section among the current fragments.

  The three-dimensional signaling information of the svfi box shown in FIG. 7 further includes information 'stereo_section_ID' for identifying a stereoscopic video period. If the current fragment includes a plurality of stereoscopic video sections, each stereoscopic video section is distinguished using stereo section identification information 'stereo_section_ID'.

  FIG. 8 illustrates a syntax of 3D signaling information set for a file in a streaming file format transmitted in fragment units according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment can transmit 3D configuration information and left / right order information from a file channel of the streaming file format 500. Three-dimensional configuration information ‘stereoscopic_composition_type’ and left-right order information ‘is_left_first’ regarding the video of the current file are inserted into the svmi box included in the moov box.

  The multimedia streaming service receiving apparatus 200 according to an embodiment parses the svmi box included in the moov box of the streaming file format 500, extracts the 'stereoscopic_composition_type' field and the 'is_left_first' field, and receives the field through the streaming file format 500. A configuration in which a stereoscopic video is formed with respect to the video of the current file and information regarding the order of the left viewpoint video and the right viewpoint video in the stereoscopic video are read.

  FIG. 9 illustrates a syntax of 3D signaling information set for a sample in a streaming file format transmitted in fragment units according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment transmits stereo confirmation information from a track, fragment, or sample channel of the streaming file format 500.

  The syntax of FIG. 9 is a form in which stereo confirmation information 'stereo_flag' according to an embodiment is added to the syntax of a trex box included in a moov box, a tfhd box included in a moof box, or a trun box.

  That is, stereo confirmation information z′stereo_flag ’regarding the video of the current track is inserted into the trex box included in the moov box and transmitted. Stereo confirmation information 'stereo_flag' regarding the video of the current fragment is inserted into the tfhd box included in the moof box and transmitted. Alternatively, stereo confirmation information 'stereo_flag' relating to the video of the sample constituting the current fragment is inserted into the trun box included in the moof box and transmitted.

  The multimedia streaming service receiver 200 according to an embodiment parses the trex box included in the moov box of the streaming file format 500, the tfhd box included in the moof box, or the trun box to extract 'stereo_flag', and performs streaming. Read whether the current track, current fragment or current sample video received through file format 500 is monoscopic or stereoscopic.

  FIG. 10 illustrates two streaming file formats for transmitting stereoscopic video left-view video and right-view video, respectively, according to one embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment transmits a left-view ES stream (Elementary Stream) and a right-view ES stream each including a left-view video and a right-view video that form a stereoscopic video. In addition, the multimedia streaming service providing apparatus 100 according to an embodiment may include a left-view video file 'test_1644_L. mp4'1010, the left viewpoint ES stream is inserted, and the right viewpoint ES stream is individually added to the right viewpoint video file 'test_1644_R. The right viewpoint ES stream is inserted into mp4′1020 and transmitted.

  That is, the multimedia streaming service providing apparatus 100 according to an exemplary embodiment transmits two ES streams through streaming of two files in order to transmit stereoscopic video.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment transmits related information including reference information and position information regarding another video to be referred to in order to construct a stereoscopic video of a current file. For example, the multimedia streaming service providing apparatus 100 according to the embodiment inserts and transmits a reference type related to a reference video, position information, and the like in a trak box 1040 that is a lower box of the moov box 1030.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment parses the received left-view video file and extracts reference types and position information regarding other videos to be referred to in constructing a stereoscopic video of the current file. And read.

  The multimedia streaming service receiving apparatus 200 according to an exemplary embodiment detects a streaming file in which another video referred to using the read information is received, and thereby extracts the video extracted from the two streaming files. Combine to restore stereoscopic video.

  FIG. 11 shows relationship information set for a stereoscopic video included in a file according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment performs other operations to configure a stereoscopic video in a tref box 1110 included in a trak box that is a lower box of a moov box of the streaming file format 500 according to an embodiment. Insert relationship information to search for videos.

  The track reference type box 1120 included in the trak box according to an exemplary embodiment includes reference track identification information 'track_ID' for indicating a track including a reference video for the video of the current track for each track of the file.

  In addition, the track reference type URL box 1130 included in the trak box according to an exemplary embodiment includes information 'entry_count' indicating the number of reference videos in order to indicate the positions of a plurality of reference videos with respect to the current track for each track of the file. Location information 'location' is included for each reference video.

  The location information 'location' according to an embodiment is a character string encoded in a language such as UTF-8 and indicates a remote location of the reference video. Accordingly, the multimedia streaming service receiving apparatus 200 according to an embodiment parses the moov box, the trak box, and the tref box 1110 sequentially from the received streaming file, and then extracts the external file of the current file from the track reference type URL box 1130. May be detected and referenced.

  FIG. 12 illustrates one piece of relation information set in a file when transmitting stereoscopic video using two streaming files according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment includes a left-view video file 'test_1644_L. Information related to one video referred to by the video of the current track is inserted and transmitted in a dinf box 1060 which is a lower box of the minf box 1050 of the trak box 1040 included in the moov box 1030 of mp4′1010.

  The dinf box 1060 according to an embodiment includes a dref box 1070 for transmitting related information, and the dref box 1070 includes reference URL information 1080 indicating remote position information of a reference track of the current track.

  The reference URL information 1080 according to an embodiment includes not only the URL information “location” but also a reference flag “flags”. Through the reference flag 'flags' according to an embodiment, the current video is a left-view video or a basic time-point video among stereoscopic videos, and is described as being included in the current file, and the right-view as a reference target. The video or the additional time point video is described as being included in the position indicated by “location”.

  That is, the URL information ‘location’ of the reference URL information 1080 includes the right-view video file ‘test — 1644_R. The remote location of mp4'1020 is shown. The 'flags' of the reference URL information 1080 indicates that the current video is the current left viewpoint video file' test_1644_L. mp4'1010 and the reference video of the current video is a right-view video file 'test_1644_R. It shows that it is transmitted through mp4'1020.

  According to one embodiment, the left viewpoint video file 'test_1644_L. When the related information is recorded only in mp4'1010, the left viewpoint video file 'test_1644_L. When a fragment of mp4'1010 is requested, the right-view video file 'test_1644_R. It is recognized that the corresponding fragment of mp4'1020 is also requested.

  FIG. 13 illustrates a plurality of pieces of related information set in a file when transmitting stereoscopic video using two streaming files according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment includes a left-view video file 'test_1644_L. Reference information relating to a plurality of videos currently referred to by a video is inserted and transmitted in a dinf box 1360 included in the moov box 1310 of mp4′1010.

  For example, the dref box 1370 of the dinf box 1360 includes two pieces of reference URL information 1380 and 1390 indicating remote position information of two reference videos of the current video.

  The first reference URL information 1380 included in the dref box 1370 according to the embodiment includes the left-view video file 'test_1644_L. mp4'1010 of the current video first fragment 1320 refers to the right-view video file 'test_1644_R. Contains information to indicate the third fragment 1340 of mp4'1020.

  In a similar manner, the second reference URL information 1390 included in the dref box 1370 according to an exemplary embodiment is the left-view video file 'test_1644_L. mp4'1010, the right-view video file 'test_1644_R. Contains information to indicate the fourth fragment 1350 of mp4'1020.

  The multimedia streaming service receiver 200 according to an embodiment includes a left-view video file 'test_1644_L. The mp4'1010 is received, and the moov box 1310, the dinf box 1360, and the dref box 1370 are sequentially parsed to extract the first reference URL information 1380 and the second reference URL information 1390.

  The multimedia streaming service receiving apparatus 200 according to an exemplary embodiment may determine that the first fragment 1320 of the current track is the current left-view video file 'test_1644_L. mp4'1010 and the reference track of the first fragment 1320 of the current track is a right-view video file 'test_1644_R. It can be read as being transmitted through the third fragment 1340 of mp4'1020.

  In a similar manner, the multimedia streaming service receiving apparatus 200 according to an embodiment determines that the second fragment 1330 of the current track is based on the 'flags' of the second reference URL information 1390 and the current left-view video file' test_1644_L. mp4'1010 and the reference track of the second fragment 1330 of the current track is a right-view video file 'test_1644_R. It can be read as being transmitted through the fourth fragment 1350 of mp4'1020.

  FIG. 14 illustrates a syntax of track information including related information regarding a plurality of reference videos when transmitting stereoscopic video in two streaming files according to an embodiment.

  In the multimedia streaming service providing apparatus 100 according to an embodiment, when there are a plurality of reference videos for a current track of a file, the reference video number and the position of each reference video are associated with each other, and an index of the position information of the reference video is obtained. The information shown can be transmitted.

  For example, the information 'entry_count' indicating the number of reference videos in the dref box, which is a lower box of the dinf box, is linked with the position information 'location' of each reference video to indicate the index of the position information of the reference video. Data_reference_index ′ information is transmitted through the dref box.

  FIG. 15 illustrates a syntax of relation information set in a file when transmitting stereoscopic video using two streaming files according to an exemplary embodiment.

  The multimedia streaming service providing apparatus 200 according to an embodiment uses a reference video location box 1510, a reference video name box 1520, and a reference entry box 1530 to inform a reference video of a current video.

  For example, the reference video position box 1510 includes position information based on http and ftp as the position information ‘location’ of the reference video. The reference video name box 1520 according to an exemplary embodiment includes URN information 'name' and position information 'location' of the reference video.

  Through the reference entry box 1530 according to an embodiment, reference information “data_entry” determined based on reference entry number information “entry_count” and information “entry_flags” indicating an entry type for each reference entry is transmitted.

  For example, 'entry_flags = 0x000001' indicates that the current video is inserted in the current file. If the entry flag according to an embodiment is entry_flags = 0x000002 ′, it is described that the current video is a left-viewpoint video or a basic point-in-time video among stereoscopic videos, and is included in the current file. It is described that a certain right viewpoint video or additional time point video is included in a position indicated by “location”.

  FIG. 16 illustrates relationship information set using an index file outside a file when transmitting stereoscopic video using two streaming files according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an embodiment may transmit related information for indicating a reference video of a current video using an external index file that describes index information related to the external file.

  Information about a plurality of components for one content is transmitted through the external index file SMIL according to an embodiment. That is, the external index file SMIL according to an embodiment includes both index information 1010 for the current file and index information 1020 for the reference file of the current file.

  The multimedia streaming service receiving apparatus 200 according to an embodiment extracts, from a received streaming file, related information on a reference video that a video of a current file refers to make up a stereoscopic video, A reference index for identifying a plurality of reference videos and a position with respect to each reference video are read. The multimedia streaming service receiver 200 according to an embodiment detects a reference video and combines it with the current video to restore a stereoscopic video.

  In addition, the multimedia streaming service receiving apparatus 200 according to an embodiment uses an external index file to detect a reference video for a current video and combine it with the current video to restore a stereoscopic video.

  17 and 18 illustrate a single streaming file format for transmitting left and right viewpoint video of stereoscopic video according to an embodiment.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment transmits a left-view ES stream and a right-view ES stream for stereoscopic video through one fragment-based streaming file.

  The multimedia streaming service providing apparatus 100 according to the embodiment inserts and transmits both the left-view ES stream and the right-view ES stream in the streaming files 1700 and 1800 based on the streaming file format 500 according to the embodiment.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment transmits related information related to other videos to be referred to to compose a stereoscopic video of a current file. For example, the multimedia streaming service providing apparatus 100 according to an embodiment inserts and transmits related information in the trak boxes 1712, 1714, 1812, and 1814, which are lower boxes of the moov boxes 1710 and 1810.

  The multimedia streaming service providing apparatus 100 according to an embodiment parses the received left-view video file, and extracts and reads related information regarding other videos to be referred to in constructing the stereoscopic video of the current file. .

  The multimedia streaming service receiving apparatus 200 according to an embodiment combines a video extracted from two streaming files by detecting a streaming file including one other video referred to by using the read relationship information. To restore stereoscopic video.

  First, referring to FIG. 17, each fragment of the streaming file 1700 includes both a left-view video as a first track and a right-view video as a second track. That is, the left-view video 1722 and the right-view video 1724 are transmitted through the first fragment 1720, and the left-view video 1732 and the right-view video 1734 are transmitted through the second fragment 1730.

  The moov box 1710 includes, for each track, a first trak box 1712 for the first track and a second trak box 1714 for the second track. Of the stereoscopic video, the related information according to the embodiment is inserted and transmitted in the tref box 1715 of the second trak box 1714 related to the second track in which the right viewpoint video for the additional time point video is transmitted. In the related information according to an exemplary embodiment, the tref box 1715 includes reference type information 'reference_type' and basic time track identification information 'track_ID'.

  Referring to FIG. 18, for each fragment of the streaming file 1800, the left-view video as the first track and the right-view video as the second track are individually inserted. That is, the left-view video 1822 is transmitted through the first fragment 1820, the right-view video 1832 is transmitted through the second fragment 1830, the left-view video 1842 is transmitted through the third fragment 1840, and the right-view video 1852 is transmitted through the fourth fragment 1850. Is transmitted.

  In the case of FIG. 18, the relationship information according to an embodiment may include offset information regarding a fragment including a right-view video referred to by the left-view video.

  If the offset information regarding the fragment in which the right-view video is transmitted is not transmitted separately as the related information, the corresponding left-view video fragments 1820 and 1840 and the right-view video fragments 1830 and 1850 are included in the streaming file 1800. It is desirable to insert and transmit sequentially. Accordingly, the multimedia streaming service receiver 200 according to the embodiment receives the right-view video fragments 1830 and 1850 immediately after the left-view video fragments 1820 and 1840 while receiving the streaming file 1800.

  The moov box 1710 includes, for each track, a first trak box 1712 for the first track and a second trak box 1714 for the second track. Of the stereoscopic video, the related information according to the embodiment is inserted and transmitted in the tref box 1715 of the second trak box 1714 related to the second track in which the right viewpoint video for the additional time point video is transmitted. In the reference information according to the embodiment, the tref box 1715 includes reference type information 'reference_type' and basic time point track identification information 'track_ID'.

  The multimedia streaming service receiver 200 according to an embodiment sequentially parses the moov boxes 1710 and 1810 and the trak boxes 1712, 1714, 1812 and 1814 of the received streaming files 1700 and 1800, and the tref boxes 1715 and 1815 It is read that the right-view video as the additional time point video is included in the second track corresponding to the second trak boxes 1714 and 1814 to be extracted. Also, since the second track is read as belonging to the first track based on the reference type information 'reference_type' and the basic time point track identification information 'track_ID' in the tref boxes 1715 and 1815, it corresponds to the first trak boxes 1712 and 1812. The left viewpoint video is extracted from the fragment of the first track, and the right viewpoint video is extracted from the fragment of the second track corresponding to the second trak boxes 1714 and 1814.

  The tfra boxes 1762 and 1862, which are subordinate boxes of the mfra boxes 1760 and 1860, include fragment offset information indicating the distance from the playback time of the stereoscopic video to the fragment. Since the playback time for the left viewpoint video and the right viewpoint video of the stereoscopic video is the same, the fragment offset information for extracting the stereoscopic video of the streaming file 1700 is only included in the tfra boxes 1762 and 1862 for the first track. Provided.

  19 and 20 show a method of determining a reference video using an index file outside the file when transmitting stereoscopic video using a single streaming file.

  The multimedia streaming service providing apparatus 100 according to an embodiment may transmit related information for indicating a reference video of a current video using an external index file.

  Information about a plurality of components for one content is transmitted through the external index file SMIL according to an embodiment. That is, the external index file SMIL according to an embodiment includes both index information for the left-view video track and index information for the right-view video track transmitted through the current file “test — 1644.mp4”.

  Referring to FIG. 19, the index file outside the file is used to identify the left-view video track and the right-view video track included in the current streaming file, with track identification information 'trackID' and stereotype 'stereotype'. 'including.

  Referring to FIG. 20, a reference video is searched using an index file outside the file and a tref box inside the file. The index file outside the file includes only track identification information 'trackID' and stereotype 'stereotype' for identifying the basic time point video track included in the current streaming file.

  In the case of FIG. 20, the multimedia streaming service receiver 200 according to an embodiment includes a trak box including a tref box among the trak boxes of the moov box of the received streaming file. It can be read that the track corresponding to the box is the additional time point video track. Therefore, the basic time point video is extracted from the track fragment indicated by the index file outside the file, and the additional time point video is extracted from the track fragment corresponding to the trak box including the tref box.

  The multimedia streaming service receiving apparatus 200 according to an exemplary embodiment extracts a basic time point video and an additional time point video constituting a stereoscopic video from the received streaming file, and extracts the basic time point video and the additional time point video of the current streaming file. The index information is extracted to read the reference relationship between the basic time point video and the additional time point video. The multimedia streaming service receiver 200 according to an exemplary embodiment restores a stereoscopic video by combining the base time video and the additional time video extracted from the current streaming file.

  Also, the multimedia streaming service receiving apparatus 200 according to an embodiment reads the relationship between the basic time point video and the additional time point video of the current streaming file using the external index file, and combines the basic time point video and the additional time point video. The stereoscopic video may be restored.

  As described above, in order to reproduce the stereoscopic video transmitted / received through the fragment-based streaming file described above with reference to FIGS. 10 to 20, the left-view video and the right-view video must be synchronized. The left-view video and the right-view video are synchronized by having the same playback time and the same sample duration between the corresponding left-view sample and right-view sample.

  Hereinafter, with reference to FIG. 21 and FIG. 22, a method of using relation information between video data constituting a multi-channel video group according to an embodiment will be described in detail for a multi-channel video group for an adaptive streaming service. Is done.

  For multimedia adaptive streaming services, multi-channel video groups are provided to provide video that is differentially encoded according to video quality or bit rate. Encoded video data of a channel selected from the multi-channel video group is transmitted / received.

  The multimedia streaming service receiving apparatus 200 according to an embodiment requests the multimedia streaming service providing apparatus 100 for a video of a predetermined channel in a multi-channel video group in consideration of a network environment, a current state, capability, and the like.

  The multimedia streaming service providing apparatus 100 according to an exemplary embodiment includes information related to videos constituting a multi-channel video group so that the multimedia streaming service receiving apparatus 100 can select a desired type of video from the multi-channel video group. To transmit.

  The multimedia streaming service receiver 200 according to an exemplary embodiment selects, requests, and receives a video of a desired channel using the multi-channel video group relationship information according to an exemplary embodiment. The multimedia streaming service receiver 200 according to an embodiment restores a video using the received video of a predetermined channel.

  FIG. 21 illustrates afgp box syntax including relationship information of a multi-channel video group according to an embodiment for an adaptive streaming service.

  The multimedia streaming service providing apparatus 100 according to an embodiment adds afgp boxes to the multimedia streaming file formats 300 and 500, and transmits related information related to the multi-channel video group related to the video to be transmitted through the current streaming file.

  The afgp box according to an embodiment is configured in the same channel as the ftyp box, the moov box, and the like.

  The afgp box according to an exemplary embodiment includes related information about other videos in the multi-channel video group related to the current video that can replace the current video. The afgp box according to an embodiment includes information about the number of videos constituting the multi-channel video group ‘entry_count’ and position information ‘file_location’ of other videos.

  The location information 'file_location' of the afgp box according to an embodiment indicates a remote location, URL, etc. of another video. That is, the afgp box according to an embodiment transmits position information about a video located in an external file as well as a current streaming file.

  The position information 'file_location' is a character string encoded in a language such as UTF-8. By sharing the value of the multi-channel video group name information 'alternate_group' to indicate that the external file indicated by the position information 'file_location' and the current file are the same multi-channel video group, the current file and the external file are shared. A replacement relationship is expressed.

  For example, if the 'alternate_group' value of the video of the current file including the afgp box and the 'alternate_group' value of the video included in the predetermined external file described by the 'file_locations' of the afgp box are not the same value, The video of the current file and the video of the predetermined external file indicate that they are interchangeable videos constituting a multi-channel video group.

  The multimedia streaming service receiving apparatus 200 according to an embodiment parses the afgp box according to an embodiment from the received streaming file, and reads the number of videos and the position information constituting the multi-channel video group.

  The multimedia streaming service receiving apparatus 200 according to an exemplary embodiment selects a video of a desired channel from videos constituting a multi-channel video group read from an afgp box, and is located in a currently received file or an external file. Request and receive video.

  FIG. 22 illustrates a case in which relationship information between multi-channel video groups is set using an index file external to a file for an adaptive streaming service.

  In the multimedia streaming service providing apparatus 100 according to an embodiment, information related to a plurality of videos constituting a multi-channel video group is transmitted together using an external index file. For example, the external index file SMIL according to an embodiment includes both index information 2210 regarding the first file and index information 2220 regarding the second file for each video of the multi-channel video group.

  The external index file SMIL according to an embodiment includes video position information test — 1644. mp4, test_3288. mp4, bit rate information 1644000 or 3288000, and group name information alternate_group for identifying the current multi-channel video group.

  The external index file according to an embodiment includes remote location information or URL information of a currently received file or an external file as location information regarding video data included in the multi-channel video group.

  In addition, if the first file “test — 1644.mp4” is a video track inside the currently received file, the external index file SMIL according to an embodiment may further include track identification information trackID.

  Accordingly, the multimedia streaming service receiver 200 according to an embodiment reads the position information and characteristic information of the video of the multi-channel video group to which the current video belongs from the external index file of the received streaming file. The multimedia streaming service receiver 200 according to an exemplary embodiment selects a replacement video from a multi-channel video group read from an external index file, and requests a currently received file or a video located in the external file. Receive.

  FIG. 23 shows a flowchart of a multimedia streaming service providing method according to an embodiment. In step 2310, video encoded data including at least one of 2D video data and 3D video data is divided into fragment units.

  According to one embodiment, the fragment includes one of a monoscopic video segment, a stereoscopic video segment, and a mixed video segment of a monoscopic video segment and a stereoscopic video segment.

  In step 2320, attribute information including at least one of the three-dimensional signaling information and the relationship information is determined for each fragment.

  According to one embodiment, at least one of three-dimensional configuration information, left-right order information, and stereo mono section change count information is determined for each fragment.

  According to one embodiment, at least one of stereo confirmation information and section sample number information is determined for each section of a fragment. For each stereoscopic video section of the fragment, sample three-dimensional configuration information and left-right order information are determined.

  According to one embodiment, stereoscopic camera display information about a sample is defined for each stereoscopic video segment of the fragment. The stereoscopic camera display information according to an embodiment is expressed using an item to which the stereoscopic camera display information is applied.

  According to an exemplary embodiment, at least one of three-dimensional configuration information, left-right order information, and stereo mono-fragment change count information is determined for the current file video. At least one of stereo mono section change count information, section sample number information, and stereo confirmation information is further determined for each fragment together with the three-dimensional configuration information, left-right order information, and stereo mono-fragment change count information regarding the file. According to one embodiment, stereoscopic video segment identification information is defined for each stereoscopic video segment of the current fragment.

  According to one embodiment, the reference video located outside the current file as well as the location information of the reference video in the current file is used as relation information for searching the reference video constituting the stereoscopic video according to the current file. At least one of remote location information, URL information, and URN information is used.

  According to an embodiment, when the video of the current file refers to a plurality of videos, at least one of reference video number information, reference index information for identifying the plurality of reference videos, and position information is determined.

  For an adaptive streaming service according to an embodiment, a multi-channel video group is generated that provides multiple channels of video differentially according to video quality. In this case, at least one of information on the number of videos constituting the multi-channel video group and reference information indicating video positions of other channels is determined. In particular, the position information of the videos constituting the multi-channel video group may indicate the position of the video existing in the file outside the current file.

  In step 2330, fragments including attribute information about the fragmented video and the encoded video data are continuously transmitted. The attribute information about the fragment is inserted and transmitted in the auxiliary data area of the fragment, and the encoded video data divided into fragments is transmitted by being inserted into the media data area of the fragment.

  FIG. 24 shows a flowchart of a multimedia streaming service receiving method according to an embodiment. In step 2410, multiple fragments are received sequentially. In step 2420, encoded video data of a fragment composed of a two-dimensional video section or a three-dimensional video section is extracted from the media data area for each fragment.

  In step 2430, attribute information is parsed from the auxiliary data area for each fragment, and two-dimensional reproduction of the fragment video data or three-dimensional signaling information for three-dimensional reproduction, and relation information for reference between a plurality of videos. At least one is extracted and read.

  According to an exemplary embodiment, the three-dimensional configuration information of the fragment is extracted, and a form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video of the current fragment is read. According to an embodiment, the left-right order information of the fragments is extracted, and the order of the left viewpoint video and the right viewpoint video is read from the stereoscopic video of the current fragment.

  According to one embodiment, the stereo mono section change count information of a fragment is extracted, and the change count of the current fragment monoscopic video section and stereoscopic video section is read.

  According to an embodiment, at least one of the fragment stereo confirmation information and the section sample number information is extracted, and the monoscopic video section or the stereoscopic video section of the fragment is distinguished.

  According to an exemplary embodiment, at least one of the 3D configuration information and the left-right order information is extracted for each stereoscopic video section of the fragment, and the 3D configuration information regarding the samples constituting the current stereoscopic video section is read. The left-right order information is read.

  According to one embodiment, for each stereoscopic video section of the fragment, stereoscopic camera display information is extracted to read information about the stereoscopic camera and display standard that acquired the samples that currently make up the stereoscopic video section. It is done. The stereoscopic camera display information according to one embodiment is read relative to the target item to be applied.

  According to an exemplary embodiment, at least one of 3D configuration information, left-right order information, and stereo mono-fragment change count information regarding the video data of the current file is extracted, and for each fragment of the current file, the stereo mono-section change count regarding the fragment video At least one of information, section sample number information, and stereo confirmation information is further extracted and read.

  According to an exemplary embodiment, identification information about each stereoscopic video segment is extracted from one or more stereoscopic video segments of the fragment to distinguish the stereoscopic video segment.

  According to an exemplary embodiment, at least one of the reference video position information, the reference video identification information, the remote position information of the reference video located outside the current file, the URL information, and the URN information is extracted and read. Video data and other one video to be referenced are searched.

  According to an embodiment, when the video of the current file refers to a plurality of videos, at least one of the reference video number, the reference index, and the position is read, and the referenced video is selectively detected.

  In order to receive an adaptive streaming service according to an embodiment, related information regarding a multi-channel video group providing video that is differentially encoded into multiple channels according to video quality is received. According to an embodiment, related information regarding a multi-channel video group is received, and a desired replacement video is selectively requested and received based on the relationship information and location information regarding the replacement video. The position information regarding the replacement video may include not only the identification information regarding the video inside the current file but also the position information of the video existing outside the current file.

  In step 2440, the video data extracted for each fragment is decoded and combined based on the information read from the three-dimensional signaling information or the relationship information for each fragment, and restored to the two-dimensional video or the three-dimensional video. Using the 3D signaling information or relationship information according to one embodiment, the recovered video may be played back in 2D or 3D.

  The multimedia streaming service providing apparatus 100 according to an embodiment and the multimedia streaming service receiving apparatus 200 according to an embodiment are stereoscopically processed in a file unit, a fragment unit, or a sample unit through a fragment-based streaming file format according to an embodiment. Transmit and receive video 3D signaling information.

  In addition, through one or more streaming files based on a fragment-based streaming file format according to an embodiment, reference information, characteristic information, and the like between a plurality of videos, such as stereoscopic basic time point video and additional time point video, may be used. Related information is sent and received.

  This allows you to reference an existing file or a file that exists elsewhere, so you can regenerate the file to transmit the reference video, or link or combine multiple videos without physical constraints Service is provided.

  For an adaptive streaming (HTTP Adaptive Streaming) service using Hypertext Transfer Protocol (HTTP), in a unified form to be compatible with an ISO-based media file format using a streaming file format according to an embodiment, Relationship information and three-dimensional signaling information between a plurality of videos constituting a stereoscopic video are provided. In addition to multimedia such as video and audio encoded in multiple channel quality, stereoscopic multimedia is provided on one or more multimedia files to improve transmission bandwidth and terminal device performance. In addition, adaptively variable quality multimedia is provided to the user seamlessly.

  The block diagram disclosed in the present invention is interpreted by those skilled in the art as a conceptual representation of a circuit for embodying the principles of the present invention. Similarly, any flowcharts, state transition diagrams, pseudocode, etc. may be substantially represented on a computer-readable medium to represent the various types performed by such a computer or processor as soon as the computer or processor is explicitly illustrated. It will be appreciated by those skilled in the art that it represents a simple process. Therefore, the above-described embodiment of the present invention can be created in a computer-implemented program and is implemented by a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes a recording medium such as a magnetic recording medium (for example, ROM, floppy (registered trademark) disk, hard disk, etc.), an optically readable medium (for example, CD-ROM, DVD, etc.). .

  The functions of various elements shown in the drawings are provided not only by hardware that can execute software but also by use of dedicated hardware with respect to appropriate software. When provided by a processor, such functionality is provided by a single dedicated processor, a single shared processor, or multiple individual processors that are partially shared. Also, the explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of performing software, without limitation digital signal processor (DSP) hardware, software Implicitly includes a read only memory (ROM), a random access memory (RAM), and a non-volatile storage device.

  In the claims of this specification, an element expressed as a means for performing a specific function includes any system that performs the specific function, and these elements perform a combination of circuit elements that perform the specific function, or perform a specific function. Including any form of software, including firmware, microcode, etc., coupled with circuitry suitable for performing software.

  In this specification, 'one embodiment' of the principle of the present invention and designation of various variations of such expressions are related to this embodiment, and specific features, structures, characteristics, etc. Means included in the form. Thus, the expression 'one embodiment' and any other variations disclosed throughout this specification are not necessarily all referring to the same embodiment.

  In the present specification, in the case of “at least one of A and B”, the expression “at least one of” is only the selection of the first option (A) or the second option (B ) Selection only, or to encompass the selection of both options (A and B). As an additional example, in the case of 'at least one of A, B, and C', only the first listed option (A) is selected, or the second listed option (B) is selected. Only, or the selection of the third listed option (C), or only the selection of the first and second listed options (A and B), or the second and third listed options (B and C) selection only or all three option selections (A, B and C). If there are more items, they will obviously be extended to those skilled in the art. In the above, this invention was demonstrated centering on the desirable embodiment.

  All embodiments and conditional illustrations disclosed throughout this specification are written with the intention of assisting the reader in understanding the principles and concepts of the present invention by those skilled in the art. It will be understood that the invention can be embodied in a modified form without departing from the essential characteristics of the present invention. Accordingly, the disclosed embodiments should be considered in an illustrative rather than a limiting perspective. The scope of the present invention is shown not by the above description but by the claims, and all differences within the equivalent scope should be construed as being included in the present invention.

Claims (15)

In a streaming service providing method for transmitting multimedia using a network,
Dividing encoded data of a video including at least one of 2D video and 3D video into fragments;
An attribute including both three-dimensional signaling information including information for two-dimensional reproduction or three-dimensional reproduction of the current video and relation information for the video to refer to one or more other videos for each fragment. Determining information,
Continuously transmitting a fragment composed of an auxiliary data area including attribute information about the fragment and a media data area including the encoded video data;
The three-dimensional signaling information and the relationship information are transmitted through the auxiliary data area, and the encoded video data is transmitted through the media data area.
A method of providing a multimedia streaming service. Determining the attribute information for each fragment includes
For each fragment, three-dimensional configuration information indicating a form in which a left viewpoint video and a right viewpoint video constitute a stereoscopic video, and left-right order information indicating an order of the left viewpoint video and the right viewpoint video in the stereoscopic video Determining at least one of stereo mono section change count information indicating the number of changes between the monoscopic video section and the stereoscopic video section;
At least one of stereo confirmation information indicating whether the current section is the monoscopic video section or the stereoscopic video section, and section sample number information indicating the number of samples constituting the current section. And providing a multimedia streaming service providing method according to claim 1. Determining the attribute information for each fragment includes
For each stereoscopic video section of the current fragment, each of the three-dimensional configuration information about the samples constituting the current stereoscopic video section, the left-right order information, and one or more stereoscopic video sections of the current fragment Identification information about a stereoscopic video section, information about a stereoscopic camera that has acquired a sample constituting a current telescopic video section, and information about a display standard for safely three-dimensional reproduction using the sample. Determining at least one of the stereoscopic camera display information including,
The stereoscopic camera display information includes information indicating the number of items to which the stereoscopic camera display information is applied in the current stereoscopic video section of the current fragment, and the current stereoscopic video referenced by each item. 3. The multimedia streaming service providing method according to claim 2, further comprising at least one of identification information of camera display information and information on a length of a section to which the current stereoscopic camera display information is applied. . Determining the attribute information for each fragment includes
Reference video position information to be referred to for constructing a stereoscopic video with respect to the video of the current file includes reference video identification information in the current file, remote position information outside the current file, URL (Uniform Resource Locator) ) Information and a stage determined using at least one of URN (Uniform Resource Name) information;
Determining at least one of reference video number information, reference index information for identifying a plurality of reference videos, and the position information about each reference video when the video of the current file refers to a plurality of videos; The multimedia streaming service providing method according to claim 1, further comprising: The multimedia streaming service providing method includes:
Generating a multi-channel video differentially encoded with the video quality;
The attribute information determination step for each fragment includes a step of determining at least one of multi-channel number information indicating the number of videos of other channels of the video of the current file and reference information indicating a position of video of the other channels. The multimedia streaming service providing method according to claim 1, further comprising: In a streaming service receiving method for receiving multimedia using a network,
Receiving a plurality of fragments sequentially;
Extracting encoded data of a video including at least one of a 2D video and a 3D video from the media data area for each fragment in a form divided into fragment units;
Parsing attribute information from the auxiliary data area for each fragment, for 3D signaling information for 2D playback or 3D playback of the video of the fragment, and for the video to reference one or more other videos Extracting and reading the relationship information and both ,
Restoring the encoded data of the fragment into 2D video or 3D video based on information read from the attribute information for each fragment;
The three-dimensional signaling information and the relationship information are transmitted through the auxiliary data area, and the encoded video data is transmitted through the media data area.
A multimedia streaming service receiving method. The step of extracting and reading the attribute information includes:
Three-dimensional configuration information set for the fragment video data for each fragment and indicating a form in which the left viewpoint video and the right viewpoint video constitute a stereoscopic video, the left viewpoint video and the right in the stereoscopic video Extracting and reading at least one of left-right order information indicating the order of viewpoint video and stereo-mono section change count information indicating the number of changes between the monoscopic video section and the stereoscopic video section; and
For each fragment, stereo confirmation information indicating whether the current section is the monoscopic video section or the stereoscopic video section, and section sample number information indicating the number of samples constituting the current section. The method of claim 6, further comprising: extracting and reading at least one of them. The step of extracting and reading the attribute information includes:
The three-dimensional configuration information, the left-right order information, and the one or more stereoscopic video sections of the current fragment, which are set for each stereoscopic video section of the current fragment and are related to the samples constituting the current stereoscopic video section. Among them, identification information about each stereoscopic video section, information about a stereoscopic camera that has acquired a sample constituting the current stereoscopic video section, and a display standard for safely three-dimensional reproduction using the sample. Extracting and reading at least one of the stereoscopic camera display information including the information,
The stereoscopic camera display information includes information indicating the number of items to which the stereoscopic camera display information is applied in the current stereoscopic video section of the current fragment, and the current stereoscopic video referenced by each item. The multimedia streaming service receiving method according to claim 7, further comprising at least one of identification information of camera display information and information on a length of a section to which the current stereoscopic camera display information is applied. . The step of extracting and reading the attribute information includes:
The reference video position information that is set for the video of the current file and that is referred to for constructing the stereoscopic video is extracted, and the reference video identification information in the current file and the remote position information outside the current file are extracted. Reading at least one of URL information and URN information;
When the video of the current file refers to a plurality of videos, at least one of reference video number information, reference index information for identifying a plurality of reference videos, and the position information about each reference video is extracted. And receiving the multimedia streaming service according to claim 6. The multimedia streaming service receiving method includes:
A method of selectively receiving a predetermined channel of a plurality of channels of video differentially encoded according to video quality;
The step of extracting and reading the attribute information extracts at least one of multi-channel number information indicating the number of videos of other channels in the current file video and reference information indicating a position of the video of the other channels. The method of claim 6, further comprising the step of: The multimedia streaming service receiving method includes:
The multimedia streaming according to claim 6, further comprising the step of playing the restored 2D video or 3D video based on information read from the 3D signaling information for each fragment. Service reception method. In a streaming service providing apparatus for transmitting multimedia using a network,
A fragment video data generation unit that divides encoded data of video including at least one of two-dimensional video and three-dimensional video into fragment units;
Attribute information including both three-dimensional signaling information including information for two-dimensional reproduction or three-dimensional reproduction of the current video for each fragment, relation information for the video to refer to one or more other videos, and the like. A fragment attribute information determination unit for determining
A streaming unit that continuously transmits a fragment composed of an auxiliary data area including attribute information about the fragment and a media data area including the encoded video data;
The three-dimensional signaling information and the relationship information are transmitted through the auxiliary data area, and the encoded video data is transmitted through the media data area.
An apparatus for providing a multimedia streaming service. In a streaming service receiver for receiving multimedia using a network,
A receiving unit for continuously receiving a plurality of fragments;
A fragment video data extraction unit that extracts encoded data of a video including at least one of a two-dimensional video and a three-dimensional video from the media data area for each fragment in a form divided into fragments;
Parsing attribute information from the auxiliary data area for each fragment, for 3D signaling information for 2D playback or 3D playback of the video of the fragment, and for the video to reference one or more other videos A fragment attribute information extraction unit that extracts and reads both relationship information and
A restoration unit that restores the encoded data of the fragment to 2D video or 3D video based on information read from the attribute information for each fragment;
The three-dimensional signaling information and the relationship information are transmitted via an auxiliary data area, and the encoded data is transmitted via a media data area.
A multimedia streaming service receiver.   A computer-readable recording medium having recorded thereon a program for implementing the multimedia streaming service providing method according to claim 1 by an arithmetic processor.   A computer-readable recording medium having recorded thereon a program for implementing the multimedia streaming service receiving method according to claim 6 on an arithmetic processor.

Images