JP2016015534A - Information processor and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of swiftly switching among media.SOLUTION: A processing to previously open plural media responding to a request from a regenerator which reproduces the media. When receiving a request for switching a medium from a regenerator, transmission of a switched medium is started without performing any processing to open the medium. When receiving a request to open another medium different from a predetermined media specifying another medium while the regenerator is reproducing the predetermined medium, responding to the request, a piece of media information of the other medium specified by the regenerator is transmitted to the regenerator. This technique is applicable to, for example, a distribution device for distributing content data by using MPEG-DASH.

Description

  The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method capable of switching media at high speed.

  In recent years, MPEG-DASH (Moving Picture Experts Group-Dynamic Adaptive Streaming over HTTP) has been standardized as a content distribution technology using HTTP (Hyper Text Transfer Protocol) (see, for example, Non-Patent Document 1). In MPEG-DASH, multiple encoded data in which the same content is expressed at different bit rates are stored in the content server, and the client selects one of the multiple encoded data according to the network bandwidth. However, it uses ABS (Adaptive Bitrate Streaming) technology for playback.

  By the way, it has been considered to adaptively select and distribute a partial image which is a part of the image instead of the entire image. For example, a partial image of a part selected by the terminal that receives the image data in the entire image is distributed, the terminal performance (for example, the processing capacity of the CPU, the size of the display, etc.), the transmission path, and the load on the server It was considered to control the size of the partial image to be distributed according to the situation.

MPEG-DASH (Dynamic Adaptive Streaming over HTTP) (URL: http://mpeg.chiariglione.org/standards/mpeg-dash/media-presentation-description-and-segment-formats/text-isoiec-23009-12012-dam -1)

  However, in the conventional MPEG-DASH standard, when a medium is switched, a process for opening the medium after the switching is performed, and then transmission of the medium after the switching is started. For this reason, a seam occurred in the processing, and switching could not be performed at high speed.

  The present disclosure has been made in view of such a situation, and enables media to be switched at high speed.

  An information processing device according to an aspect of the present disclosure is requested to open a plurality of media in advance in response to a request from a playback device that plays back the media, and to switch the media from the playback device. A transmission unit that starts transmission of the media after switching without performing the process of opening the media.

  An information processing method according to an aspect of the present disclosure performs processing for opening a plurality of media in advance in response to a request from a playback device that plays back media, and when the playback device requests switching of the media, A step of starting transmission of the media after switching without performing a process of opening the media.

  In one aspect of the present disclosure, a process of opening a plurality of media is performed in advance in response to a request from a playback device that plays back the media. The transmission of the media after switching is started without performing the opening process.

  According to one aspect of the present disclosure, media can be switched at high speed.

It is a figure explaining the outline | summary of MPEG-DASH. It is a figure which shows the structural example of MPD. It is a figure explaining the time division | segmentation of a content. It is a figure which shows the example of the hierarchical structure below the Period in MPD. It is a figure explaining the structural example of a MPD file on a time-axis. It is a block diagram which shows the main structural examples of a delivery system. It is a figure explaining the transfer method of the segment data between the server-client in each of conventional HTTP and a web socket (WebSocket). It is a figure explaining the transfer method at the time of applying the band adaptive logic of the segment data between the server and client in each of conventional HTTP and web socket (WebSocket). It is a figure explaining the protocol of the subframe of the web socket applied to DASH. It is a figure explaining the syntax of the control data of dash. It is a figure which shows the list of the operation code used for the sub-frame which transmits control data. It is a figure explaining the syntax of operation code openMedia. It is a figure explaining the syntax of operation code mediainfo. It is a figure explaining the syntax of operation code closeMedia. It is a figure explaining the syntax of operation code playStream. It is a figure explaining the syntax of operation code StreamInfo. It is a figure explaining the syntax of operation code stopStream. It is a figure explaining the syntax of operation code changeMedia. It is a figure explaining the syntax of operation code playerInfo. It is a figure explaining the syntax of operation code capability. It is a figure explaining the syntax of operation code preference. It is a timing chart explaining the sequence which starts VOD reproduction | regeneration. It is a timing chart explaining the sequence which interrupts and restarts VOD reproduction | regeneration. It is a timing chart explaining the sequence which reproduces | regenerates VOD reproduction to the last and complete | finishes. It is a timing chart explaining the sequence which starts live reproduction in the case of viewing from the beginning of live. It is a timing chart explaining the sequence which starts live reproduction in the case of viewing from the middle of the live. It is a functional block diagram which implement | achieves the function of a server. It is a functional block diagram which implement | achieves the function of a client. It is a timing chart explaining the sequence of Multiple Representations Transfer at startup. It is a figure explaining the sequence of Multiple Representations Transfer at startup. 6 is a timing chart illustrating a sequence of Low delay Transfer by Sub-Segment. It is a figure explaining the sequence of Low delay Transfer by Sub-Segment. It is a timing chart explaining the sequence performed when network congestion generate | occur | produces by live reproduction. It is a timing chart explaining the sequence which switches and reproduces several media. It is a timing chart explaining the sequence which switches and reproduces several media. And FIG. 18 is a block diagram illustrating a configuration example of an embodiment of a computer to which the present technology is applied.

Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. Overview 2. First embodiment (MPEG-DASH over WebSocket)
3. Second embodiment (Multiple Representations Transfer at startup)
4). Third Embodiment (Low delay Transfer by Sub-Segment)
5. Fourth embodiment (Auto Segment Skip for Live Streaming)
6). Fifth embodiment (Fast Switching of multiple media)
7). Sixth embodiment (computer)

<1. Overview>
<DASH>
Conventionally, as a content distribution technique using HTTP (HyperText Transfer Protocol), for example, as described in Non-Patent Document 1, there is MPEG-DASH (Moving Picture Experts Group-Dynamic Adaptive Streaming over HTTP). In MPEG-DASH, multiple encoded data in which the same content is expressed at different bit rates are stored in the content server, and the client selects one of the multiple encoded data according to the network bandwidth. However, it uses ABS (Adaptive Bitrate Streaming) technology for playback.

  A procedure of content transmission by DASH will be described with reference to FIG. First, in the video playback terminal on the content acquisition side, the streaming data control software selects an MPD (Media Presentation Description) file of the desired content and acquires it from the Web server. MPD is metadata for managing content such as moving images and audio to be distributed.

  When the MPD is obtained, the streaming data control software of the video playback terminal analyzes the MPD and generates data (DASH segment) that matches the quality of the communication line and the performance of the video playback terminal of the desired content. Control to obtain from Web server. The client software for HTTP access acquires the DASH segment from the Web server using HTTP according to the control. The moving image reproduction software reproduces the content acquired in this way.

  The MPD has a configuration as shown in FIG. 2, for example. In MPD analysis (parsing), the client selects an optimum attribute from the representation attribute included in the period of the MPD (Media Presentation in FIG. 2).

  The client reads the first segment (Segment) of the selected representation (Representation), acquires an initialization segment (Initialization Segment), and processes it. Subsequently, the client acquires and reproduces the subsequent segment (Segment).

  In addition, the relationship between a period (Period), a representation (Representation), and a segment (Segment) in MPD is as shown in FIG. That is, one media content can be managed for each period (Period) that is a data unit in the time direction, and each period (Period) should be managed for each segment (Segment) that is a data unit in the time direction. Can do. In addition, for each period (Period), a plurality of representations (Representations) having different attributes such as bit rate can be configured.

  Therefore, this MPD file (also referred to as MPD file) has a hierarchical structure as shown in FIG. 4 below the period. Further, when the MPD structures are arranged on the time axis, an example shown in FIG. 5 is obtained. As is clear from the example of FIG. 5, there are a plurality of representations (Representations) for the same segment (Segment). The client can acquire and reproduce appropriate stream data according to the communication environment, its decoding capability, and the like by adaptively selecting one of these.

<Distribution system>
Next, an apparatus and method for realizing the present technology as described above will be described. FIG. 6 is a diagram illustrating a distribution system, which is an aspect of a system to which the present technology is applied. A distribution system 100 shown in FIG. 6 is a system that can adaptively distribute data of a partial image that is a part of an entire image.

  As illustrated in FIG. 6, the distribution system 100 includes a distribution data generation device 101, a distribution server 102, and a terminal device 103.

  The distribution data generation device 101 generates a content file (content file) such as an image or sound and an MPD file distributed by the distribution server 102 and supplies the generated MPD file to the distribution server 102. The distribution server 102 publishes the content file and the MPD file supplied from the distribution data generation apparatus 101 to the network 104, and performs adaptive distribution of partial images.

  The terminal device 103 accesses the distribution server 102 via the network 104 and acquires an MPD file of desired content that the distribution server 102 discloses.

The terminal device 103 accesses the distribution server 102 via the network 104 in accordance with the MPD file, adaptively selects an appropriate content file corresponding to the MPD file, and acquires it using the HTTP protocol. The terminal device 103 reproduces the acquired content file.
<2. First Embodiment>
<MPEG-DASH over WebSocket>
Next, a method for transferring segment data (Seg. Data in the figure) between the server and the client in each of conventional HTTP and Web socket (WebSocket) will be described with reference to FIG. Note that the server 201 in FIG. 7 corresponds to the distribution server 102 in FIG. 6 described above, and the client 202 corresponds to the terminal device 103 in FIG. 6 described above.

  Transfer of segment data between the server 201 and the client 202 by HTTP is a process as shown in FIG. 7A. Here, it is assumed that buffering is performed in units of three segment data (Seg. Data) at the start of decoding.

  That is, when playback is instructed first in the client 202 (PlayStart), a request consisting of a Get command (Req at the beginning of the third stage client 202 in FIG. 7A) is transmitted to the server 201.

  Next, the server 201 transmits the MPD to the client 202 as a response (Resp. At the head of the third-stage client 202 in FIG. 7A) in response to this Get command.

  Based on the MPD information, the client 202 sequentially requests segment data (Seg. Data) from the server 201 (second and subsequent Req. Of the third-stage client 202 in FIG. 7A).

  In response to this request, the server 201 transmits segment data (Seg. Data) to the client 202 (the second and subsequent Resp. Of the third-stage server 201 in FIG. 7A).

  Here, when the client 202 requests the segment data (Seg.Data) from the server 201 and the server 201 transmits the segment data (Seg.Data) to the client 202, the second stage of FIG. As described above, a delay time (Delay to Rendering) occurs from the timing of PlayStart to the timing of playback start by the client 202 (DecStart).

  FIG. 7A shows that the server 201 and the client 202 are connected at the top, and in the middle, playback is actually started from the timing when playback is requested (Play Start). The delay time (Delay to Rendering) until the start timing (DecStart) is shown. Further, in the lowermost part of FIG. 7A, the state in which segment data (Seg. Data) is transferred on the communication line drawn in a pipe shape is drawn in time series in the right direction in the figure. In FIG. 7A, the upper side of the pipe-shaped communication line indicates the operation timing on the client 202 side, and the lower side indicates the operation timing on the server 201 side. Further, in the following description, the same applies to FIGS. 7B and 8.

  That is, in the conventional HTTP, since the server 201 and the client 202 are so-called Pull lines, GAP occurs between transactions, and it takes time to read a plurality of files. In particular, every time a TCP connection is made, more time is required due to delays in opening and slow start of flow control.

  As a result, at the start of playback, a sufficient amount of data is buffered so that it takes a long time to display the playback image and the transfer throughput does not increase, so a video with a low bit rate is selected. There is a risk that an image of quality will be transferred.

  On the other hand, the transfer of the segment data by the web socket between the server 201 and the client 202 is a process as shown in FIG. 7B.

  That is, first, when playback is instructed in the client 202 (PlayStart), a request including a Get command (Command Send at the head of the third stage client 202 in FIG. 7B) is transmitted to the server 201.

  Next, in response to this Get command, the server 201 sends segment data (Seg. Data) to the client 202 together with MPD as a response (Data (MPD) at the head of the client 202 in the third row in FIG. 7B). Transmit sequentially (second and subsequent Seg.Data of the third-stage server 201 in FIG. 7B).

  Again, when the client 202 requests the segment data (Seg.Data) from the server 201 and the server 201 transmits the segment data (Seg.Data) to the client 202, the second row in FIG. As described above, although a delay time (Delay to Rendering) occurs from the timing of PlayStart to the timing when playback is started by the client 202 (DecStart), it is shorter than in the case of HTTP.

  That is, in the web socket, since the server 201 and the client 202 are a so-called push line, no GAP occurs between transactions, and the time required for reading a plurality of files can be shortened. In particular, since the transfer is performed in the same TCP session, there is no connection problem.

  As a result, it is possible to shorten the time until the image at the start of playback is displayed and to increase the transfer throughput, so that a higher bit rate video can be selected, and a relatively high image quality can be selected from the start of playback. You can expect to transfer images.

  However, also in HTTP, considering the case where Pipelining, which is a bandwidth adaptation logic, is supported, the delay time can be shortened compared to the processing in FIG. 7A described above. Pipelining means that the client 202 determines the state of the bandwidth, and can make a plurality of requests to the server 201 at a time according to the bandwidth. This is a function that can receive data in response to a request.

  That is, as shown in FIG. 8A, at the timing after the response from the server 201 by the MPD, the client 202 requests the server 201 for three segment data at the same time. The three segment data are received collectively.

  By doing so, gaps from when a request as shown in FIG. 7A is issued until a response is received can be omitted, and the delay time (Delay to Rendering) can be shortened.

  However, since the next segment data is requested before the response to the request is returned, the bandwidth of the ongoing transaction is monitored, and a different representation (Representation) is selected according to the situation. Therefore, it cannot be reflected in the next segment data request. For this reason, the transaction result in progress can be reflected only in the request for the segment after three.

  That is, for example, as shown in FIG. 8A, in order to determine the bandwidth on the client 202 side, the timing after sending the first request is actually the next, even though there is sufficient bandwidth. Until the request is transmitted, three pieces of low-quality segment data must be received from the server 201. Note that Seg.Data (rep1) and Seg.Data (rep3) at the bottom of FIG. 8A indicate segment data with different representations, and rep3 is segment data of a representation with higher image quality than rep1. It shows that there is. That is, in the lowermost part of FIG. 8A, in response to the first request, three segment data (Seg.Data (rep1)) with low image quality are transmitted, and then segment data (Seg.Data (rep3) with high image quality is sent. )) Is shown to be sent.

  On the other hand, in the web socket, the server 201 can switch the representation of the segment data by determining the bandwidth state. For this reason, as shown in FIG. 8B, since the server 201 makes a determination according to the bandwidth, even in the timing immediately after sending the first segment data, the segment of the different representation depends on the bandwidth status. Data can be transmitted. That is, at the bottom of FIG. 8B, high-quality segment data (Seg.Data (rep3)) is transmitted immediately after the first low-quality segment data (Seg.Data (rep1)) is transmitted. It is shown.

  As a result, it is possible to switch and transmit segment data for a quick and appropriate representation according to the state of the band, and to simplify the implementation content on the client 202 side. .

<Web socket subframe protocol>
In the web socket, data is transmitted and received in units called frames.

  Then, with reference to FIG. 9, the protocol of the subframe of the web socket applied to DASH is demonstrated next.

  On the left side of FIG. 9, examples of the web socket base framing protocol and operation codes (operation code, opcode) described in the header portion of the subframe are shown.

  In the header part of the subframe, in addition to the operation code, for example, FIN indicating whether this subframe is the end of data, MASK indicating whether the data is masked, Payload indicating the length of the application data len etc. are described.

  Further, when the operation code is 0, it indicates a subframe (Continuation Frame) used to continue the connection. When the operation code is 1, the subframe for text data (Text Frame) It is shown that. In addition, when the operation code is 2, it indicates a subframe (Binary Frame) for binary data, and when the operation code is 8, it is a subframe (Connection Close Frame) for terminating the connection. Is shown. Further, when the operation code is 9, it indicates a subframe (Ping Frame) used for confirming whether the connection can be maintained. Further, when the operation code is 10, it indicates a subframe (Pong Frame) used for a response to Ping Frame.

  When the web socket is applied to DASH, as shown on the right side of FIG. 9, two data of control data transmitted as a text frame and segment data transmitted as a binary frame are included in the payload portion of the subframe. Is defined.

  For example, the control data is composed of JSON Data described by Jason (JavaScript Object Notation) which is one of lightweight data description languages. The segment data is composed of data described in accordance with the ISO / IEC 14496-12 (MPEG-4) standard.

  As described above, when the web socket is applied to the DASH, the control data and the segment data are not mixed in the payload portion of the common subframe, but are arranged in the payload portions of the individual subframes.

  Thereby, it is possible to avoid transmission of the control data with a delay. In other words, because segment data has a large amount of data, transmission may be delayed. When control data is mixed with segment data, control data is recognized only after all the mixed data has been read. I can't. For this reason, it is assumed that a delay time occurs until the control data is recognized. On the other hand, if control data and segment data are arranged in the payload part of individual subframes, control data can be recognized because relatively small control data can be transmitted before segment data is transmitted. Thus, transmission can be performed with a reduced delay time.

  Also, the received segment data can be added (Append) to the MSE source buffer without parsing with JavaScript (registered trademark) using HTML5 web socket API.

  Next, control data will be described with reference to FIGS.

  FIG. 10 shows the syntax of the control data described by Jason.

  In the example of FIG. 10, the protocol ID is dash and the protocol version is 0.2. Further, the time stamp (1403627982268 in the example of FIG. 10) can correct a transmission delay occurring in the application layer and below.

  For example, a plurality of control data can be transmitted together in one subframe. As a result, the receiving side can easily perform batch processing. That is, in the example of FIG. 10, operation codes of capability, openMedia, and playStream are collectively described as control data.

  FIG. 11 shows a list of operation codes used for subframes for transmitting control data.

  The openMedia is an operation code for specifying a MPD URL so that a media segment can be streamed to the server, and is sent from the client to the server. For example, openMedia is described as shown in FIG. 12, and “opcode”: “openMedia” on the first line is an operation code for specifying a URL of an MPD so that a media segment can be streamed to a server. It represents that. The MPD is specified by the URL (“http: //cdn.example/movies/134532.mpd” in the example of FIG. 12). Then, the server that received openMedia opens the specified MPD and sends mediaInfo back to the client.

  mediaInfo is an operation code transmitted immediately after receiving openMedia or when media information is updated, and is transmitted from the server to the client.

  For example, mediaInfo is described as shown in FIG. 13, and “opcode”: “mediaInfo” on the first line is an operation code transmitted immediately after receiving openMedia or when media information is updated. It represents something. In the second and subsequent lines, the target media is designated by mediaDescriptor (152736 in the example of FIG. 13). Further, the network operation state (activity) (4 in the example of FIG. 13) is specified as defined in HTML5 Video by networkState. Also, the ready state of the target medium (4 in the example of FIG. 13) is designated by readyState. Further, the MPD is specified by the URL (“http: //cdn.example/movies/134532.mpd” in the example of FIG. 13).

  closeMedia is an operation code for releasing resources secured by the server using openMedia, and is sent from the client to the server. For example, closeMedia is described as shown in FIG. 14, and “opcode”: “closeMedia” represents an operation code for releasing a resource secured by the server with openMedia. The media to be closed is specified by mediaDescriptor (152736 in the example of FIG. 14).

  playStream is an operation code for starting a stream and specifying an open medium, and is transmitted from the client to the server.

  For example, playStream is described as shown in FIG.

  More specifically, “opcode”: “playStream” on the first line represents an operation code for starting a stream and designating an opened medium.

  In the second and subsequent lines, the segment of the segment number specified by segmentNumber of the medium specified by mediaDescriptor (152736 in the example of FIG. 15) in the stream specified by streamDescriptor (4323 in the example of FIG. 15) (see FIG. 15). In the example of 15, the stream for the number of segments (1000 in FIG. 15) designated by numSegment is started from 155). In the case of VOD, the stream starts from the first segment of the media, and in the case of Live, the stream starts from the current latest segment.

  The streamInfo is an operation code for notifying the state of the stream and the attribute of the segment data to be transmitted next, and is transmitted from the server to the client.

  For example, streamInfo is described as shown in FIG. Further, streamInfo indicates attribute information of segment data that is transmitted periodically and transmitted after the control data.

  More specifically, “opcode”: “StreamInfo” on the first line represents an operation code for notifying the state of the stream and the attribute of the segment data to be transmitted next.

  In the second and subsequent lines, the target stream is specified by streamDescriptor (4323 in the example of FIG. 16). Further, the target medium is designated by mediaDescriptor (1527364235312 in the example of FIG. 13). Furthermore, the network operation state (activity) (4 in the example of FIG. 16) is designated as defined in HTML5 Video by networkState. Also, the ready state (4 in the example of FIG. 16) of the target medium is described by readyState.

  With multipleRepresentation, it is specified whether or not there is a possibility that a plurality of representations may be transmitted for the same segment data. In the example of FIG. 16, “true” is specified, which indicates that there is a possibility that a plurality of representations may be transmitted for the same segment data. In that case, the client waits for reception of the next StreamInfo and then decides whether to use or discard the current Segment data. When “false” is specified, it is designated that a plurality of representations are not transmitted for the same segment data.

  With segmentSkipped, it is specified whether or not there is a possibility of performing segment skip, which causes the delivery of a segment whose playback has been delayed with respect to real time to be performed during Live distribution. In the example of FIG. “false” indicates that there is no possibility of segment skipping. When “true” is set, it is specified that there is a possibility of segment skipping.

  The minimum buffering time is specified by minBufferTime, and in FIG. 16, “4.00” specifies 4 seconds.

  The minNetworkBufferTime specifies an encouragement value that the client should start decoding after buffering data. In FIG. 16, “5.00” specifies 5 seconds.

  In other words, the server uses MinNetworkBufferDuration to specify as an encouragement value that the client should start decoding after buffering, and judging from the network status, etc., the client buffer is depleted due to bandwidth fluctuations. StreamInfo is signaled (transmitted) at the start of the stream so that playback does not stop.

  Segment data is designated by segment, and among these, the time of segment data is designated by presentationTime, and in FIG. 16, “331533” is designated in milliseconds. In addition, the reproduction start position is designated by “periodID”. In FIG. 16, “1” is designated, and reproduction from the leading period is designated.

  Further, the type of segment data is designated by “adaptationSetID”, and “video” is designated in FIG. Also, the type of representation is specified by representationID, and “rep4” is specified in the example of FIG. The segment data number is specified by "segmentNumber", and "1324" is specified in Fig. 16. If there is no information specifying these segments, the target segment is the head of the initialization. It is considered as a segment.

  “SAPType” designates the type of the start position SAP (Stream Access Point) at which the separated presentation such as a segment or subsegment changes. In other words, the SAP type is defined from the description Type 1 to 6 in which the first picture of the Segment SubSegment is the first picture of the Closed GOP or Open GOP. In the example of FIG. 16, 1 is specified. Yes.

  The subsegmentNumber designates a number for identifying a subsegment constituting one segment data, and “0” is designated in FIG. As for the number identifying the sub-segment, when three sub-segments constitute one segment data, as shown in the example of FIG. 16, if the number is the first sub-segment, “0” In the second sub-segment, “1” may be set, and in the third sub-segment, “2” may be set.

  That is, the server signals (transmits) the SAPType and the SubSegmentNumber in order to divide the Segment Data and transmit it as a SubSegment.

  In this way, by defining the sub-segment, in StopStream and ChangeMedia, the server can stop the stream in the middle of SubSegment. As a result, the stream can be stopped immediately by StopStream without transmitting unnecessary Segment Data, and ChangeMedia can be realized with low delay. Furthermore, ChangeMedia can be switched at a higher speed by streaming the switched Media from the SubSegment.

  stopStream is an operation code for stopping the stream, and is transmitted from the client to the server.

  For example, stopStream is described as shown in FIG. 17, and the stop of the stream designated by the streamDescriptor (4323 in the example of FIG. 16) is instructed as the target stream.

  More specifically, “opcode”: “stopStream” on the first line indicates an operation code for instructing to stop the stream, and “params”: {“streamDescriptor”: “on the second to fourth lines. 4323 "} represents a parameter for designating a stream to be stopped.

  changeMedia changes the media to be streamed, specifies an open media, and switches without stopping the stream. changeMedia is sent from the client to the server.

  For example, changeMedia is described as shown in FIG.

  “Opcode”: “changeMedia” on the first line represents an operation code for designating a stream medium to be reproduced after the change. 2nd to 7th line "params": {"streamDescriptor": "4323", "mediaDescriptor": "152736", "segmentNumber": "155", "numSegment": "1000"} , A parameter for specifying a medium to be switched, a parameter for specifying a segment number, and a parameter for specifying the number of segments when there is no end of the media.

  In this way, by setting a plurality of media in an open state to the server in advance and immediately allowing a segment to be streamed, immediately after switching the media, as will be described later with reference to FIGS. , Another segment can be transmitted. Further, the above-mentioned OpenMedia and CloseMedia manage resources related to media held by the server.

  playerInfo is an operation code for notifying the playback status of the player who is playing the stream, and is sent from the client to the server.

  For example, playerInfo is described as shown in FIG. 19, and is notified if there is a change in the playback status of the player.

  “Opcode” on the first line: “playerInfo” represents an operation code for notifying the playback status of the player. “Params” on lines 2 to 8: {“streamDescriptor”: “4323”, “networkState”: “4”, “readyState”: “4” “currentPresentationTime”: “478777”, “bufferedTime”: “4000”} Represents a parameter for identifying a stream, a parameter representing a network status, a parameter representing a reading status, a parameter representing the current playback time in milliseconds, and a parameter representing the buffered time in milliseconds.

  In this way, the server can detect that the client cannot be played back in real time using the currentPresentationTime included in the playerInfo. As a result, as will be described later with reference to FIG. 33, the server can skip unnecessary segments that are not reproduced and continue to transfer the latest live segment.

  The capability is an operation code for notifying the client's capability, such as a codec, and is transmitted from the client to the server.

  For example, capability is described as shown in FIG.

  That is, “opcode”: “capability” in the first line represents an operation code for notifying the client capability, such as a codec. In the second and subsequent lines, it is specified by multipleRepresentation whether or not the client can accept a plurality of Representations for the same segment data. For example, in FIG. 20, “true” is specified to specify that a plurality of representations can be accepted for the same segment data. When “false” is specified, it is specified that a plurality of representations cannot be accepted for the same segment data.

  By segmentSkip, it is specified whether or not it is possible to cope with segment skipping. For example, it is set to “false” in FIG. 20 and it is designated that segment skipping is not possible. When “true” is set, it is specified that segment skipping can be supported.

  The type of decoding that can be supported is specified by decode, the mime type is specified by mimeType, and in FIG. 20, “video / mp4” is specified, and it is specified that mp4 video can be decoded. Further, it is specified by codec that [“avc1.640828”] is possible as the type of codec. At this time, the maximum width, the maximum height, and the maximum bandwidth of the image that can be processed are designated by maxWidth, maxHieght, and maxBandwidth. In FIG. 20, “1920”, “1080”, and “101024” are set, respectively. It is specified that an image with a 1920 × 1080 pixel bandwidth of 101024 can be processed.

  In addition, “segmentAlignment!” Specifies whether or not the segments of the representation are aligned. In FIG. 20, “false” is specified, and it is specified that the segments of each representation are aligned. If true, it is specified that there are no segments for each representation.

  Furthermore, the type of segment start position SAP (Stream Access Point) is specified by SAPType. In the case of FIG. 20, it is specified that the SAP type is either 1 or 2.

  In FIG. 20, “audiomp4” is designated as a mime type that can be further dealt with, and it is designated that audio mp4 can be decoded. Furthermore, [“mp4a.40”] is designated as the codec type, and segmentAlignment! Designates whether or not the segments of the representation are aligned. In FIG. 20, it is set to false. It is specified that the segment is complete.

  The preference is an operation code for notifying the client's preference (setting that should be given priority) such as AdaptationSetID, and is transmitted from the client to the server.

  For example, preference is described as shown in FIG.

  That is, “opcode”: “preference” on the first line represents an operation code for notifying a client preference (a setting that is to be given priority). In the second and subsequent lines, the target medium is designated by mediaDescriptor (152736 in the example of FIG. 21) by mediaDescriptor. In segmentSkip, it is specified whether or not the target media is not in a segment skip state. In FIG. 21, it is specified by “false” that it is requested to set a state in which segment skip is not performed. Yes.

  The language to be used is specified by language, and in the case of FIG. 21, it is specified by [“en”, “fr”] that it is required to preferentially set the use of English and French.

  The segment specifies a preference for setting the segment data, and the first period ID among them specifies the image playback start position. In FIG. 21, this is set to “1”, and the start period is set as the image playback start position. Is specified to be set with priority.

  The image type of the segment data is specified by the first adaptationSetID. In the case of FIG. 21, it is indicated that the main image is requested to be set preferentially by “mainVideo”.

  The type of image representation is specified by the first reprsentationID. In FIG. 21, it is set to [“1”, “2”, “3”], and the type of reprsentation is any one of 1 to 3. It is shown that it is demanded to set with priority. In this case, three types of representations are designated, but the server determines the state of the band among these three types of representations and sets an optimal representation.

  Furthermore, the second period ID specifies the audio playback start position. In the case of FIG. 21, it is set to “1”, and it is required to preferentially set the leading period as the audio playback start position. It is specified.

  The audio type of the segment data is specified by the second adaptationSetID, and in the case of FIG. 21, it is indicated that “mainAudio” is required to set the main audio preferentially.

  Further, the type of audio representation is specified by the second reprsentationID. In FIG. 21, it is set to [“1”], and the type of audio representation is preferentially set to 1. Is shown to be required.

  In other words, the preference of the client specifies which MediaSegment of Media is to be streamed. (If nothing is specified, the server automatically selects according to the Capability of the client.)

  If multiple representationIDs are specified, the server executes Network Adaptation (processing according to the network status), and within that range, the server determines appropriate bandwidth based on the decision of the server such as bandwidth adaptation. Select a presentation and stream it.

  Such control data is transmitted and received between the server 201 and the client 202, and distribution of the content file is controlled.

  FIG. 22 shows a sequence for starting VOD reproduction.

  For example, the processing is started when the user performs an operation on the client 202 so as to start VOD playback. In step S11, the client 202 transmits control data (capability, openMedia, playStream) to the server 201, notifies the client 202 of the capability, and requests to start a stream by designating the medium.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by transmitting control data (mediaInfo) to the client 202 in step S12.

  In step S13, the server 201 transmits control data (streamInfo) to the client 202 to notify the attribute of segment data to be transmitted next. Subsequently, the server 201 transmits segment data (RepID 1, Segment No 0) to the client 202 in step S14.

  The client 202 receives the segment data and starts decoding. In step S15, the client 202 transmits control data (playerInfo) to the server 201, thereby notifying the elapsed time after starting the decoding of the stream.

  Thereafter, the server 201 sequentially transmits segment data to the client 202. At this time, the server 201 can perform appropriate band adaptation by estimating the buffer amount of the client from CurrentPresentanTime with PlayerInfo that changes the representation (Rep) by processing (Network Adaption) according to the network state. .

  Accordingly, for example, the server 201 transmits control data (streamInfo) in step S16, and transmits segment data (RepID 3, Segment No 1) in step S17. Subsequently, in step S18, the server 201 transmits control data (streamInfo) to the client 202. In step S19, the server 201 transmits segment data (RepID 3, Segment No 2).

  In addition, the server 201 can continue to send segment data as transmission progresses by the client 202 by transmission control. That is, the server 201 transmits control data (streamInfo) to the client 202 at a timing in accordance with the reproduction progress of the client 202, and transmits segment data (RepID 3, Segment No 3) at step S21.

  On the other hand, when there is a change in the playback status, the client 202 transmits control data (playerInfo) to the server 201 in step S22, and notifies the elapsed time since the start of stream decoding.

  In this way, processing for starting VOD playback is performed, and similarly, segment data is transmitted from the server 201 to the client 202.

  FIG. 23 shows a sequence for interrupting and resuming VOD playback.

  As described above, transmission of segment data is continued, and the server 201 transmits control data (streamInfo) in step S31, and transmits segment data (RepID 2, Segment No 153) in step S32. Then, the client 202 transmits control data (playerInfo) to the server 201 in step S33.

  When the user performs an operation on the client 202 so as to interrupt reproduction (for example, pause or seek), the client 202 transmits control data (stopStream) to the server 201 in step S34. As a result, the client 202 notifies the server 201 that the stream is to be stopped.

  In response to this, the server 201 transmits control data (streamInfo) and segment data (RepID 3, SegmentNo 154) in steps S35 and S36, respectively, and then, in step S37, control data (streamInfo) notifying the interruption. Is transmitted to the client 202.

  Thereafter, when the user performs an operation on the client 202 so as to resume reproduction, in step S38, the client 202 transmits control data (Start Stream), and specifies the segment number to resume (155 in the example of FIG. 23). specify.

  In response, the server 201 transmits segment data (RepID 3, Segment No 155) to the client 202 in step S39, and the client 202 transmits control data (playerInfo) to the server 201 in step S40.

  In this way, processing for interrupting and resuming VOD playback is performed, and segment data is transmitted from the server 201 to the client 202 in the same manner.

  FIG. 24 shows a sequence in which VOD playback is played back to the end and finished.

  As described above, transmission of segment data is continued, and the server 201 transmits control data (streamInfo) in step S51, and transmits segment data (RepID 2, Segment No 873) in step S52. Then, the client 202 transmits control data (playerInfo) to the server 201 in step S53.

  In step S54, the server 201 transmits control data (streamInfo). In step S55, the server 201 transmits segment data (RepID 2, SegmentNo 874). If the transmitted segment is the end of the stream, the server 201 transmits control data (streamInfo) in step S56 to notify the client 202 that the stream has ended.

  In response to this, the client 202 transmits control data (playerInfo) to the server 201 in step S57, and then transmits control data (closeMedia, stopStream) to the server 201 in step S58. As a result, the client 202 notifies the server 201 that the playback of all received segments is finished and the media is closed.

  When the server 201 receives the control data (closeMedia, stopStream), the server 201 discards the media information. In step S59, the server 201 transmits control data (MediaStatus) to the client 202, and the VOD reproduction is ended.

  In this way, a process for ending VOD playback is performed.

  FIG. 25 shows a sequence for starting live reproduction when viewing from the beginning of the live.

  For example, the processing is started when the user performs an operation on the client 202 so as to start live reproduction. In step S71, the client 202 transmits control data (capability, openMedia, playStream) to the server 201, notifies the client 202 of the capability, and requests to start the stream by specifying the medium.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by transmitting control data (mediaInfo) to the client 202 in step S72.

  Here, for example, when live broadcasting is not started, MediaStatus is periodically notified, and in step S73, the server 201 transmits control data (mediaInfo) to the client 202 to notify MediaStatus.

  Thereafter, when live broadcasting is started, transmission of a segment is started, and in step S74, the server 201 transmits control data (mediaInfo, streamInfo) to the client 202. Subsequently, the server 201 transmits segment data (Segment Data of Segment No 0) to the client 202 in step S75.

  The client 202 receives the segment data and starts decoding. In step S76, the client 202 transmits control data (playerInfo) to the server 201, for example, to notify the elapsed time since the start of decoding the stream. .

  In this way, the process of starting live playback when viewing from the beginning of live is performed, and similarly, segment data is transmitted from the server 201 to the client 202.

  FIG. 26 shows a sequence for starting live reproduction when viewing from the middle of live performance.

  For example, the processing is started when the user performs an operation on the client 202 so as to start live reproduction. In step S91, the client 202 transmits control data (capability, openMedia, playStream) to the server 201, notifies the client 202 of the capability, and requests the start of the stream by specifying the medium.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by transmitting control data (mediaInfo) to the client 202 in step S92.

  Here, for example, when live broadcasting has already started, transmission is performed from the latest segment, and the server 201 transmits control data (streamInfo) to the client 202 in step S93. Subsequently, the server 201 transmits segment data (Segment Data of Segment No 1447) to the client 202 in step S94.

  The client 202 receives the segment data and starts decoding. In step S95, the client 202 transmits control data (playerInfo) to the server 201, for example, to notify the elapsed time since starting the decoding of the stream. .

  Thereafter, the server 201 can continue to send segment data, and transmits control data (streamInfo) and segment data (Segment Data of Segment No 1448) in steps S96 and S97, respectively. Subsequently, in steps S98 and S99, the server 201 transmits control data (streamInfo) and segment data (Segment Data of Segment No 1449), respectively.

  In this way, the process of starting live playback when viewing from the middle of live is performed, and the segment data is transmitted from the server 201 to the client 202 in the same manner.

<3. Second Embodiment>
<Multiple Representations Transfer at startup>
<Configuration of server realizing distribution system>
As illustrated in FIG. 27, the server 201 includes a reception unit 211, a transmission unit 212, a storage unit 312, and a communication processing unit 214.

  The receiving unit 211 receives data transmitted to the server 201 via the network and supplies the data to the communication processing unit 214.

  The transmission unit 212 transmits, for example, various control data to the client 202 according to the communication processing executed in the communication processing unit 214, or appropriately reads the content file segment from the storage unit 312 and transmits it to the client 202.

  The storage unit 312 stores content files such as images and sounds distributed from the server 201.

  For example, the communication processing unit 214 performs communication processing using the control data as described above with the communication processing unit 227 of the client 202.

<Configuration of client to realize distribution system>
As shown in FIG. 28, the client 202 includes a receiving unit 221, a transmitting unit 222, a buffer 223, a reproduction processing unit 224, a display 225, a speaker 226, and a communication processing unit 227.

  For example, the reception unit 221 receives control data transmitted from the server 201 and supplies the control data to the communication processing unit 227, or receives a segment of a content file transmitted from the server 201 and supplies the segment to the buffer 223. The buffer 223 temporarily holds a segment of the content file supplied from the receiving unit 221.

  The reproduction processing unit 224 performs reproduction processing for reading and decoding the segment held in the buffer 223, supplies video data to the display 225, and supplies audio data to the speaker 226. Thereby, the display 225 displays an image, and the speaker 226 outputs sound.

  For example, the communication processing unit 227 performs communication processing using the control data as described above with the communication processing unit 214 of the server 201.

  Next, referring to the flowchart of FIG. 29, a process (Multiple Representations Transfer) of transmitting a plurality of representations at the start of content reproduction by the distribution system including the server 201 of FIG. 27 and the client 202 of FIG. at startup). That is, in this process, segment data of different representations are transmitted with the same segment number at the start of reproduction. As a result, it is possible to reproduce from the segment data of the representation with the image quality corresponding to the band state, and it is possible to always prevent the reproduction from starting from the image with the low image quality first.

  That is, for example, when the user performs an operation on the client 202 so as to start VOD playback, the processing is started.

  In step S <b> 111, the communication processing unit 227 of the client 202 controls the transmission unit 222 to transmit control data (capability, openMedia, playStream) to the server 201. As a result, the capabilities of the client 202 are notified to the server 201, the media is specified, and the start of the stream is requested.

  The communication processing unit 214 of the server 201 controls the reception unit 211 to receive this request, and reads the media information of the media designated by the client 202 from the storage unit 213. In step S <b> 112, the communication processing unit 214 controls the transmission unit 212 to transmit the control data (mediaInfo) to the client 202, thereby returning the media information.

  In step S113, the communication processing unit 214 of the server 201 controls the transmission unit 212 to transmit control data (streamInfo (RepID 1, Segment No 0, RepRep: true)) to the client 202, and then transmits the control data. Notify the attribute of the segment data to be processed. Subsequently, in step S <b> 114, the communication processing unit 214 controls the transmission unit 212 to transmit segment data (RepID 1, SegmentNo 0) to the client 202.

  That is, when the web socket is applied to the DASH, the control data and the segment data are respectively arranged in the payload portion of the individual subframe as described with reference to FIG.

  Therefore, in step S113, the communication processing unit 214 of the server 201 controls the transmission unit 212 to transmit segment data (MediaMedia) at the next timing using StreamInfo using the control data before transmitting the segment data. (RepRep: true) that indicates that there is a possibility of transmitting segment data of different representations with the same Segment No. Send.

  That is, for example, in the control frame (Control Frame StreamInfo) in the upper left part of the upper stage of FIG. 30, the segment data to be transmitted next has a segment number of 0 (Segment No. 0), and the segment representation is Video1 ( It is indicated that the information is RepID: Video1). Furthermore, information (Multiple Representation: True) indicating that there is a possibility of sending segment data of different representations with the same Segment No is transmitted. Note that Multiple Representationl: True corresponds to RepRep: true in FIG.

  Then, in step S114, segment data is transmitted as shown in the second from the left in the upper part of FIG.

  The communication processing unit 227 of the client 202 controls the receiving unit 221 to receive this segment data and start decoding. In step S115, the communication processing unit 227 controls the transmission unit 222 to transmit control data (playerInfo (CurrentPresentanTime)) to the server 201, thereby notifying the elapsed time from the start of stream decoding. To do.

  Thereafter, the communication processing unit 214 of the server 201 controls the transmission unit 212 to sequentially transmit segment data to the client 202. At this time, the communication processing unit 214 of the server 201 estimates the buffer amount of the buffer 223 of the client 202 from CurrentPresentanTime by PlayerInfo that changes the representation (Rep) by processing according to the network state (Network Adaption). Can adapt to the appropriate band.

  That is, for example, when it is determined that there is sufficient bandwidth, the communication control unit 214 of the server 201 determines that the segment data of the same segment number and different bit rate is transmitted. In step S116, a control data frame (Control Frame streamInfo) composed of control data (streamInfo (RepID 3, Segment No 0, RepRep: false)) is transmitted as shown in the lower leftmost column of FIG.

  At this time, the client 202 discards the segment data (RepID 1, Segment No 0) that has been transmitted immediately before and has a low bit rate.

  In step S117, the communication control unit 214 of the server 201 controls the transmission unit 212 so that the segment data (RepID 3,. A segment data frame (Media Data Frame 0) consisting of Segment No 0) is transmitted.

  That is, as shown in the lower left part of the lower part of FIG. 30, the segment data to be transmitted next has the same segment number as the segment data sent immediately before (Segment No. 0), and the bit rate is higher. A control frame composed of control data indicating Video3 (RepID: Video3) of the representation is transmitted. Subsequently, the communication control unit 214 of the server 201 controls the transmission unit 212, and in step S117, as shown second from the left in the lower part of FIG. 30, a segment data frame (RepID 3, SegmentNo. 0)) is sent as a segment data frame (Media Data Frame 0).

  Subsequently, the communication control unit 214 of the server 201 controls the transmission unit 212, and in step S118, as shown in the third from the left in the lower part of FIG. 30, a control frame (Control Frame) composed of control data (streamInfo). streamInfo) to the client 202. In step S 119, the communication control unit 214 of the server 201 controls the transmission unit 212, and segment data including segment data (RepID 3, SegmentNo 1) as shown in the fourth from the left in the lower part of FIG. 30. Send a frame (Media Data Frame1).

  The communication control unit 227 of the client 202 controls the reception unit 221 to receive the segment data and start decoding. In step S120, the communication control unit 227 controls the transmission unit 222 to transmit control data (playerInfo (CurrentPresentanTime)). To the server 201. As a result, the server 202 is notified of the elapsed time from the start of the stream decoding in the client 202.

  In addition, the communication control unit 214 of the server 201 can continue to send segment data as transmission progresses by the client 202 by transmission control. That is, the communication control unit 214 of the server 201 controls the transmission unit 212 in step S121 at the timing in accordance with the progress of reproduction of the client 202, and the control data as shown in the fifth from the lower left in FIG. A control frame (Control Frame streamInfo) composed of (streamInfo) is transmitted to the client 202.

  In step S122, the communication control unit 214 of the server 201 controls the transmission unit 212, and, as shown in the lower rightmost column of FIG. 30, a segment data frame (Media ID) including segment data (RepID 3, Segment No 2). Data Frame2) is transmitted.

  On the other hand, when the reproduction processing unit 224 changes the reproduction status, the communication control unit 227 of the client 202 controls the transmission unit 222 in step S123 to transmit the control data (playerInfo) to the server 201, thereby Notify the elapsed time since the start of decoding.

  As described above, at the start of reproduction, the server 201 switches the representation as necessary according to the state of the band and transmits the segment data to the client 202. When switching the representation of the segment data, the segment data that is the same as the segment number sent immediately before and is switched to the representation according to the band status is transmitted.

  If it is determined by the processing in step S115 that the current representation is appropriate for the current band, for example, the communication processing unit 214 of the server 201 orders the segment numbers of the segment data transmitted immediately before. 1 is incremented and the segment data of the same representation is transmitted. That is, in the next step, as shown in the third column from the left in the upper part of FIG. 30, the server 201 controls the control frame (Control Frame consisting of control data (streamInfo (RepID 1, SegmentNo 1, RepRep: false))). streamInfo) to the client 202.

  Then, in the next step, as shown in the fourth from the left in the upper part of FIG. 30, the server performs a segment data frame (Media Data Frame 1) composed of segment data (RepID 1, Segment No 1) notified in advance by the previous control data. Thereafter, the segment number is sequentially incremented by one, and the control data frame made up of control data and the segment data frame made up of segment data are sequentially sent alternately.

  As described above, a plurality of types of representations are transmitted for the same segment data at the start of playback in accordance with the band status. It becomes possible to transmit. As a result, it is possible to prevent playback from starting from an image with a low image quality, and it is possible to reproduce from an image with an appropriate image quality according to the state of the band. As a result, the operability in the reproduction process can be improved.

<4. Third Embodiment>
<Low delay Transfer by Sub-Segment>

  Next, a sequence for managing segment data with sub-segment data in smaller units will be described with reference to the flowchart of FIG.

  The sub-segment data is a unit smaller than the segment data. As a result, a higher-speed operation is realized than the processing in units of segment data, thereby preventing a transmission delay or the like.

  For example, the processing is started when the user performs an operation on the client 202 so as to start VOD playback.

  In step S141, the communication control unit 227 of the client 202 controls the transmission unit 222 to transmit control data (capability (subSegment = true), openMedia, playStream) to the server 201. As a result, the capability of the client 202 is notified, and a request is made to start the stream by specifying the media. At this time, when the capability can support transmission in units of subsegments, the communication control unit 227 of the client 202 controls the transmission unit 222 to confirm that transmission in units of subsegments is possible. Information (subSegment = true) is transmitted to the server 201. Accordingly, the communication control unit 214 of the server 201 can recognize that the client 202 supports transmission in units of subsegments.

  In accordance with this request, the communication control unit 214 of the server 201 reads the media information of the media designated by the client 202 from the storage unit 213, controls the transmission unit 212 in step S142, and sends control data (mediaInfo) to the client 202. The media information is returned by sending to.

  In step S143, the communication control unit 214 of the server 201 controls the transmission unit 212 to control data (streamInfo (RepID 1, SegmentNo 1, SubsegmentNo, SAPType 1) as shown in the leftmost part of FIG. ) Is transmitted to the client 202, and the attribute of the sub-segment to be transmitted next is notified.

  FIG. 32 shows a control frame (Control Frame) composed of control data transmitted when two segment data of different representations on the left and right in the figure are each composed of two sub-segment data, and sub-segments It is a figure explaining the transmission order of the media data frame (Media Data Frame) which consists of data.

  That is, in FIG. 32, from the left, the procedure for transmitting segment data in which the segment number of video1 (RepID: video1) is 1 (Segment No 1) and the representation is video2 (RepID: video2) The procedure when the segment data of (Segment No 2) with the segment number of 2 is transmitted is shown.

  The sub-segment data is, for example, a GOP (Group of Picture) when the segment data is MPEG image data, and is data in a constituent unit having a smaller data size than the constituent unit of the segment data.

  In this case, as shown in FIG. 32, since transmission is performed in units of subsegments, control data is also configured in units of subsegments.

  Accordingly, in FIG. 32, the control data (streamInfo (RepID video1, SegmentNo.) Is transmitted at the timing before the first sub-segment data (Media Data Frame: Segment Data of Subsegment 1) for the segment data on the left side in the figure is transmitted. 1, a control data frame (Control Frame) consisting of SubsegmentNo 1, SAPType 1)) is transmitted.

  Similarly, at the timing before the second subsegment data (Media Data Frame: Segment Data of Subsegment 2) for the left segment data in FIG. 32 is transmitted, the control data (streamInfo (RepID video1, SegmentNo 1, SubsegmentNo. 2, a control data frame (Control Frame) consisting of SAPType 3)) is transmitted.

  Subsequently, at the timing before the first sub-segment data (Media Data Frame: Segment Data of Subsegment 1) for the segment data on the right side in FIG. 32 is transmitted, the control data (streamInfo (RepID video2, SegmentNo 2, SubsegmentNo 1) , SAPType 1)) is transmitted.

  Then, at the timing before the second subsegment data (Media Data Frame: Segment Data of Subsegment 2) for the right segment in the figure is transmitted, the control data (streamInfo (RepID video2, SegmentNo 2, SubsegmentNo 2, SAPType A control data frame consisting of 2)) is transmitted.

  Here, it returns to description of FIG.

  Therefore, in this case, the communication control unit 214 of the server 201 controls the transmission unit 212 in step S144, so that the first subsegment data (Media Data Frame: Segment Data of Subsegment 1) for the left segment in FIG. (RepID video1, SegmentNo 1, SubsegmentNo 1, SAPType 1)).

  The communication control unit 227 of the client 202 controls the receiving unit 221 to receive this segment data and store it in the buffer 223. At this time, the reproduction processing unit 224 starts decoding the segment data buffered in the buffer 223. In step S145, the communication control unit 227 controls the transmission unit 222 to transmit control data (playerInfo) to the server 201, thereby notifying the elapsed time from the start of the stream decoding in the reproduction processing unit 224. To do. At this time, the client 202 can execute a process of displaying an image based on smaller data than the segment data in sub-segment units, so that the image can be displayed at a higher speed.

  In step S146, the communication processing unit 214 of the server 201 controls the transmission unit 212 to control the second sub-segment data of the segment data on the left side of FIG. 32 (streamInfo (RepID video1, SegmentNo 1 , SubsegmentNo 2, SAPType 3)) is transmitted to the client 202, and the attribute of the next subsegment to be transmitted is notified.

  Therefore, in this case, the communication control unit 214 of the server 201 controls the transmission unit 212 in step S147, so that the second subsegment data (Media Data Frame: Segment Data of Subsegment 2) for the left segment in FIG. (RepID video1, SegmentNo 1, SubsegmentNo 2, SAPType 3)).

  Here, when the user gives an instruction to stop playback, the communication control unit 227 of the client 202 controls the transmission unit 222 to transmit control data (closeMedia, stopStream) to the server 201 in step S148. As a result, the playback processing unit 224 of the client 202 ends the playback of all received segments stored in the buffer 223. Here, the communication processing unit 227 controls the transmission unit 222 to notify the server 201 that the medium is to be closed.

  When the communication processing unit 214 of the server 201 receives the control data (closeMedia, stopStream) by controlling the reception unit 211, the communication processing unit 214 discards the media information and starts the stop processing. In step S 149, the communication processing unit 214 of the server 201 controls the transmission unit 212 to transmit control data (MediaStatus) to the client 202.

  However, since the communication control unit 214 of the server 201 controls the transmission unit 212 to perform processing in units of sub-segments and cannot stop the processing for which transmission has been started, for example, in step S150, it continues. A control frame (Control Frame) made up of control data (streamInfo (RepID video2, SegmentNo2, SubsegmentNo1, SAPType2)) of the first subsegment data in the segment data on the right side of FIG. Then, the attribute of the sub-segment to be transmitted next is notified.

  Then, in step S151, the communication control unit 214 of the server 201 controls the transmission unit 212 to perform the first subsegment data (Media Data Frame: Segment Data of Subsegment 1 (RepID video2) for the right segment in FIG. , SegmentNo 2, SubsegmentNo 1, SAPType 2)).

  Finally, in step S152, the communication control unit 214 of the server 201 controls the transmission unit 212 to notify the client of the operation stop.

  As described above, the control data StreamInfo (SegmentNo, SubSegmentNo, SAP Type) is added and managed in sub-segment units smaller than the segment data unit, so that the image playback display time, the time until completion of the stop, In addition, it is possible to realize switching of contents and the like at higher speed. That is, as described above, since the process for which transmission has been started cannot be stopped, by managing transmission in units smaller than segment data, such as sub-segment data, the image display time, the time until completion of the stop, In addition, it is possible to realize switching of contents and the like at higher speed.

Further, for example, in live broadcasting, it is possible to further reduce transmission delay by configuring subsegments in units of GOPs, for example. Further, when there is no sub-segment information in the MPD, the communication control unit 214 of the server 201 analyzes (parses) the segment data, divides it into sub-segments if possible, and transmits them in units of sub-segments. In this case, the segment data can be managed more quickly. As a result, it is possible to improve the operability in reproduction.
<5. Fourth Embodiment>
<Auto Segment Skip for Live Streaming>
Next, a fourth embodiment to which the present technology is applied will be described with reference to FIG.

  FIG. 33 shows a sequence performed when network congestion occurs during live reproduction.

  For example, the processing is started when the user performs an operation on the client 202 so as to start live reproduction. In step S161, the client 202 transmits control data (capability, openMedia, playStream) to the server 201, notifies the client 202 of the capability, and requests to start the stream by specifying the medium. Here, as shown in FIG. 20 described above, in capability, it is possible to specify whether segment skip can be supported by segmentSkip. In the example of FIG. 33, it is possible to handle segment skip. It is set to “true”.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by transmitting control data (mediaInfo) to the client 202 in step S162.

  Here, for example, when live broadcasting has already started, transmission is performed from the latest segment. In step S163, the server 201 transmits segment data (MediaDisc 1, SegmentNo 1699) to the client 202 in step S164 after transmitting the control data (streamInfo) to the client 202.

  Thereafter, segment data is sequentially transmitted, and the server 201 transmits control data (streamInfo) in step S165, and then transmits segment data (MediaDisc 1, SegmentNo 1700) in step S166.

  In step S167, the client 202 transmits control data (playerInfo) to the server 201. For example, the control data (playerInfo) includes currentPresentationTime and BufferedTime as shown in FIG.

  At this time, it is assumed that network congestion is eliminated after network congestion occurs and transmission / reception between the server 201 and the client 202 becomes impossible.

  When network congestion occurs in this way, the server 201 takes time to transmit the previous segment data (SegmentNo 1700), and when it is determined that the segment No 1701 is not reproduced, the server 201 transmits the segment No 1701. skip.

  That is, the server 201 transmits control data (mediaInfo) and control data (streamInfo) in steps S168 and S169, and then transmits segment data (MediaDisc 1, SegmentNo 1701) in step S170. That is, the server 201 resumes transmission from the segment that is played live by the client 202.

  For example, in addition to SegmentNo and SubSegment signaling, the server 201 can signal how many segments are buffered to the client 202 adaptively to the band. That is, the server 201 can signal currentPresentationTime or BufferedTime by the control data (playerInfo) transmitted from the client 202 in step S167. The currentPresentationTime indicates an elapsed time since the playback processing unit 224 starts decoding the stream in the client 202. BufferedTime indicates the time during which playback can be continued with the segment data held in the buffer 223 of the client 202, and is equivalent to the time when playback can be continued in a state where no segment data can be received from the network. is there.

  In other words, in the client 202, the communication processing unit 227 acquires an elapsed time since the playback processing unit 224 started decoding the stream, for example. In addition, the communication processing unit 227 obtains a playback continuation possible time during which playback can be continued only with the segments held in the buffer 223. The communication processing unit 227 notifies the server 201 of the elapsed time or the playback continuation time as appropriate. Accordingly, in the server 201, the communication processing unit 214 receives the control data (playerInfo) in step S167, thereby grasping the reproduction status of the live reproduction in the client 202, and returning due to occurrence of network congestion or the like. At this time, communication processing can be performed so as to skip transmission of a segment that is not reproduced, and the transmission unit 212 can transmit the segment.

  Therefore, for example, since unnecessary segments that are not reproduced are not transmitted, the bandwidth can be used more effectively, and when network congestion occurs during live reproduction, appropriate restoration can be performed.

Note that the control data (playerInfo) for notifying the currentPresentationTime and BufferedTime does not need to be transmitted every time. For example, the client 202 transmits when playback is started or when playback is resumed due to data depletion. be able to.
<6. Fifth embodiment>
<Fast Switching of multiple media>
Next, a fifth embodiment to which the present technology is applied will be described with reference to FIGS. 34 and 35.

  34 and 35 show a sequence for switching and reproducing a plurality of media.

  For example, the process starts when the user performs an operation on the client 202 so as to start reproduction. In step S181, the client 202 transmits the control data (capability, openMedia, playStream) to the server 201, notifies the client 202 of the capability, and requests the start of the stream by specifying the medium.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by transmitting control data (mediaInfo) to the client 202 in step S182. At this time, in the example of FIG. 34, the medium to be reproduced is MediaDesc 1, and the server 201 secures resources for distributing MediaDesc 1.

  In step S183, the server 201 transmits control data (streamInfo) to the client 202 to notify the attribute of segment data to be transmitted next. Subsequently, the server 201 transmits segment data (MediaDesc 1, Segment No 1447) to the client 202 in step S184. At this time, for example, in the case of live reproduction and broadcasting has already started, transmission is performed from the latest segment.

  The client 202 receives this segment data and starts decoding, and transmits control data (playerInfo) to the server 201 in step S185.

  Subsequently, the server 201 continues to transmit segment data, and transmits control data (streamInfo) and segment data (MediaDesc 1, SegmentNo 1448) to the client 202 in steps S186 and S187, respectively.

    While the client 202 receives and decodes the segment data, in step S188, the client 202 transmits the control data (openMedia) to the server 201 and requests to open by designating another medium.

  In accordance with this request, the server 201 reads the media information of the media designated by the client 202, and returns the media information by sending control data (mediaInfo) to the client 202 in step S189. At this time, in the example of FIG. 34, the opened medium is MediaDesc 2 different from MediaDesc 1 being played back, and the server 201 also secures resources for distributing MediaDesc 2 (open processing).

  Then, the server 201 continues to transmit segment data, and transmits control data (streamInfo) and segment data (MediaDesc 1, Segment No 1449) to the client 202 in steps S190 and S191, respectively. Subsequently, as shown in FIG. 35, control data (streamInfo) and segment data (MediaDesc 1, Segment No 1450) are transmitted to the client 202 in steps S192 and S193, respectively.

  At this time, when the user performs an operation on the client 202 so as to switch the media, in step S194, the client 202 transmits control data (changeMedia) to the server 201, and the server 202 switches the streamed media to MediaDesc 2. Request to 201.

  Here, the media information of MediaDesc 2 has already been transmitted to the client 202 in step S189 of FIG. Accordingly, the server 201 transmits segment data (MediaDesc 2, Segment No. 7337) in step S196 following transmission of control data (streamInfo) in step S195. Thereby, stream reproduction is continued without interruption.

  Then, the server 201 continues to transmit segment data, and transmits control data (streamInfo) and segment data (MediaDesc 2, Segment No 7338) to the client 202 in steps S197 and S198, respectively.

  When the user performs an operation on the client 202 again to switch the media, in step S199, the client 202 transmits control data (changeMedia) to the server 201, and switches the streamed media to MediaDesc 1. Request to the server 201.

  Here, the media information of MediaDesc 1 has already been transmitted to the client 202 in step S182 of FIG. Accordingly, the server 201 transmits segment data (MediaDesc 1, Segment No. 1452) in step S201 following transmission of control data (streamInfo) in step S200. At this time, the server 201 transmits from the latest segment in the case of live playback, and transmits from the continuation of the previous media segment in the case of VOD playback.

  Then, the server 201 continues to transmit segment data, and transmits control data (streamInfo) and segment data (MediaDesc 1, SegmentNo 1453) to the client 202 in steps S202 and S203, respectively.

  In this way, a process of switching and reproducing a plurality of media is performed, and similarly, segment data is transmitted from the server 201 to the client 202.

  As described above, the server 201 can open a plurality of media in advance in response to a request from the client 202. Thus, the client 202 transmits the control data (changeMedia) in the stream, so that the server 201 can transmit the segments of each medium without breaks. That is, since the client 202 does not need to load a new MPD (media information) when switching media, the client 202 can smoothly switch channels and play media.

  Since how many media can be opened depends on the capability of the server 201, the client 202 may make an inquiry about whether the media can be opened to the server 201. It should be noted that MPD information can be maintained statefully in the OpenMedia and CloseMedia commands. Also, in response to the client 202 sending OpenMedia, the server 201 returns a MediaDescriptor.

  For example, in the case of a broadcast type service, channels can be switched at high speed by opening a plurality of media from the beginning.

<7. Sixth Embodiment>
<Computer>
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer that can execute various functions by installing a computer incorporated in dedicated hardware and various programs.

  FIG. 36 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer 900 shown in FIG. 36, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are connected to each other via a bus 904.

  An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

  The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 912 includes, for example, a display, a speaker, an output terminal, and the like. The storage unit 913 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit 914 includes a network interface, for example. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 901 loads the program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the program, for example. Is performed. The RAM 903 also appropriately stores data necessary for the CPU 901 to execute various processes.

  The program executed by the computer (CPU 901) can be recorded and applied to, for example, a removable medium 921 as a package medium or the like. In that case, the program can be installed in the storage unit 913 via the input / output interface 910 by attaching the removable medium 921 to the drive 915.

  This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

  In addition, this program can be installed in the ROM 902 or the storage unit 913 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

  In addition, in the above description, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit). Of course, a configuration other than that described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). .

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  For example, the present technology can take a configuration of cloud computing in which one function is shared by a plurality of devices via a network and jointly processed.

  In addition, each step described in the above flowchart can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  The information processing apparatus according to the above-described embodiment includes, for example, a transmitter or a receiver, an optical disc, a magnetic disc, a wired broadcast such as a satellite broadcast and a cable TV, a delivery on the Internet, and a delivery to a terminal by cellular communication. The present invention can be applied to various electronic devices such as a recording device that records an image on a medium such as a flash memory or a reproducing device that reproduces an image from the storage medium.

In addition, this technique can also take the following structures.
(1)
In response to a request from a playback device that plays media, an open processing unit that performs processing to open a plurality of media in advance;
An information processing apparatus comprising: a transmission unit that starts transmission of the media after switching without performing processing to open the media when switching of the media is requested from the playback device.
(2)
When there is a request to specify and open the media different from the predetermined media when the playback device is playing the predetermined media, the open processing unit, in response to the request, The information processing apparatus according to (1), wherein media information of another medium designated by the playback apparatus is transmitted to the playback apparatus.
(3)
In response to a request from a playback device that plays back media, perform processing to open multiple media in advance,
An information processing method including a step of starting transmission of the media after switching without performing processing of opening the media when switching of the media is requested from the playback device.

  Note that the present embodiment is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the present disclosure.

100 delivery system, 101 delivery data generation device, 102 delivery server, 1
03 terminal device, 104 network, 201 server, 202 client

Claims (3)

In response to a request from a playback device that plays media, an open processing unit that performs processing to open a plurality of media in advance;
An information processing apparatus comprising: a transmission unit that starts transmission of the media after switching without performing processing to open the media when switching of the media is requested from the playback device. When there is a request to specify and open the media different from the predetermined media when the playback device is playing the predetermined media, the open processing unit, in response to the request, The information processing apparatus according to claim 1, wherein media information of another medium designated by the playback apparatus is transmitted to the playback apparatus. In response to a request from a playback device that plays back media, perform processing to open multiple media in advance,
An information processing method including a step of starting transmission of the media after switching without performing processing of opening the media when switching of the media is requested from the playback device.

Images