KR101986995B1 - Media playback apparatus and method for synchronously reproducing video and audio on a web browser - Google Patents

Abstract

The present invention relates to a media playback device and method for synchronizing and playing video and audio in a web browser. In the present invention, the audio is buffered to be output at the time point at which the video is output, so that the output of the video and the audio is synchronized. In addition, according to another embodiment of the present invention it is possible to synchronize the video and the audio in accordance with the time the audio is output.
According to the present invention, even if video and audio are decoded by different decoders, synchronization does not become a problem, thereby providing an environment in which video and audio can be processed separately. Therefore, it is possible to implement and use the decoder separately from the decoder embedded in the web browser in the plug-in environment, thereby reducing the dependency on the codec format of the media according to the decoder.

Description

Media playback apparatus and method for synchronously reproducing video and audio on a web browser}

The present invention relates to an apparatus and method for playing media in a web browser, and more particularly, to an apparatus and method for playing media by synchronizing video and audio when decoding video or audio in a container unit. .

In order for a user to play media data in a web browser via the Internet, a method using a plug-in in which a code decoder and a renderer is written in native code is known. ActiveX and NPAPI are typically used as web browser plug-ins.

ActivieX is a combination of two technologies, Microsoft's Component Object Model (COM) and Object Linking and Embedding (OLE). However, in a narrow sense, it refers to an ActiveX control that is used in the form of add-ons in Internet Explorer, a web browser. ActiveX is used to play media in Internet Explorer web browsers.

NPAPI is an API that was developed in the Netscape era and is similar in function to Internet Explorer's ActivieX. When the initial web environment is poor, it is an API provided to draw external applications in the form of plug-ins to enhance the function of the web browser. In other words, it was developed to play music or video in an initial web page, such as Java's Applet, Adobe's Flash, and Real Player.

But plug-ins are used as a way for hackers to distribute malicious code, which is no longer supported by web browser vendors. NPAPI does not support NPAPI anymore since Google Chrome 45, which produces and distributes Chrome.In the case of ActiveX, the Edge version used by Microsoft as the default browser of Windows 10 is no longer supported. It no longer supports ActivieX.

One way to play media in a web browser without a plug-in is to perform decoding and rendering using HTML5 video tags. HTML5 is the fifth markup language for HTML, developed by W3C and Web browser developers, and provides video playback without plug-ins. Starting with Chrome 4.0, Explorer 9.0, Firefox 3.5, Safari 4.0, and Opera 10.5, HTML5 video tags are fully supported.

When decoding using video tags, high resolution video and high fps video can be processed with good performance. However, video formats supported by video tags have limitations and currently only support three formats: MP4, WebM, and Ogg. More specifically, it supports codec formats such as H.264 for MP4, ACC, MP3, VP8, VP9 for WebM, Vorbis, Opus, and Theora, Vorbis for Ogg, and video codec format H.265 or audio codec format G. .711 and G.726 are not supported in the video tag.

The limitations due to the codec format limitation of the video tag can be compensated for by using a decoder implemented in JavaScript. JavaScript decoders can decode unlimited video formats, so it is possible to decode media in codec formats that are not supported by video tags.

However, there are still limitations to this supplement. Since video tag basically receives and decodes video in a container unit that packs a plurality of frames, synchronization is problematic when decoding video and audio using different decoders of video tag decoder and JavaScript decoder. It can be. For example, if H.264 format video is decoded by video tag on a container basis and G.711 format audio is decoded by a JavaScript decoder, the number of frames used in a container is 30 and the video fps is 1 In this case, video may be output after a maximum of 30 seconds and audio may be played first.

Accordingly, the present invention is to propose a method that can play media without the problem of video and audio synchronization.

SUMMARY OF THE INVENTION An object of the present invention is to propose a media playback method in which synchronization of video and audio is not a problem when video is decoded in container units.

The objects of the present invention are not limited to the above-mentioned objects, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a media player for synchronizing and playing video and audio on a web browser according to an embodiment of the present invention may receive media data generated by a media service apparatus using a communication protocol supporting a web service. Receiving unit; A data separator that separates the received media data into video data and audio data; A container unit for packaging the frames constituting the video data into unit numbers and converting the frames into chunk data; Reconstructs the video by decoding the converted chunk data by a decoder embedded in a web browser, and when the output of the reconstructed video is output, time information about a time point at which the reconstructed video is output, in units of the chunk data. A media restoration unit provided; And an audio sink configured to output the audio data in synchronization with the restored video based on time information provided in the chunk data unit.

The media service apparatus according to the embodiment of the present invention for solving the above problems is a media service apparatus for transmitting media data to the media playback apparatus, the script required to play the media data on a web browser of the media playback apparatus. A module storage unit for storing a module; A module transmitter for transmitting the script module to the media player in response to a connection of the media player; A packetizer configured to packetize the media data to generate a transport packet; And a web server for establishing a communication session with the playback device, the web server for transmitting the transport packet to the media playback device in response to a request from the media playback device, wherein the script module is further configured to send the communication packet to the communication session. A process of receiving through a process, a process of packaging video frames included in the communication packet into unit numbers and converting the chunk data into chunk data, and the converted chunk data are decoded and output by a media restoring unit installed in the media player. The audio data included in the communication packet may be synchronized with the chunk data and output based on the time information about the viewpoint.

In addition, the media playback apparatus for synchronizing and playing video and audio on a web browser according to an embodiment of the present invention for solving the above problems, receives the media data generated by the media service apparatus using a communication protocol supporting a web service Receiving unit; A data separator configured to separate first media type data and second media type data from the received media data; A container unit for packaging the frames constituting the first media data into unit numbers and converting the frames into chunk data; Decode the converted chunk data by a decoder embedded in a web browser to restore a first media, and time information regarding a time point at which the restored first media is output when the restored first media is output. A media restoring unit for providing a chunk data unit; And a sink unit configured to output the second media data in synchronization with the restored first media based on time information provided in the chunk data unit.

 Other specific details of the invention are included in the description and drawings.

According to embodiments of the present invention has at least the following effects.

By solving the above problem, it is possible to reproduce the synchronized media by restoring the video and the audio according to the container specification.

In addition, even if video and audio are decoded by different decoders by the solution of the above problem, synchronization does not become a problem, thereby providing an environment in which video and audio can be processed separately.

In addition, audio in codec formats that are not supported by video tags can be used without synchronization issues, thereby reducing the dependency on codec formats.

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 shows an embodiment of an overall system for media playback.
FIG. 2 is a diagram illustrating a TCP / IP four-layer model hierarchically defined for communication between devices.
3 illustrates a process of making a web socket connection between the media service apparatus 110 and the media playback apparatus 120.
4 shows an example of a process of transmitting and receiving data through a web socket connection.
5 is a diagram illustrating a structure of a communication packet transmitted through the network interface 21.
6 illustrates one embodiment of a media service device 110.
7 illustrates another embodiment of a media service device 110.
8 illustrates an embodiment of a script module.
9 illustrates another embodiment of a script module.
10 illustrates another embodiment of a script module.
11 shows another embodiment of a script module.
12 illustrates one embodiment of a media playback device 120.
13 shows another embodiment of a media playback device 120.
14 shows another embodiment of a media playback device 120.
15 shows a process of generating chunk data.
16 shows an audio chunk generation process.
17 is an exemplary diagram for describing a process of generating a script module implemented with JavaScript according to an embodiment of the present invention.
18 is an illustration of a computing device 400 for implementing a media playback device 120.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 shows an embodiment of an overall system for media playback. The overall system of FIG. 1 is comprised of a media service device 110 and a media playback device 120 and a network 430 connecting the two devices.

The media service device 110 includes a computing or processing device suitable for providing computing services for one or more video playback devices. For example, the media service apparatus 110 includes a device capable of generating or storing an image stream and transmitting the same to a user device, such as a network camera, a network video recorder (NVR), a digital video recorder (DVR), or the like.

Media playback device 120 includes a computing or processing device suitable for interacting with media service device 110 or other computing user devices via a network. For example, media playback device 120 may include a desktop computer, mobile phone or smartphone, personal digital assistant (PDA), laptop computer, and tablet computer.

Media captured or stored in real time by the media service device 110 is transmitted through the network 430 at the request of the media playback device 120. The user can play the transmitted media through a user interface implemented on the web browser 210 of the media playback device 120. In this case, the web browser 210 may include not only commonly known browsers mounted on desktop computers or mobile phones, such as Google Chrome, Microsoft Explorer, Mozilla Firefox, Apple Safari, etc. It also covers software applications produced separately.

Hereinafter, the RTSP / RTP protocol transmitted on a web socket, which is a network communication method between the media service apparatus 110 and the media playback apparatus 120, will be described with reference to FIGS.

FIG. 2 is a diagram illustrating a TCP / IP four-layer model hierarchically defined for communication between devices.

The four layers consist of a network interface layer 21, an internet layer 22, a transport layer 23, and an application layer 24. In the RTSP / RTP protocol transmitted on the websocket, the websocket connection is located on top of the transport layer 23 connection. Therefore, in order to use the websocket connection, the TCP transport connection is performed by the media service device 110 and the media playback device 120. Must be established first. Once the websocket connection is established between the media service device 110 and the media playback device 120 via, for example, a three-way handshake process, the websocket communication is performed by sending websocket packets. Is performed. The websocket connection and the websocket packet will be described in detail with reference to FIGS. 3 to 5 below.

3 illustrates a process of making a web socket connection between the media service apparatus 110 and the media playback apparatus 120.

The media playback device 120 requests the media service device 110 to initiate a websocket connection using the websocket URI. The WebSocket URI can be obtained using the GetServiceCapabilities command. The web socket URI is expressed in the same manner as, for example, "ws: //192.168.0.5/webSocketServer" (S1000).

The media playback device 120 makes a web socket upgrade request to the media service device 110. The media service apparatus 110 responds with a 101 code indicating a status code of acknowledging the protocol switch request (S1100).

After the websocket connection is established between the media service device 110 and the media playback device 120, data is exchanged using the RTSP / RTP protocol transmitted over the websocket, not the HTTP / 1.1 protocol. DESCRIBE, SETUP, PLAY, PAUSE, and TEARDOWN in FIG. 3 are RTSP commands. The DESCRIBE request includes a URL. The response message to DESCRIBE also contains a description of what was requested. The SETUP request specifies whether a single media stream should be sent. The PLAY request is a request to play one or all media streams. Multiple requests can be made. The PAUSE request commands a pause for one or all media streams. You can restart it with a PLAY request. The TEARDOWN request is a command to terminate a session. In response to the TEARDOWN request, playback of all media streams is stopped and all sessions associated with related data are released (S1200).

Examples of the request message sent from the media playback device 120 and the response message of the media service device 110 in the websocket connection process shown in FIG. 3 are described in Table 1 below.

Media playback device 120-> Media service device 110
GET / webSocketServer HTTP / 1.1
Host: 192.168.0.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ ==
Origin: http://example.com
Sec-WebSocket-Protocol: rtsp.onvif.org
Sec-WebSocket-Version: 13.

Media service device 110-> media playback device 120
HTTP / 1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK + xOo =
Sec-WebSocket-Protocol: rtsp.onvif.org

This websocket connection is made according to the websocket protocol, which is an HTML5 standard. In particular, since the websocket connection continuously supports two-way communication, it is possible to continuously transmit and receive data without disconnecting the connection between the media service device 110 and the media playback device 120.

4 shows an example of a process of transmitting and receiving data through a web socket connection.

Referring to FIG. 4, first, the media playback device 120 transmits a TCP / IP connection request message to the media service device 110, and the media service device 110 accepts the TCP / IP connection request. Sending a message SYN-ACK to the media playback device 120 establishes a TCP / IP connection. TCP transport connections can be formed by pairs of local and remote TCP sockets, each TCP socket being defined by at least an identifier such as a port number and an IP address. Of course, instead of such a TCP / IP-based connection may establish a UDP / IP-based connection between the two.

Thereafter, when the websocket connection is established between the media player 120 and the media service device 110 through a handshake process, it is possible to continuously transmit and receive data thereafter. That is, the media playback device 120 transmits a media streaming request to the media service device 110 in the form of a transmission web socket packet (socket.send), and the media service device 110 sends media to the media playback device 120. Send the stream in the form of a response WebSocket packet (socket.onMessage). This process can be continued between the two until the media stream transmission is stopped or completed.

5 is a diagram illustrating a structure of a communication packet transmitted through the network interface 21.

When the RTP header 44 is added to the RTP payload corresponding to the data 45, it becomes an RTP packet. The RTP packet is the same as the websocket payload, and the websocket header 43 is added to the websocket packet. The websocket packet is the same as the TCP payload, and the TCP header 42 is added thereto to form a TCP packet. Finally, the TCP packet is equal to the IP payload, and when the IP header 41 is added thereto, the communication packet, that is, the IP packet is finally completed. As such, the process of completing the IP packet and the process of removing each header are performed by the media service apparatus 110 and the media playback apparatus 120.

Since the communication between the media service apparatus 110 and the media playback apparatus 120 is performed through the HTML5-based websocket protocol described above with reference to FIGS. 2 to 5, the module in charge of RTSP / RTP transmission / reception control is parsed in HTML5. This can be implemented by script code. For example, it can be implemented in JavaScript, a script that can be parsed in HTML5. Therefore, media playback using the RTSP / RTP protocol can be implemented in a web browser in an HTML5 environment without installing a plug-in as in the related art.

So far, the network communication method between the media service apparatus 110 and the media playback apparatus 120 has been described. Hereinafter, the configuration and operation of the media service apparatus 110 and the media playback apparatus 120 will be described with reference to FIGS. 6 to 18.

6 illustrates one embodiment of a media service device 110. In one embodiment, the media service apparatus 110 includes a real-time video camera 111, an encoder 112, a packetizer 113, a web server 114, a module storage 115, a module transmitter 116, and The control unit 117 is included.

The real-time video camera 111 is a means for capturing the media in real time. The recording includes a method of performing both video capture and audio recording and a method of performing only video capture.

The encoder 112 performs a function of compressing and encoding media photographed by the real time video camera 111. The encoding by the encoder 112 does not necessarily have to be in a specific codec format supported by the decoder embedded in the web browser, but may be encoded in any codec format.

The packetizer 113 generates a transport packet by packetizing the encoded media data. Packetization means dividing the media data into appropriate lengths for easy transmission through the network 430, or assigning control information such as an address to be received to each of the media data if the media data is short. At this time, the control information is located in the header of the transport packet. The transport packet is in the form of the above-described web socket packet.

The packetizer 113 may packetize media data according to a method requested by the media reproducing apparatus 120. For example, when the media playback device 120 requests a frame-by-frame video, the packetizer 113 generates a transport packet in a frame format, and the media playback device 120 supports a container unit supported by a decoder embedded in a web browser. When the video request is received, the packetizer 113 may generate a transport packet in a container format.

The web server 114 performs a function of establishing a communication session with the media playback device 120. That is, a web socket connection is established between the web server 114 of the media service device 110 and the media playback device 120 through a handshake process. Thereafter, the transmission packet generated by the packetizer 113 is transmitted through the web server 114 at the request of the media playback apparatus 120.

The module storage unit 115 is a module that stores a script module for playing media in the media playback device 120. The script module is a code written as a script that can be parsed by a web browser, and is a module that enables media playback in a web browser in an HTML5 environment without installing a plug-in or a separate application in the media playback device 120. The script module may be, for example, code written in JavaScript. The script module will be described later with reference to FIGS. 8 to 11.

The module transmitter 116 is a module that transmits the script module stored in the module storage 115 to the media player 120. The module transmitter 116 transmits the script module in response to the media playback device 120 accessing the media service device 110 through the network 430.

The controller 117 is a module that controls other configuration modules of the media service apparatus 110. For example, when the media playback device 120 is connected to the web server 114 via the network 430, the script module stored in the module storage 115 is transferred to the media playback device 120 through the module transmission unit 116. The control unit 117 transmits and receives a signal from each module to control the operation to be performed smoothly.

An operation method will be described based on the description of the configuration modules of the media service apparatus 110 of FIG. 6. When the media player 120 is connected to the web server 114 via the network 430, the module transmitter 116 transmits the script module stored in the module storage 115 to the media player 120. . When the script module is installed in the media player 120, the user requests media playback through the user interface. Accordingly, the media service apparatus 110 encodes real-time live media captured by the real-time video camera 111 in the encoder 112 and packetizes the media data into a transport packet in accordance with a frame or container format by the packetizer 113 to form a web. The server 114 transmits the data to the media playback device 120.

7 illustrates another embodiment of a media service device 110. The media service apparatus 110 of FIG. 6 is an embodiment for transmitting real-time live media by the real time video camera 111, and the media service apparatus 110 of FIG. 7 stores media data stored in the media storage unit 118. An embodiment for transmitting is shown.

Although the media storage unit 118 includes a network video recorder (NVR) or a personal recorder (PVR), the embodiment of FIG. 7 will be described taking the case of a network video recorder as an example.

The media storage unit 118 receives and compresses media data from a camera or a server and stores the compressed data. When the media data transmission request is received from the media player 120, the media service device 110 packetizes the media data stored in the media storage unit 118 in the packetizer 113 and transmits the packetized data through the web server 114. .

The packetizing unit 113, the web server 114, the module storage unit 115, the module transmission unit 116, and the control unit 117 of the media service apparatus 110 configuration module of FIG. 7 are described in the embodiment of FIG. 6. Overlapping descriptions of modules are omitted.

8 through 11 illustrate embodiments of a script module transmitted from the media service apparatus 110 to the media playback apparatus 120. The script module may be implemented as a script that can be parsed by a web browser, and the embodiments of FIGS. 8 to 11 describe a case where the script module is implemented as JavaScript.

8 illustrates an embodiment of a script module. The script module of FIG. 8 is a module for restoring video on a frame basis and includes an RTSP / RTP client module 121, a depacketization module 122, a JS decoder module 124, and a JS renderer module 125. do.

The RTSP / RTP client module 121 performs a function of supporting RTSP / RTP communication with the media service device 110. The transport packet may be received from the web server 114 of the media service apparatus 110 through the RTSP / RTP client module 121. At present, it is not possible to process media according to RTSP / RTP on a web browser without a plug-in. If the RTSP / RTP client module 121 is used, the data transmitted in the RTSP / RTP protocol is stably transmitted even though the web browser uses the HTTP method. Can be received.

The depacketization module 122 is a module for depacketizing a packet received from the RTSP / RTP client module 121. Depacketization is the opposite concept of packetization. If packetization means that media data is formed by dividing the media data into proper lengths, the packet is re-connected to restore the media data to the state before packetization.

The JS decoder module 124 is a module that decompresses, or decodes, the encoded video. The JS decoder module 124 may be implemented in JavaScript like other modules of the script module. Since the JS decoder module 124 is implemented in JavaScript, it is possible to decode any codec format without restriction on the codec format unlike the decoder embedded in the web browser. It can also be decoded frame by frame.

If the JS decoder module 124 is implemented in JavaScript according to the embodiment of FIG. 8, for example, the JS decoder module 124 may be represented by code such as Table 2 and Table 3 below (Table 3 is the content following Table 2).

function HevcDecoder () {

var _name = "HevcDecoder";
var self = this;

this._decoderParameters = null;
this._isRequestActive = false;
this._player = null;
this._requestContext = null;
this._requestContextQueue = [];

this.pushRequestContext = function (requestContext) {
self._requestContextQueue.push (requestContext);
};
this.decode = function () {
if (self._isRequestActive) {
return;
}

if (self._requestContextQueue.length) {
self._isRequestActive = true;
self._requestContext = self._requestContextQueue.pop ();
self._playStream (self._requestContext.stream);
}
};
this._createDecodeCanvas = function (parentElement) {
self.canvas = document.createElement ("canvas");
self.canvas.style.display = "none";
self.canvas.id = "decode-canvas";
parentElement.appendChild (self.canvas);
self.ctx = self.canvas.getContext ("2d");
};
this._playStream = function (buffer) {
this._reset ();
this._handleOnLoad (buffer);
}
this._onImageDecoded = function (image) {
var width = image.get_width ();
var height = image.get_height ();
this.canvas.width = width;
this.canvas.height = height;
this._imageData = this.ctx.createImageData (width, height);

image.display (this._imageData, function (displayImageData) {
var itemId = self._requestContext.itemIds [self._requestContext.currentFrameIndex];
var payload = self._requestContext.payload;
if (height> payload.displayHeight) {
payload.displayHeight = height;
}

if (! (itemId in self._requestContext.dependencies)) {
if (width> payload.displayWidth) {
payload.displayWidth = width;
}
payload.frames.push ({
canvasFrameData: displayImageData.data,
itemId: itemId,
width: width,
height: height
});
}
self._requestContext.currentFrameIndex ++;
if (self._requestContext.currentFrameIndex> = self._requestContext.itemIds.length) {
self._requestContext.callback (payload);
self._isRequestActive = false;
self.decode (); // Decode next queued request
}
});
};

...

this._createDecodeCanvas (document.documentElement);
this._reset ();
}

The JS renderer module 125 renders the video decoded by the JS decoder module 124 to be displayed on an output device such as a monitor. The JS renderer module 125 uses WebGL and can convert YUV format video to RGB format. WebGL is a web-based graphics library that lets you create 3D graphical interfaces that can be used via JavaScript.

9 illustrates another embodiment of a script module. The script module of FIG. 9 is a module used to restore video in a container format, and includes an RTSP / RTP client module 121, a depacketizing module 122, and a container generating module 127. The RTSP / RTP client module 121 and the depacketization module 122 will not be described repeatedly with the modules described with reference to FIG. 8.

Looking at the script module of FIG. 9 in detail, it can be seen that the container generation module 127 is included unlike the case of FIG. 8. When the video is not packaged in a container unit, the container generation module 127 may perform a function of packaging a frame by a unit number to form a container. In addition, video frames and audio may be packaged in unit numbers. In this case, the container generation module 127 may variably adjust the number of units according to frames per second (fps) of the video. In the container generation module 127, packaging of video frames by a unit number or packaging of video frames and audio by a unit number will be referred to as chunk data.

A container is a concept including a container supported by a video tag such as an MPEG-DASH container. Since the container generation module 127 can organize the media in a container format that is compatible with HTML5 video tags, the use of the video tag in the media playback device 120 can be easily performed without the compatibility problem even if the image capturing device does not transmit the media in the container format. To make it possible. In other words, it provides an environment in which a video tag can be used without modifying an existing image capturing apparatus.

10 illustrates another embodiment of a script module. The script module of FIG. 10 includes a RTSP / RTP client module 121, a depacketization module 122, and an audio transcoder 123 as a module for audio reproduction. The RTSP / RTP client module 121 and the depacketization module 122 will not be described repeatedly with the modules described with reference to FIG. 8.

The audio transcoder 123 is a module that performs transcoding when audio data is configured in a codec format not supported by a decoder embedded in a web browser. Transcoding is a conversion of a codec format. The audio transcoder 123 decompresses audio data and performs compression again in a codec format different from the previous one. For example, it converts audio data of G.711 codec format which is not supported by video tag which is a decoder embedded in web browser into codec format of ACC supported by video tag.

Since the audio transcoder 123 converts the audio data into a codec format supported by the decoder embedded in the web browser, the audio transcoder 123 can restore the audio data together with the video data using the decoder embedded in the web browser and output the same without a synchronization problem. Provide an environment that is

11 shows another embodiment of a script module. The script module of FIG. 11 has a different module or configuration for audio reproduction similar to the script module of FIG. 10. The script module of FIG. 11 includes an RTSP / RTP client module 121, a depacketizing module 122, an audio decoder 126, an audio chunk unit 128, and a buffer controller 129. 121 and the depacketization module 122 are the modules described with reference to FIG. 8.

The audio decoder 126 is a module that performs decoding of audio data. Like other modules, the audio decoder 126 may be implemented in JavaScript, which is a script that can be parsed by a web browser.

The audio data decoded by the audio decoder 126 is packaged into an audio chunk in the audio chunk 128. In this case, the audio chunk is generated by packaging the audio data in the unit number of frames constituting the chunk data generated by the container generation module 127. That is, the audio chunk is generated in dependence on the chunk data. FIG. 12 illustrates a media player 120 including both the container generation module 127 and the audio chunk unit 128. The audio chunk generation in FIG. 12 will be described again.

The buffer controller 129 receives the audio chunk from the audio chunk 128 and buffers the audio chunk in the audio buffer, and provides the buffered audio chunk to the audio renderer 136. The audio renderer 136 is shown in FIGS. 12 and 13 as a configuration module of the media playback device 120.

The buffer controller 129 receives time information from another module for restoring the video and synchronizes the audio with the video. The time information is information about a time point at which the chunk data is decoded and output by the media restoring unit 143 installed in the media playback device 120. The start portion of the chunk data is determined by a renderer in the media playback device 120. Indicates the time to render. The buffer controller 129 synchronizes the video with the audio by buffering the audio chunk or delivering the audio chunk to the audio renderer 136 so that the audio is output at the time when the video is displayed through the time information.

The script module of FIG. 11 can be configured in a different order, which moves the audio decoder 126 to the rear end of the buffer controller 129. In this case, synchronization of the chunk data and the audio data is first performed by the buffer controller 129, and the audio decoder 126 decodes the buffered audio data.

The script module described with reference to FIGS. 8 to 11 is transmitted from the media service apparatus 110 to the media playback apparatus 120 as the media playback apparatus 120 connects to the media service apparatus 110, and thus the media playback apparatus 120. In the web browser 210 of) provides a environment that enables media playback without a plug-in. That is, the script module is installed in the media playback device 120 to configure a system for media playback. An embodiment of the media player 120 having the script module installed will be described below with reference to FIGS. 12 to 14.

12-14 illustrate embodiments of media playback device 120. The main modules for media playback in the media playback device 120 are constituted by a script module. Since the functions of the script module have been described with reference to FIGS. 8 to 11, FIGS. 12 to 14 illustrate the media playback apparatus 120 based on the structure and operation method.

12 illustrates one embodiment of a media playback device 120. The media player 120 of FIG. 12 includes a receiver 141, a data separator, a container 142, a media reconstructor 143, and an audio sink 144. In this case, the receiver 141, the data separator, the container 142, and the audio sink 144 may be implemented in JavaScript. The media player 120 of FIG. 12 may be configured to receive the script modules of FIGS. 9 and 11.

The receiver 141 receives media data generated by the media service apparatus 110 using a communication protocol supporting a web service. In this case, the communication protocol supporting the web service may be an RTSP / RTP protocol transmitted on the web socket. The receiver 141 includes a web socket client 131 and an RTSP / RTP client module 121.

The websocket client 131 is a module for establishing a websocket connection with the web server 114 of the media service device 110. The media playback device 120 and the media service device 110 exchange transport packets through a handshake between the websocket client 131 and the web server 114, respectively.

As described in the embodiment of FIG. 8, the RTSP / RTP client module 121 performs a function of supporting RTSP / RTP communication to the web browser 210 of the user. Therefore, the user can play the media through the web browser 210 of the HTML5 environment using the RTSP / RTP protocol without installing a separate plug-in.

The media data that has passed through the receiver 141 is separated into video data and audio data by the data separator. Video data is passed from the RTSP / RTP client module 121 to the depacketization modules 122a and 122b, respectively, along the lower left arrow and the audio data along the lower right arrow. The depacketizing modules 122a and 122b depacketize the video data and the audio data, and the depacketized video data is delivered to the container unit 142 and audio data is delivered to the audio sink unit 144.

The container unit 142 includes a container generation module 127. When the video data is not in the container format, the container generation module 127 packages the video frames into unit numbers and converts the chunks into chunk data.

Referring to FIG. 15 to describe the process of generating the chunk data in the container generation module 127. The video frames 311 of FIG. 15 represent video data configured in units of frames, which are obtained by depacketizing in the depacketizing module 122a. In this case, the video frames 311 are data compressed with various video codecs such as MPEG-2, H.264, H.265.

When the depacketization module 122a delivers the video frames 311 to the container generation module 127, the container generation module 127 determines whether the received data is in a container format using header information. Since the video frames 311 are frame format data, the container generation module 127 converts the video frames 311 into a container format in order to be processed in the MSE 134 and the video tag 135.

The conversion from the frame format to the container format is achieved by packaging a plurality of frames in unit numbers. In this case, the number of units may be variably selected according to frames per second (fps) of the video data. 15 illustrates a process of generating chunk data when the number of units is three. The container generation module 127 parses each frame header information (frame size, intra or inter, inter frame type (I, B or P), etc.) of the video frames 311 and parses them. The container header is generated, and the container header and the data of the video frames 311 are packaged and converted into the chunk data 312 in the container format. The chunk data 312 converted into the container format may be processed by the media reconstruction unit 143 including the MSE 134 and the video tag 135 without a compatibility problem.

12, the dotted line arrow is drawn from the container generation module 127 to the audio chunk 128, which means that information for packaging audio data is delivered. The transmitted information will be described in detail through the audio chunk 128.

The media reconstruction unit 143 decodes the chunk data by a decoder embedded in a web browser to reconstruct the video, and output the reconstructed video. In this case, the decoder embedded in the web browser may be a video tag. In the embodiment of FIG. 12, the media reconstruction unit 143 includes an MSE 134 and a video tag 135.

MSE 134 is a JavaScript API for HTML5 that is designed for video streaming playback using HTTP download. Standardized by the W3C, the technology enables streaming playback on console game machines such as Xbox and Playstation 4 (PS4) or Chromecast browsers.

The video tag 135 performs decoding and rendering so that the video is displayed in the web browser. Using the decoder of the video tag 135 allows for better decoding than the JS decoder module 124, which has limitations due to the dynamic language nature of JavaScript. That is, high resolution video and high fps decoding are possible.

To summarize the video output process by the modules described above, the video data is separated from the data separating unit and transferred to the container unit 142, and the container unit 142 selects frames when the video data is not in a container unit. Packaged in unit numbers and converted to chunk data. The video data composed of the chunk data is output by the media reconstruction unit 143 after decoding and rendering. Next, the output process of the audio data will be described.

The audio data separated by the data separator is depacketized by the depacketization module 122b and then output in synchronization with the video data through the audio sinker 144. The audio sink 144 may include an audio decoder 126, an audio chunk 128, a buffer controller 129, and an audio renderer 136.

The audio decoder 126 decodes the separated audio data by a script capable of parsing the web browser. In this case, the script that can be parsed by the web browser may be JavaScript.

The audio chunk unit 128 generates audio chunks by packaging audio data decoded by the audio decoder 126. Since the audio chunk unit 128 should package audio data in a range corresponding to the chunk data generated by the container generation module 127, the audio chunk unit 128 receives information about the chunk data from the container generation module 127. Get provided. A dotted arrow connected to the audio chunk unit 128 in the container generation module 127 means that the information is transmitted.

Reference is made to FIG. 16 to describe the process of generating an audio chunk. The audio data 321 of FIG. 16 shows audio data after being decoded by the audio decoder 126 and decompressed, and before being rendered by the audio renderer 136. For example, the audio data 321 of FIG. It may be a wave file of. Since the audio signal is defined in the form of amplitude over time, it is possible to extract an audio signal in a corresponding range by cutting out an arbitrary section. For example, if the frame rate of the video data is 30 fps and the container generation module 127 has a unit number of frames constituting the chunk data, the container generation module 127 may use the information of 30 fps and the unit number 3. The audio chunk 128 is transmitted. Since the time corresponding to one video frame is 1/30 second and the number of units is 3, the audio chunk unit 128 packages the audio data 321 at 1/10 second corresponding to three video frames. Create chunk 322.

As described above, when audio chunks, which are audio data synchronized with chunk data, and chunk data generated by the container generation module 127 are compared, the chunk data may be packaged in units of video frames or in units of video frames and audio. The audio chunk is a package of audio data according to the number of units. The chunk data is compressed data before being decoded by the decoder 135a and the audio chunk is data decoded and decompressed by the audio decoder 126.

12, the buffer controller 129 receives the audio chunk from the audio chunk 128 and buffers the audio chunk in the audio buffer, and provides the buffered audio chunk to the audio renderer 136.

The buffer controller 129 is a module that makes it possible to output audio data in synchronization with video data through buffering. In detail, when the media reconstructor 143 transmits time information about a time point at which a video is output to the renderer 135b to the buffer controller 129 in chunk data units, the buffer controller 129 buffers the audio chunk. The audio chunk is provided to the audio renderer 136 according to the time point at which the video is output through the time information. A dotted arrow from the renderer 135b of FIG. 12 to the buffer controller 129 means that the time information is transferred from the media restoring unit 143 to the buffer controller 129. In this case, the time information indicates a time point at which the chunk data corresponding to the time information is rendered by the renderer 135b, and a criterion for the time point at which the chunk data is rendered is the start of the chunk data corresponding to the time information by the renderer. It may be the time at which it is rendered.

The audio renderer 136 performs rendering to output the audio chunk received from the buffer controller 129. The audio renderer 136 may be implemented with a Web Audio API supported by a web browser.

To summarize the process of outputting audio through the above-described modules, the audio data is separated from the data separator and transferred to the audio decoder 126. In the audio decoder 126, the audio data is decoded and reconstructed, and the reconstructed audio is packaged in the audio chunk 128 as an audio chunk. The packaging is performed according to the unit number of frames constituting the chunk data. The audio data packaged into the audio chunk is buffered by the buffer controller 129 and then output through the audio renderer 136 in accordance with the time point at which the video is output. In this case, the time point at which the video is output is contained in time information transmitted from the media reconstruction unit 143 to the buffer controller 129.

Since the media reproducing apparatus 120 of FIG. 12 synchronizes the video data and the audio data without transcoding the audio data, there is no fear of damaging the audio data due to transcoding. In addition, according to the media reproducing apparatus 120 of FIG. 12, the synchronization is not a problem even if the video data and the audio data are decoded by different decoders, thereby providing an environment in which the video data and the audio data can be processed separately. That is, even if the audio data of a codec format not supported by the video tag is restored to a separate decoder, synchronization is not a problem, so the dependency on the codec format supported by the video tag is reduced.

13 shows another embodiment of a media playback device 120. In the embodiment of FIG. 12, if audio data is restored and then synchronized with video data, the audio data is decoded and restored after synchronizing with video data. Compared to the embodiment of FIG. 12, in the embodiment of FIG. 13, the position of the audio decoder 126 is changed to the rear end of the buffer controller 129. Since the rest of the configuration is the same as in the embodiment of FIG. 12, overlapping descriptions are omitted and the changed audio sink unit 144 will be described below.

According to the embodiment of FIG. 13, the audio chunk unit 128 generates audio chunks by packaging audio data before decoding and restoring the audio chunks 128. The audio chunk unit 128 receives information about the chunk data from the container generation module 127 in order to package the audio data into a range corresponding to the chunk data generated by the container generation module 127.

The buffer controller 129 synchronizes the audio chunk with the video data through the time information received from the renderer 135b and transmits the audio chunk to the audio decoder 126. In the embodiment of FIG. 13, the audio data synchronized with the chunk data is data before being decoded by the audio decoder 126, which corresponds to the distinguishing point from the embodiment of FIG. 12.

The audio decoder 126 restores the synchronized audio data to the audio renderer 136 and the audio renderer 136 performs rendering to output the restored audio data.

12 and 13 illustrate a case in which audio data is synchronized based on video data. On the contrary, video data may be synchronized based on audio data.

The media data is referred to as the first media type data (audio) and the second media type data (video), and is applied to the embodiments of FIGS. 12 and 13.

The receiver 141 receives the media data and transmits the media data to the data separator. The data separator separates media data into first media type data (audio) and second media type data (video).

When the first media type data (audio) is not in a container unit, the container unit 142 packages the frames constituting the first media type data (audio) by the unit number and converts the chunk data into chunk data.

The media reconstruction unit 143 decodes the chunk data by a decoder embedded in a web browser, restores first media type data (audio), and outputs the first media type data (audio). Information is transmitted to the sink unit (corresponding to the audio sink unit in FIGS. 12 and 13) in chunk data units. In this case, the decoder embedded in the web browser may be a video tag, and since the audio data may be decoded and rendered by the video tag, the media reconstructor 143 may not output the first media type data (audio).

The sink unit outputs the second media type data (video) in synchronization with the restored first media type data (audio) based on the received time information.

As described above, even when the first media type data is audio and the second media type data is video, the synchronized media can be reproduced according to the embodiments of FIGS. 12 and 13.

In this case, the first media type data is audio and the second media type data is video is only a setting for showing various embodiments and does not limit the types of the first media type data and the second media type data.

Next, an embodiment of another structure of the media reproducing apparatus 120 will be described. 14 shows another embodiment of a media playback device 120. The media player 120 of FIG. 14 includes a receiver, a data separator, a container unit, a media decompression unit, and an audio decompression unit. In this case, the receiver, the data separator, the container, and the audio restorer may be implemented in JavaScript. The media player 120 of FIG. 14 may be configured to receive the script modules of FIGS. 9 and 10.

The receiver and the data separator are configured similarly to FIG. The media data generated by the media service apparatus 110 is received through a receiver, and the media separator separates the media data into video data and audio data. The separated video data is depacketized by the depacketization module 122a and transferred to the container unit.

The container unit includes a container generation module 127, and the container generation module 127 packages the frames constituting the video data into unit numbers and converts the chunk data into chunk data. In this case, the unit number of frames may be variably adjusted according to frames per second (fps) of video data.

The media reconstruction unit includes an MSE 134 and a video tag 135, and decodes the chunk data received from the container to reconstruct the video and output the reconstructed video.

If the media restoring unit restores and outputs the video data, the audio restoring unit decodes and restores the audio data separated by the data separating unit, and outputs the restored audio data in synchronization with the video data output from the media restoring unit.

The audio recovery unit may include a transcode unit. The transcode unit includes an audio transcoder 123 as a module for transcoding audio data into another codec format. The audio transcoder 123 may transcode and output audio data in a codec format supported by the video tag when the input audio data is in a format not supported by the media restoring unit, that is, a codec format not supported by the video tag. have.

The transcoded audio is transmitted to the container generation module 127 in the audio reconstruction unit. When the video data received from the depacketization module 122a is not a container unit, the container generation module 127 packages the audio data received from the audio reconstruction unit while packaging the frames constituting the video data in units of units. Can be packaged in number to generate chunk data. The chunk data generated in this way may be transferred to the MSE 134 without compatibility problems.

The process of decoding, rendering and outputting audio data in the audio reconstruction unit may be included in the media reconstruction unit because the container unit generates chunk data from the video data and the audio data. Accordingly, the video data and the audio data can be output by decoding and rendering the chunk data through the MSE 134 and the video tag 135 of the media reconstruction unit.

Using the media reproducing apparatus 120 of FIG. 14, it is possible to decode and render audio data of a codec format that is not supported by a video tag by transcoding and converting the audio data into chunk data. Therefore, the audio data is restored along with the video data, so that the media data can be reproduced without a synchronization problem.

17 is an exemplary diagram for describing a process of generating a script module implemented with JavaScript according to an embodiment of the present invention.

Referring to FIG. 17, a script module implemented in JavaScript may convert a source code written in native C and C ++ native code into a converter such as Emscripten to obtain JavaScript code that can be used in a browser.

When using a converter such as Emscripten, a decoder or a container implemented in JavaScript can be obtained from conventional native code, thereby reducing the codec dependency.

JavaScript code is used instead of plug-ins, so you don't have to worry about web browser makers deprecating. In addition, there is no need to worry about whether to use the ActiveX interface or the NPAPI interface depending on the web browser. In other words, the web browser dependency can be lowered.

The media playback device 120 illustrated in FIG. 1 may be implemented with, for example, the computing device 400 illustrated in FIG. 18. Computing device 400 may be, but is not limited to, mobile handheld devices (smart phone, tablet computer, etc.), laptop or notebook computer, distributed computer system, computing grid or server. Computing device 400 may include processor 401 and memory 403 and storage 408 in communication with each other or with other elements via bus 440. The bus 440 may be connected to the display 432, one or more input devices 433, and one or more output devices 434.

All these elements may be connected directly to the bus 400 or through one or more interfaces or adapters. Bus 440 is connected to a wide variety of subsystems. The bus 440 may include a memory bus, a memory controller, a peripheral bus, a local bus, and a combination thereof.

The memory 403 may be configured of a random access memory (RAM) 404, a read-only component (ROM) 405, and a combination thereof. In addition, a BIOS (or firmware) having basic routines necessary for booting within the computing device 400 may be included in the memory 403.

Storage 408 is used to store operating system 409, executable files (EXEC, 410), data 411, API application 412, and the like. The storage 408 may include a hard disk drive, an optical disk drive, a solid-state memory device (SSD), or the like.

Computing device 400 may include an input device 433. The user may input commands and / or information to the computer device 400 through the input device 433. Examples of the input device 433 include a keyboard, a mouse, a touch pad, a joystick, a game pad, a microphone, an optical scanner, a camera, and the like. The input device 433 may be connected to the bus 440 through an input interface 423 including a serial port, a parallel port, a game port, a USB, and the like.

In certain embodiments, computing device 400 is connected to network 430. Computing device 400 is coupled with other devices through network 430. At this time, the network interface 420 receives communication data in the form of one or more packets from the network 430, and the computing device 400 stores the received communication data for processing by the processor 401. Similarly, the computing device 400 stores the transmitted communication data in the form of one or more packets on the memory 403, and the network interface 420 transmits the communication data to the network 430.

The network interface 420 may include a network interface card, a modem, and the like. Examples of the network 430 include the Internet, a wide area network (WAN), a local area network (LAN), a telephone network, direct connection communication, and the like, and may employ a wired and / or wired communication scheme.

The execution result of the software module by the processor 401 may be displayed on the display 432. Examples of the display 432 include a liquid crystal display (LCD), an organic liquid crystal display (OLED), a cathode ray tube (CRT), a plasma display panel (PDP), and the like. Display 432 is connected to bus 440 via video interface 422, and data transmission between display 432 and bus 440 may be controlled by graphics controller 421.

In addition to the display 432, the computing device 400 may include one or more output devices 434, such as audio speakers, printers, and the like. Output device 434 is coupled to bus 440 via output interface 424. The output interface 424 is, for example, a serial port, a parallel port, a game port, a USB, and the like.

Those skilled in the art will appreciate that the present invention can be embodied in other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

110: media service device 111: camera
112: encoder 113: packetizer
114: web server 115: module storage unit
116: module transmission unit 117: control unit
120: media playback device 121: RTSP / RTP client module
122: depacketization module 123: audio transcoder
124: JS decoder module 125: JS renderer module
126: audio decoder 127: container generation module
128: audio chunk 129: buffer controller
141: receiver 142: container
143: media recovery unit 144: audio sink unit
210: web browser

Claims (20)

A receiving unit which receives the media data generated by the media service apparatus using a communication protocol supporting a web service;
A data separator that separates the received media data into video data and audio data;
A container unit for packaging the frames constituting the video data into unit numbers and converting the frames into chunk data;
Reconstructs the video by decoding the converted chunk data by a decoder embedded in a web browser, and when the output of the reconstructed video is output, time information about a time point at which the reconstructed video is output, in units of the chunk data. A media restoration unit provided; And
An audio sink configured to output the audio data in synchronization with the restored video based on time information provided in the chunk data unit;
The media reconstruction unit includes a renderer that renders before outputting the reconstructed video, wherein the time information indicates a time point at which chunk data corresponding to the time information is rendered in the renderer, thereby synchronizing the video and audio on a web browser. Media playback device to play. The container unit of claim 1, wherein the container unit
And controlling the number of units in accordance with a frame per second (FPS) of the video data. delete The method of claim 1,
And the time information indicates a time point at which a start portion is rendered by the renderer among chunk data corresponding to the time information. The audio sink of claim 1, wherein the audio sync unit
And decoding the separated audio data to restore audio, and outputting the restored audio in synchronization with time information provided in the chunk data unit. The audio sink of claim 5, wherein the audio sync unit
An audio decoder which decodes the separated audio data by a parsable script in the web browser;
A buffer controller for synchronizing the decoded audio data with the chunk data corresponding to the time information and providing the decoded audio data to an audio renderer; And
And the audio renderer for rendering the decoded audio data. The method of claim 6,
And the chunk data is compressed data before being decoded, and the audio data synchronized with the chunk data is decoded and decompressed data. 7. The audio renderer of claim 6 wherein the audio renderer
A media playback device for synchronizing and playing video and audio on a web browser, which is implemented by an audio application program interface (API) supported by the web browser. The audio sink of claim 1, wherein the audio sync unit
And buffering the separated audio data in synchronization with the time information provided in the chunk data unit, and decoding and outputting the buffered audio data to restore and output the audio. The method of claim 9, wherein the audio sink unit
A buffer controller configured to buffer the separated audio data in synchronization with the chunk data corresponding to the time information;
An audio decoder to decode the buffered audio data by a parsable script in the web browser; And
And an audio renderer for rendering the decoded audio data. The method of claim 10,
And the chunk data is compressed data before being decoded, and the audio data synchronized with the chunk data is data before being decoded by the audio decoder. The method of claim 1,
The decoder embedded in the web browser is a video tag supported by HTML5, and the receiving unit, the container unit, and the audio sink unit are implemented in JavaScript. The method of claim 12,
And the JavaScript is downloaded to the media player from the media service device. A media service device for transmitting media data to a media playback device.
A module storage unit for storing a script module necessary for playing the media data on a web browser of the media player;
A module transmitter for transmitting the script module to the media player in response to a connection of the media player;
A packetizer configured to packetize the media data to generate a transport packet; And
A web server for establishing a communication session with the playback device and transmitting the transport packet to the media playback device in response to a request from the media playback device,
The script module may include: receiving the transport packet through the communication session; packaging the video frames included in the transport packet in unit numbers and converting the chunk data into chunk data; and converting the chunk data into the media playback. A process of synchronizing with the chunk data and outputting audio data included in the transport packet based on time information about a time point decoded and output by a media restoring unit installed in the apparatus,
The time information indicates a time point at which chunk data corresponding to the time information is rendered by a renderer included in the media restoring unit. The media service device of claim 14, wherein the script module is code written in JavaScript that is parseable in the web browser. The method of claim 14,
The time information indicates a time point at which a start portion of the chunk data corresponding to the time information is rendered by the renderer in the media playback device. 15. The process of claim 14, wherein the process of synchronizing and outputting the audio data
And restoring audio by decoding the audio data included in the transport packet, and outputting the restored audio by synchronizing with the time information provided in the chunk data unit. The process of claim 14, wherein the process of synchronizing and outputting the audio
Buffering the audio data in synchronization with chunk data corresponding to the time information;
Decoding the buffered audio data by a parsable script in the web browser; And
And process for rendering the decoded audio data. The method of claim 14,
And the chunk data is compressed data before being decoded, and the audio data synchronized with the chunk data is data decoded and decompressed by the media player. A receiving unit which receives the media data generated by the media service apparatus using a communication protocol supporting a web service;
A data separator configured to separate first media type data and second media type data from the received media data;
A container unit for packaging the frames constituting the first media data into unit numbers and converting the frames into chunk data;
Decode the converted chunk data by a decoder embedded in a web browser to restore a first media, and time information regarding a time point at which the restored first media is output when the restored first media is output. A media restoring unit for providing a chunk data unit; And
A sink unit configured to output the second media data in synchronization with the restored first media based on time information provided in the chunk data unit,
The media reconstruction unit includes a renderer that renders before outputting the first media, wherein the time information indicates a time point at which chunk data corresponding to the time information is rendered by the renderer. Media playback device to play.

Images