KR101821123B1 - Method and apparatus for playing media stream on web-browser - Google Patents

Abstract

A media stream reproducing apparatus for receiving a media stream transmitted from a media server and reproducing the received media stream on a web browser includes a transmission module for establishing a communication connection at a transmission layer level with the media server, A web socket module for establishing a web socket connection through a handshake and transmitting and receiving a web socket packet to and from the media server while continuing to maintain the established web socket connection; A format converter for converting the media stream into a format reproducible by the browser built-in player if the media stream obtained from the real time transmission protocol packet is not executable by a browser built-in player; Translated US You restoring an image from the air stream is made to the output device to embed the reconstructed image and the plug-in (non-plugin) the built-in browser of the regenerator system, the web browser displayed on the screen.

Description

METHOD AND APPARATUS FOR PLAYING MEDIA STREAMS IN A WEB Browser [0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for playing a media stream on a web browser, and more particularly, to a technique for directly streaming video and audio obtained from a camera in a web browser without a plug-in.

The media content can be accessed by a plurality of user devices connected to various video sources via the Internet. To this end, a web browser is installed in the user equipments so as to be able to communicate with the server application using hypertext transfer protocol (HTTP). However, such a web browser is not suitable for streaming continuous media because it is a half-duplex method and overhead occurs due to repetitive information exchange through HTTP request and response messages.

1, the web browser 15 transmits a connection request message (a long-lived request) to the web server 25, and then the web server 25 transmits a connection request message suspended state and a connection between them is established. In this state, when the web browser 15 transmits a client action for performing a specific action to the web server 25, the web server 25 transmits a response message to the web browser 15 . For example, if the client action is a streaming request for a particular image, the response message will be packet data of the video. Such a connection can not be continuously performed, and the connection between the two is disconnected while the web server 25 transmits a long-lived request completes to the web browser 15. [ If the web browser 15 further requests the streaming, the above steps should be repeated.

That is, in the conventional method of communicating between the web server and the web browser in the HTTP connection method, a connection between the web server and the web browser is required for each specific event, and the connection is terminated when the event ends. Accordingly, although the above method is suitable for handling non-continuous events such as connection of a web page, it is not suitable for a field where continuous connection between a web server and a web browser is required, such as video and voice streaming.

Due to these problems, various plug-ins (ActiveX, NPAPI, PPAPI) are installed in the web browser, and network connection function between web browser and web server, decoding of received image and function of outputting decoded image are implemented Is widely used. Particularly, a network camera is provided with a function of receiving and outputting a video / audio generally called a web viewer. The function of this web viewer is that the plug-in is installed automatically when the user connects to the network camera through inputting the network address of the camera without installing the software such as CMS (Central Monitoring System) or VMS (Video Management System) , So that video and audio can be received through the plug-in thus installed. In order to provide such a function, the network camera mounts a web server capable of providing a web service. Therefore, in the conventional web service method, a plug-in is automatically installed when a user terminal equipped with a web browser accesses a web server through a specific URL, and a function of transmitting and receiving video and audio is performed through a plug-in.

However, the way of installing the plug-in in the web browser is decreasing the frequency of use while exposing many problems such as security vulnerability, limitation of the function of the web browser, and excessive consumption of resources. As another example of the result of standardization for transmitting a video stream, HTML5 supports a function of receiving an image file stored in a remote source through a video tag, and in WebRTC, openWebRTC, MPEG-DASH, etc. Provides a standard for real-time video transmission and reception.

In this way, due to the recent exclusion of plug-in technology, image reception, playback and display functions can be implemented according to a method supported by standards such as HTML5, WebRTC, open WebRTC, and MPEG-DASH. However, in order to process images on a web browser using RTP / RTSP (Real Time Transport Protocol / Real Time Streaming Protocol), which has been widely used as a video transmission / reception standard, a plug-in must be used. Currently, It is not possible to process images according to RTP / RTSP. However, in addition to RTP / RTSP, it may not be a practical option to mount additional heavy protocols in an embedded system that is limited by system resources such as network cameras and storage devices.

Therefore, in the technical fields of network cameras, NVR / DVRs, encoders / decoders, video management software, etc., technology capable of streaming media (video and audio) on a web browser using RTP / RTSP communication without any additional plug- .

Korean Patent Laid-Open Publication No. 2015-0119003

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide an apparatus and method for streaming video and audio on a web browser without delay, without requiring a plug-in.

The technical objects of the present invention are not limited to the technical matters mentioned above, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a media stream reproducing apparatus for receiving a media stream transmitted from a media server and playing the media stream on a web browser, the apparatus comprising: a transmission module for establishing a communication layer- A web socket module that establishes a web socket connection with the media server based on the connection at the transport layer level and transmits and receives a web socket packet to and from the media server while maintaining the established web socket connection; A streaming module for receiving a real-time transport protocol packet carried by the web socket packet; A format converter for converting the media stream obtained from the real time transmission protocol packet into a format reproducible by the browser built-in player, if the media stream is not executable by a browser built-in player; A browser built-in player of a non-plugin type for restoring an image from the converted media stream; And an output device for embedding the restored image into the web browser and displaying the embedded image on the screen.

According to the present invention, it is possible to provide a user device compatible with RTP / RTSP using a web socket without a plug-in and capable of streaming video and audio in real time. In particular, because RTSTP / RTP is controlled via a web socket, it can be easily supported by any browser that supports a web socket, and there is no need to design a new protocol.

Also, according to the present invention, it is possible to stream media on a web browser without a plug-in even in devices that do not have high system specifications such as a network camera, a network video recorder, and a digital video recorder. Particularly, there is an advantage that both the real-time method and the playback method including the search and back channel audio can be supported in the above-mentioned devices and there is no restriction on the media format.

1 is a diagram illustrating a method of communicating between a web server and a web browser in a conventional HTTP connection method.
2 is a diagram illustrating an overall system according to an embodiment of the present invention.
3 is a diagram showing an OSI 7-layer model and a TCP / IP 4-layer model that are hierarchically defined for communication between devices.
4 is a block diagram illustrating a configuration of a media stream playback apparatus according to an embodiment of the present invention.
5 is a sequence diagram showing an example of a process of transmitting and receiving data through a web socket connection.
6 is a specific structural diagram of a web socket packet transmitted and received between the media stream reproducing apparatus and the stream server.
7 is a specific structural diagram of the RTP protocol.
8 is a block diagram showing a detailed configuration of the streaming module of FIG.
9 is a diagram showing the structure of a communication packet communicating with a stream server through a network interface.
10 is a block diagram showing the configuration of a media server that communicates with the media stream playback apparatus of FIG.
11 is a sequence diagram illustrating an RTSP setup process between a streaming module and a media server in a media stream playback apparatus.
FIG. 12 is a sequence diagram illustrating a process in which the streaming module receives RTP data from the media server after the setup process of FIG. 11;
13 is a sequence diagram illustrating a connection termination process when no further connection is required between the streaming module and the media server.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2, the overall system 10 in accordance with the present invention includes a network 50 that enables communication between the various user devices 106-114 and the media servers 202, 204, do. The network 50 may be a direct or indirect physical communication connection, a mobile connection network, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a storage area network (SAN) And the like. Each media server 202, 204, 206 includes a suitable computing or processing device to provide computing services for one or more client user devices or media stream playback devices. For example, the media servers 202, 204, and 206 may generate and store media streams such as network cameras, network video recorders (NVRs), digital video recorders (DVRs) Device. Each of the client user devices 106-114 includes a computing or processing device suitable for interacting with the media server 202, 204, 206 or other computing user devices via the network 50. [ For example, the client user devices 106-114 may be a desktop computer 106, a mobile phone or smartphone 108, a personal digital assistant 110, a laptop computer 112 and a tablet computer 114 . However, it goes without saying that additional client user devices may be further included in the system 10 of FIG. These media servers 202, 204, 206 or client user devices 106-114 may include, for example, one or more controllers for processing instructions, one or more memories for storing instructions and data, Lt; RTI ID = 0.0 > and / or < / RTI >

Here, some client user devices 108-114 communicate with the network 50 indirectly. For example, the client user devices 108-110 communicate with one or more base stations 116 based on a cell network. In addition, the client user devices 112-114 communicate via one or more wireless access points 118, such as an IEEE 802.11 wireless router. The above description is exemplary and each user device may communicate directly or indirectly with the network 50, via suitable intermediate user equipments or networks. In the present invention, the network 50 enables efficient media streaming. In particular, one or more media servers 202, 204, 206 support media streaming to be performed on a web socket. One or more client user devices 106-114 may detect when the media server 202, 204, 206 supports a media stream on a web socket. One or more client user devices 106-114 establish a web socket connection to the media server 202,204, 206 and, when the media server 202,204, 206 supports media streaming on a web socket, An initial request may be sent indicating the object of the selected media and the location in the stream. Each of the client user devices 106-114 sequentially receives segments of the media stream provided by the media server 202, 204,

As shown in FIG. 2, each of the user devices 106-114 and the media media servers 202, 204, and 206 is provided with a network interface and additional modules for exchanging data with other network interfaces . FIG. 3 shows an OSI (Open System Interconnection) 7-layer model and a TCP / IP 4-layer model that are hierarchically defined for communication between devices.

The TCP / IP model is a simpler model than the classical OSI model (seven layers) for describing connection processing in a network, and classifies connection processing into four layers. The four layers are a network interface layer 61, an internet layer 62, a transport layer 63, and an application layer 64. Each layer in the TCP / IP model is associated with the OSI model in terms of similar functions and roles. For example, the network interface layer 61 corresponds to the physical layer 41 and the data link layer 42 in the OSI model. The Internet layer 62 corresponds to the network layer 43 of the OSI model and the transport layer 63 corresponds to the transport layer 44 of the OSI model. The application layer 64 also corresponds to a group comprising the session layer 45, the presentation layer 46 and the application layer 47 of the OSI model. These TCP / IP models are specifically defined in the RFC 1122 document.

In the TCP / IP model, the network interface layer 61 interfaces with the physical connection medium and implements Long-Term Evolution (LTE), 802.11 (WLAN), 802.3 (Ethernet) The Internet layer 62 provides a service for the user equipment to connect the Internet layers within a LAN or WAN. The Internet layer 62 may implement IPv4, IPv6, or other suitable protocols. The transport layer 63 is used to establish an end-to-end connection between user devices. Typical examples of transport protocols are TCP and UDP (user datagram protocol). In addition, the application layer 64 generally implements communication protocols such as HTTP, RTP, and FTP (File Transfer Protocol). Here, HTTP is generally used for stable distribution contents such as VOD, RTP is used for real-time content streaming, and FTP is used for asynchronously transmitting mass storage data. In the present invention, the term "real time" means media reproduction in a media stream reproducing apparatus with minimized time difference between the media server and the media stream reproducing apparatus.

FIG. 4 is a block diagram showing a configuration of a media stream reproducing apparatus 100 according to an embodiment of the present invention. The media stream playback apparatus 100 may be implemented as any one of the user devices in FIG. 2, and receives a media stream transmitted from the media server and plays the media stream on a web browser. The media stream playback apparatus 100 includes a network interface 120, a transmission module 130, a web socket module 135, a streaming module 140, a format converter 150, a browser built-in player 160, a web browser 170 And an output device 180. The output device 180 may be a display device. Here, the web browser 170 may be an application at a level close to an operating system (OS) (not shown) built in the media stream reproducing apparatus. In addition, streaming module 140 and format converter 150 are Java applications implemented by, for example, JavaScript, hosted by the web browser 170 and the operating system, and the browser-embedded player 160 is preferably Is a module having standard output function of video and audio supported by HTML5. Also, the network interface 120, the transmission module 130, and the web socket module 135 belong to a communication module for network communication and correspond to an operating system (OS) kernel in terms of device implementation.

Referring to FIG. 4, the network interface 120 interfaces with a physical connection medium to transmit and receive data between the media streaming apparatus 100 and the media server 200. The network interface 120 corresponds to the network interface 61 in the TCP / IP model of FIG. The physical connection medium used in the network interface 120 may include wireless media such as Long-Term Evolution (LTE), 802.11 (WLAN), 802.15.3 (WPAN), etc. and wireless media such as 802.3 have.

The transmission module 130 performs a transmission control function for transmitting and receiving data between the media stream reproducing apparatus 100 and the media server 200 and corresponds to the transport layer 63 in the TCP / IP model of FIG. Specifically, the transmission module 130 establishes a communication connection of a transport level layer between the media stream reproducing apparatus 100 and the media server 200, and performs transmission error recovery for transmission and reception of stable data, sequential delivery of data packets, and the like do. In particular, in the present invention, the transmission module 130 supports a TCP or UDP protocol for guaranteeing a Web socket connection between the media stream playback apparatus 100 and the media server 200 at the transport layer level. TCP is a protocol for reliably distributing content in a connection-oriented manner, and UDP is a protocol for providing seamless distribution of content with low overhead. In the present invention, a Web socket protocol packet may be used for reliable transmission, but UDP may be used for low overhead transmission.

Meanwhile, the web socket module 135 establishes a web socket connection through the handshake with the media server based on the connection at the transport layer level, and maintains the established web socket connection, Send and receive socket packets. The web socket module 135 may be implemented in the transmission module 130 or may be implemented as an upper layer of the transmission module 130 separately from the transmission module 130.

The web socket is a protocol that improves conventional half-duplex HTTP communication to enable a bidirectional, complete duplex communication channel over a TCP connection. The web socket protocol was standardized as RFC6455 by the Internet Engineering Task Force (IETF) standardization organization. However, the standardized Web socket protocol is intended as a general protocol, lacks extensive functions desired by the user, and allows the protocol to be extended through JavaScript or the like in a web browser to support new functions .

Since the web socket connection is located on top of the existing transport layer (TCP or UDP) connection between the two devices, in order to use the web socket connection, a TCP transport connection must first be established between the two devices. Once the web socket connection is established between the media stream playback apparatus 100 and the media server 200 via, for example, a 3-way handshake process, the web socket communication may be initiated by sending web socket packets .

FIG. 5 shows an example of a process of transmitting and receiving data through a web socket connection. These Web socket connections are made according to the Web Socket protocol, the HTML5 standard. In particular, since the web socket connection continuously supports bi-directional communication, data can be continuously transmitted and received between the web server of the network camera and the web browser of the user terminal without being disconnected.

5, the media stream reproducing apparatus 100 transmits a TCP / IP connection request message (a TCP / IP connection request message) to the stream server 200, and the stream server 200 accepts the TCP / (SYN-ACK) to the media stream playback apparatus 100, a TCP / IP connection is established. A TCP transport connection can be formed by a pair of local TCP sockets and remote TCP sockets, each of which is defined by an identifier, such as at least a port number and an IP address. Of course, instead of these TCP / IP-based connections, you can also establish a UDP / IP-based connection between them.

Thereafter, when a web socket connection is established between the media stream playback apparatus 100 and the stream server 200 through a handshake process, data can be continuously transmitted and received between the media stream playback apparatus 100 and the stream server 200 thereafter. That is, the media stream reproducing apparatus 100 transmits a media streaming request in the form of a transmitted web socket packet (socket.send) to the stream server 200, and the stream server 200 transmits the media streaming request to the media stream reproducing apparatus 100 Sends the media stream in the form of a response web socket packet (socket.onMessage). This process can be done continuously between media streams until transmission is stopped or completed.

FIG. 6 shows a specific structure of a web socket packet transmitted and received between the media stream reproducing apparatus 100 and the stream server 200 as described above. As such, the beginning of the web socket includes metadata information about the data organization contained in the web socket packet. The last part of the web socket packet also contains the actual payload data (application level data). The location of the payload data in the web socket packet depends on the size of the payload data and whether a masking key is used. Web socket packets are basic units of communication data framed in Web socket communication. By default, the Web socket protocol can use a fixed 80 port, but this is not necessarily the case. For example, port 443 may be used for Web socket connections that are tunneled through Transport Level Security (TLS).

In the TCP / IP model of FIG. 3, the Web Socket protocol is located between the application layer 64 and the transport layer 63. That is, the Web socket protocol is located between the TCP or UDP belonging to the transport layer 63 and the RTP / RTSP belonging to the application 62. Referring to FIG. 4, the transmission module 130 of FIG. 4 basically has a structure in which a Web socket is stacked on a conventional transport layer protocol such as TCP or UDP.

The web socket communication improves communication between the web browser and the web server by providing full duplex communication for at least higher applications and reducing overhead while maintaining the connection of the underlying TCP or UDP transmission. First, unlike HTTP, which is half duplex and selectively transmitted and received between a client device and a server device, communication over a web socket is fully duplex and can be transmitted and received at the same time. In addition, when communicating over a web socket, less header information is transmitted per unit message, thereby reducing transmission overhead. In order to poll the second device from the first device, the lower TCP layer connection between the first device and the second device can be maintained without exchanging HTTP request and response messages.

Referring again to FIG. 4, when the TCP / IP (or UDP / IP) and Web socket connection are established in the transmission module 130, a web socket packet is continuously transmitted between the media stream reproducing apparatus 100 and the media server 200 Receiving state. The transmission module 130 receives a packetized media stream in the form of a web socket packet transmitted from the media server 200 and transmits the packet to the streaming module 140 or transmits a command transmitted from the streaming module 140 to a web socket packet And transmits the packets to the media server 200.

The streaming module 140 requests the media server 200 to transmit the media stream in a real time transmission protocol according to a request from the web browser 170 and transmits a real time transmission protocol packet including the media stream from the media server 200 And controls reproduction of the media stream in accordance with a real-time streaming protocol.

For this purpose, the streaming module 140 may use, for example, a known Real Time Transport Protocol (RTP) and a Real Time Steaming Protocol (RTSP). RTP is a standard for transmitting real-time or stored images and is a protocol for transmitting images such as MJPEG, MPEG-4, H.264, and H.265.

Figure 7 shows a specific data structure of the RTP protocol. As shown in the figure, the RTP protocol is divided into a header and a payload. The header includes a sequence number indicating a transmission order of an RTP data packet, a time stamp (Timestamp) An SSRC identifier (Synchronization source identifier) indicating a synchronization source for the mobile station. Since the size of a packet of a media stream based on RTP is limited, the media stream may be divided into a predetermined size (e.g., 1.5 kByte) and then packetized. After the streaming module 140 receives the divided packets, the packets are combined into one video frame (depacketization). In addition, although one voice frame may be divided and transmitted in the same manner, one voice frame may be transmitted per voice frame because the amount of data is smaller than that of the video frame.

The RTSP also sets a network port between the media server 200 (e.g., a network camera) and the media stream playback apparatus 100 to receive real-time or stored images and provides commands related to playback (Play, Pause, Teardown , Setup, Option, Describe, etc.). Of these, Play is a command used to start a media stream, Pause is used to suspend a media stream that has been started, and Teardown is a command used to disassociate / destroy a specific media session. In addition, a setup is used to set media session parameters, an option to acquire optional method functions, and a disc live to acquire media parameters with a specified profile, .

As described above, the RTP protocol is a protocol for packetizing and actually transmitting a media stream, whereas the RTSP protocol is used as a protocol for starting / ending such transmission or controlling reproduction of a media stream already being transmitted.

The streaming module 140 may be specifically configured to include a streaming session module 142, a streaming client 144, a client manager 146 and a depacketization module 148, as shown in FIG. have. Each of the elements 142, 144, 146 and 148 constituting the streaming module 140 is programmed with JavaScripts supported by the HTML5 standard. Therefore, the functions of the streaming module 140 are separately Plugin is not required.

The streaming session module 142 forms a session between the media servers 200 to stably receive the media stream transmitted in the real time transmission protocol. Accordingly, the transmission module 130 can receive a media stream to be transmitted or serve as a port for transmitting a playback control command transmitted from the media stream playback apparatus 100.

The streaming client 144 generates a client module for new streaming control or terminates the generated client module according to a request from the client manager 146. [ It is also possible to receive a connection command with the media server 200 requested from the web browser 170 or receive a media stream playback command from the web browser 170 and transmit the media stream playback command to the media server 200 via the transmission module 130 (Media stream) transmitted from the media server 200 through the transmission module 130. The transmission module 130 receives the response from the media server 200, Specifically, the streaming client 144 receives and processes the RTSP packet from the web socket module 135 in the transmission module 130, while the streaming session module 142 receives the RTP packet received from the web socket module 135 , And passes it to the depacketization module 148 to generate a data frame (video frame or voice frame).

The client manager 146 creates a client module in the streaming client 144 or removes or destroys the generated client module according to a request from the web browser 170. In other words, the streaming client 144 is responsible for the creation and termination of the client module as a basis for the operation of the streaming client 144.

The depacketization module 148 accumulates a media stream delivered from the streaming client 144 in a buffer (not shown) sequentially, and then depacketizes the media stream to form a complete one frame . Of course, this process can be omitted if the media stream is not partitioned. Since video frames that constitute a video stream are difficult to send in a single packet due to their size, they are usually transmitted in a divided manner. However, since voice frames are relatively small in size, they may be transmitted in a single packet.

As described above, the image frame and the voice frame generated by the streaming module 140 are provided to the format converter 150. Format converter 150 converts the format of a media stream (in particular, an image frame) encoded according to various image coding standards such as MJPEG, MPEG-2, H.264, H.265, Format, for example, MPEG-4, webm, ogg / ogv, etc., which are supported by the HTML5 video tag. Of course, if it is confirmed that the media stream is already in a format supported by the in-browser player 160, the format converter 150 skips the conversion process.

The format conversion performed by the format converter 150 may be a conversion through decode encoding (transcode) of the media stream. However, depending on the format of the source media stream, Level conversion. The format converter 150 may be, for example, programmed with JavaScript, which is supported by the HTML5 standard, without the need for a separate plug-in for the function of the format converter 150 at this time.

The media stream converted in format converter 150 or the media stream skipped without being formatted is input to the browser built-in player 160. The in-browser player 160 is a non-plugin media playback module built in the web browser 170. The in-browser player 160 restores video and audio from the media stream and provides the video and audio to the web browser 170. [ For example, the browser-based player 160 can be implemented with a video tag of the HTML5 standard, which allows a user to play video / audio on a web page provided by a web browser with only a simple HTML script. The video tag is a native tag and is independently operated without the aid of a separate external program and is optimized for a cross browser environment.

However, Microsoft's Internet Explorer supports MPEG-4 and webm based on H.264 and H.265, while Firefox and Opera use Ogg's Theora format, which is open source. . However, the HTML5 video tag may also be used to decode the audio of a predetermined format (e.g., AAC, vorbis, mp3, etc.) included in the video frame in synchronization with the video frame It also supports. Thus, the user can reproduce the media stream at the source location only by creating a simple HTML script as illustrated in Table 1 below.

<! DOCTYPE html>
<html>
<body>

<video width = "400"controls>
<source src = "mov_bbb.mp4" type = "video / mp4">
<source src = "mov_bbb.ogg" type = "video / ogg">
Your browser does not support HTML5 video.
</ video>

</ body>
</ html>

The browser built-in player 160 parses the HTML script and provides a video and audio embedded in the web browser 170. [

In the above HTML script, the size of the video is set to "400" and the location of the source media stream is "mov_bbb.mp4" formatted with MPEG-4 (mp4) and "mov_bbb.ogg" formatted with Ogg / Ogv Able to know. The inclusion of the two source locations is intended to enable playback in various web browsers. Also, if the media stream at the source location is in a format that is not supported by HTML5, an on-screen error message ("Your browser does not support HTML5 video.") Is also included. Of course, such HTML script may further include Javascript supported by HTML5, thereby giving more various attributes to the image playback.

The browser built-in player 160 parses the HTML script to check the location of the source media stream (the media stream converted by the format converter 150), and if the source media stream is a format that can be supported in HTML5, And embeds the video and audio in the video memory 170.

The video and audio embedded in the web browser are transmitted to the output device 180 and are output to the web page on the screen as images and sounds that the user can perceive through visual / auditory sense. The web browser 170 may be a browser that does not support a plug-in, as long as it supports HTML5 regardless of the types of Internet Explorer, Chrome, Firfox, Safari, and Edge Browser. The output device 180 may include a display device that can physically represent an image such as an LCD, an LED, and an OLED, and a sound output device such as an audio receiver, an amplifier, and a speaker.

9 is a diagram showing the structure of a communication packet for communicating with the stream server 200 via the network interface 120. As shown in FIG. When the RTP header 94 is added to the RTP payload corresponding to the media stream 95, it becomes an RTP packet. The RTP packet is the same as the Web socket payload, and a Web socket header 93 is added to the RTP packet to become a Web socket packet. The Web socket packet is the same as the TCP payload, and a TCP header 92 is added to the Web socket packet to become a TCP packet. Finally, the TCP packet is the same as the IP payload, and when the IP header 91 is added thereto, a communication packet, that is, an IP packet is finally completed. The processes of completing the IP packet and removing the respective headers are performed in both the media stream reproducing apparatus 100 and the stream server 200.

When the IP packet is received through the network interface 120, the IP header 91 and the TCP header 92 are processed by the transmission module 130 and a TCP payload or web socket packet is transmitted to the web socket module 135 do. The web socket module 135 processes the web socket header 93 from the web socket packet and transmits the generated RTP packet to the streaming module 140. The streaming module 140 processes the RTP header 94 in the RTP packet to finally restore the media stream. Of course, TCP is exemplified as a sub protocol for transmitting a web socket packet, but UDP or HTTP tunneling may be used instead of the TCP. Although the RTP packet is described as a Web socket payload, the RTSP packet may be a Web socket payload.

In the media stream reproducing apparatus 100 shown in FIG. 4, since the communication between the media stream reproducing apparatus 100 and the media server 200 is performed through the HTML5-based web socket protocol, the RTP / RTSP transmission / And a format converter 150 for converting a format of a video / audio are all implemented by JavaScript code supported in HTML5. Accordingly, real-time streaming and playback of media can be implemented in a web browser without separately installing a plug-in such as an ActiveX or NPAPI as in the prior art.

According to an embodiment of the present invention, a web socket program for RTSP control can be expressed as a pseudo code as shown in Tables 2 and 3, for example, as shown in Table 3 below ).

var META = 0,
VIDEO = 1,
AUDIO = 2,
AUDIO_BACKUP = 3,
RTCP_VIDEO = 4;
RTCP_AUDIO = 5;
RTCP_META = 6;

var RtspClient = function () {
// RTSP Client
// RTSP Command Parsing
// OPTIONS, TEARDOWN, GET_PARAMETERS, SET_PARAMETERS, DESCRIBE, SETUP, PLAY, STOP
var CommandConstructor = function (method, requestURL, extHeader) {
switch (method) {
case 'OPTIONS':
// OPTIONS Reaction
case 'TEARDOWN':
// TEARDOWN Reaction
case 'GET_PARAMETER':
// GET_PARAMETER Reaction
case 'SET_PARAMETERS':
// SET_PARAMETERS Reaction
case 'DESCRIBE':
// DESCRIBE Reaction
case 'SETUP':
// SETUP Reaction
case 'PLAY':
// PLAY Reaction
case 'PAUSE':
// PAUSE Reaction
default:
break;
}
isReceive = false;
return sendMessage;
};

var parseDescribeResponse = function (message1) {
// DESCRIBE Operation
// SDP Parsing
return SDPData;
}
var parseRtspResponse = function (message1) {
// RTSP Response Operation
// Error or OK
return RtspResponseData;
}
var formDigestAuthHeader = function (stringMessage, method) {
// Authentication
};
var RtspResponseHandler = function (stringMessage) {
// RTSP Response Handler
// Error Code Handling
};
var RtpDataHandler = function (rtspinterleave, rtpheader, rtpPacketArray) {
// RTP Data Handling
// Depacketization
};
var connectionCbFunc = function (type, message) {
// Connection Callback
};
module.SetMetaCallback = function (metaCallbackFunc) {
// Metadata Callback
};
module.SetH264VideoCallback = function (videoCallbackFunc) {
// H.264 Callback
};
module.SetH265VideoCallback = function (videoCallbackFunc) {
// H.265 Callback
};
module.SetMjpegVideoCallback = function (videoCallbackFunc) {
// MJPEG Callback
};
module.SetG711AudioCallback = function (audioCallbackFunc) {
// G.711 Audio Callback
};
module.SetG726AudioCallback = function (audioCallbackFunc) {
// G.726 Audio Callback
};
module.SetAACAudioCallback = function (audioCallbackFunc) {
// AAC Audio Callback
};
...
return module;
};

10 is a block diagram showing the configuration of the media server 200 that communicates with the media stream playback apparatus 100 of FIG. The media server 200 includes a network interface 220, a web server 230, a media server 240, and a media storage 250, and may further include a media encoder 260 and a capture device 270 .

The capture device 270 converts the input video and audio into an electrical signal (analog signal or digital signal), generates a video signal and a voice signal, and transmits the video signal and the audio signal to the media encoder 260. The media encoder 260 includes at least a video encoder that encodes the input video signal in accordance with a video coding standard such as MJPEG, MPEG-4, H.264, and H.265 as a component corresponding to the format converter 150. Similarly, the media encoder 260 may further include a voice encoder for encoding the input voice signal according to a voice coding standard such as MP3, G.711, G.726, AC3, DTS, FLAC, and AAC.

In this way, the media stream generated by the media encoder 260 is stored in the media storage 250. The media storage 250, regardless of whether it is a volatile or nonvolatile medium, may be any kind of physical medium such as a magnetic storage medium, an optical storage medium, a hard disk drive, a solid state drive (SSD) The media storage 250 may store the media stream generated by the media encoder 260 for a long time or temporarily for the purpose of supporting the streaming of the media server 240. [

The media server 240 converts the media stream provided from the media storage 250 into a form of a media stream that can be transmitted over the network. To this end, the media server 240 supports the RTP / RTSP protocol. For example, the media server 240 generates an RTP packet based on the RTSP packet transmitted from the media stream reproducing apparatus 100, and controls transmission to the media stream reproducing apparatus 100. Such an RTP packet may be configured as shown in FIG.

The web server 230 hosts web content obtained by the media stream reproducing apparatus 100. At this time, any kind of data and services can be provided by the web server 230. Accordingly, the web browser 170 of the media stream reproducing apparatus 100 can access services or contents at least partially provided by the web server 230. In particular, the web server 230 may process the request and connection of the media stream playback apparatus 100 using the web socket protocol.

Specifically, the web server 230 transmits a media stream (RTP packet generated by the media server 240) to the media stream playback apparatus 100 based on a Web socket. To this end, the Web server 230 is connected to a TCP / IP (or a lower layer protocol for sending a Web socket packet (shown in FIG. 7) according to the Web socket protocol, which is a technical standard for bidirectional communication based on HTML5, UDP / IP). Basically, the web server 230 may be of any kind, but it is preferable to use a kind of resource requiring a small capacity and a small resource, such as a lighttpd, in consideration of an environment such as a network camera, which is difficult to provide high-

Meanwhile, a proxy socket 235 may be additionally disposed between the media server 240 and the web server 230. When a web socket connection is established between the web server 230 and the web browser 170 of the media stream playback apparatus 100, the web server 230 delivers the web socket connection to the proxy socket 235. At this time, the proxy socket 235 relays data transmission / reception between the media server 240 and the web server 230 through a predetermined socket irrespective of the connection method. Thus, using this proxy socket 235, the media server 240 can determine whether the connection is a normal UDP, TCP, WS / TCP (TCP based Web socket), WS / UDP (UDP based web socket) , Or regardless of the access port, data can be sent and received via the specified socket module. By default, the proxy socket (235) can be based on a TCP socket because the web socket is HTTP based on a TCP connection. To this end, the proxy socket 235 encodes the communication transmitted by the media server 200 into a web socket packet to be transmitted to the media stream reproducing apparatus 100, And decodes the packet into a data format desired by the media server 200.

The network interface 220 corresponds to the media stream reproducing apparatus 100 and the network interface 120 and is responsible for interfacing with the physical connection medium to transmit and receive data to and from the media stream reproducing apparatus 100. The network interface 220 corresponds to the network interface 61 in the TCP / IP model of FIG. The physical connection medium used in the network interface 220 includes wireless media such as Long-Term Evolution (LTE), 802.11 (WLAN), 802.15.3 (WPAN), and the like and wireless media such as 802.3 (Ethernet).

2, 8, and 10 may be implemented as software, such as a task, a class, a subroutine, a process, an object, an execution thread, a gate array, an application-specific integrated circuit (ASIC), or a combination of the above software and hardware. The components may be included in a computer-readable storage medium, or a part of the components may be distributed to a plurality of computers.

In addition, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It is also possible that in some alternative implementations the functions mentioned in the blocks occur out of order. For example, it is possible that the two blocks shown in succession may actually be performed substantially concurrently, and that the blocks are sometimes performed in reverse order according to the corresponding function.

11 to 13 are sequence diagrams illustrating RTSP control and RTP transmission with the media server 200 in the streaming module 140 and the web socket module 135 according to an embodiment of the present invention. The streaming client 144 is responsible for RTSP client instance creation and RTSP profile information, and the authentication information is also managed by the streaming client 144. The web socket module 135 is responsible for functions related to the web socket, and the web socket is used to send the RTSP command by the streaming client 144 and receive the response. In the streaming client 144, the order of the connection method is as follows: 1) Initialization of the transmission module 2) Registration of RTSP response and RTP response callbacks 3) Description (information on functions, attributes, Transmission of the RTP session, and transmission of the setup command. The arrows indicated by the solid lines in Figs. 11 to 13 indicate specific commands, and the arrows indicated by dashed lines in Fig. 11 to 13 indicate the response to the command or Ack.

FIG. 11 shows an RTSP setup process between the streaming module 140 and the media server 200 in FIG. First, when the web browser 170 requests the client manager to create an RTSP client (S1: Create RtspClient), the client manager 146 causes the streaming client 144 to create a new RTSP client (S2: new RtspClient ). After the new RTSP client is created as described above, the web browser 170 sets an RTSP URL to the streaming client 144 and performs authentication (S5: Set RtspURL, Authentication). Thus, the web browser becomes a basic state in which data can be exchanged with the media server 200.

When the web browser 170 requests the streaming client 144 to connect to the media server (S7: Connect), the streaming client 144 requests a new transmission to the web socket module 135 (S9: new Transport) , And a callback is set (S11: Set callback). When the streaming client 144 sends an RTSP command to the web socket module 135 (S13: Send RtspCommand), the web socket module 135 transmits a description command to the media server (S14: Describe command). Thereafter, when the streaming client 144 transmits a subsequent RTSP command to the web socket module 135 (S18: Send RtspCommand), the web socket module 135 transmits a setup command to the media server (S19: Setup command). Accordingly, when a setup response is received from the media server 200 (S21: setup response) to the web socket module 135, the web socket module 135 transmits an RTSP response to the streaming client 144 (S22: RtspResponse ).

Thereafter, the streaming client 144 transmits a command for generating an RTSP session to the streaming session module 142 (S24: Create RtspSession), sets a callback (S26: Set Callback) Upon completion (S28: OnConnectedCallback), the RTSP setup process between the streaming module 140 and the media server 200 is completed. Thereafter, RTP data between the streaming module 140 and the media server 200 can be transmitted / received over a web socket.

FIG. 12 shows a process in which the streaming module 140 receives RTP data from the media server 200 after the setup process of FIG. First, when the web browser 170 sends a media access command to the streaming client 144 (S30: OpenMedia), the streaming client 144 sends an RTSP command to the web socket module 135 (S32: SendRtspCommand). Then, the web socket module 135 transmits a media playback command to the media server 200 (S33: Play Command). Thereafter, when the streaming client 144 receives the RTSP response from the web socket module 135 (S35: RtspRespond), it notifies the web browser 170 that the media open state is enabled (S37: OnOpenMedia).

On the other hand, the media server 200 performs RTP data transmission to the web socket module 135 according to the reproduction command of S33 (S30: RTP Data-> OnReceive). At this time, the web socket module 135 delivers the RTP data to the streaming session module 142 (S40: SendRtpData), and the streaming session module 142 notifies the web browser 170 of the media stream Frame) to the web browser 170 (S41: OnFrameRecv). 12, it is possible to control (play, pause, etc.) the RTP data as the streaming client 144 transmits the RTSP command to the media server 200 even during the streaming of the specific RTP data.

When the streaming of RTP data as shown in FIG. 12 is completed and further connection between the streaming module 140 and the media server 200 is unnecessary, the connection termination process as shown in FIG. 13 can be performed. The streaming client 144 transmits an RTSP command to the web socket module 135 (S46: SendRtspCommand (RTSP)) when the web browser 170 transmits a command to end the RTSP to the streaming client 144 (S44: CloseRtsp) ). Accordingly, the web socket module 135 transmits a disassembly command to the media server 200 (S47: SendTeardown) and receives the Ack from the media server 200 (S48).

When the web socket module 135 sends an Rtsp response message to the streaming client 144 (S49: RtspResponse), the streaming client 144 transmits a command to end the open of the media streaming to the web browser 170 (S50 : Onclose). When the web browser 170 sends a command to remove the already generated Rtsp client to the client manager 146 (S53: RemoveRtspClient), the client manager 146 instructs the streaming client 144 to remove the Rtsp client ) Command (S54: destroy), all the connections between the streaming module 140 and the media server 200 are terminated.

As described above, according to the present invention, it is possible to stream media in real time from a media server 200 such as a network camera without installing a plug-in to the web browser 170. [ However, it is also possible to use a hybrid method in which media is streamed using a dedicated web viewer method through installation of a plug-in according to a user's selection, or a method of streaming media without a plug-in. According to this embodiment, when the user accesses the media server 200 from the media stream playback apparatus 100, the media server 200 transmits the user environment information (web browser version, resolution of the user apparatus Etc.), and can check whether the user environment is an environment in which the plug-in can be installed. If the user environment is an environment in which a plug-in can be installed, the web server 230 may present a web page for confirming whether or not a plug-in is installed in the web browser 170, and receive a user's selection. When the user selects to install the plug-in, the user downloads the web viewer from the web server 230, installs the web viewer on the web browser 170, and uses the web viewer installed in the conventional manner. On the other hand, if the user does not want to install the plug-in, the user can play the video / audio while controlling the media streaming according to the media stream playback method according to the present invention without the plug-in.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the embodiments described above are in all respects illustrative and not restrictive.

100: media stream reproducing apparatus 120, 220: network interface
130: Transmission module 135: Web socket module
140: Streaming module 142: Streaming session module
144: Streaming Client 146: Client Manager
148: depacketization module 150: format converter
160: Browser built-in player 170: Web browser
180: output device 200: media server
230: Web server 235: Proxy socket
240: Media Server 250: Media Storage
260: Media Encoder 270: Capture Device

Claims (10)

A media stream reproducing apparatus for receiving a media stream transmitted from a media server and reproducing the media stream on a web browser,
A transmission module for establishing a communication layer-level communication connection with the media server;
A web socket module that establishes a web socket connection with the media server based on the connection at the transport layer level and transmits and receives a web socket packet to the media server while maintaining the established web socket connection;
A streaming module for receiving a real-time transmission protocol packet carried by the web socket packet, and for transmitting / receiving a real-time streaming protocol packet for controlling transmission of the real-time transmission protocol packet to / from the media server;
A format converter for converting the media stream obtained from the real time transmission protocol packet into a format reproducible by a browser built-in player; And
And a non-plugin-type browser built-in player for restoring an image from the converted media stream for playback in the web browser,
Wherein the streaming module performs a setup process with the media server using the real-time streaming protocol packet before receiving the real-time transmission protocol packet after the handshake,
Wherein the streaming module performs a setup process with the media server using the real-time streaming protocol packet before receiving the real-time transmission protocol packet after the handshake,
A streaming client requesting the media server to transmit the real-time streaming protocol packet through the web socket module or requesting to terminate the transmission of the real-time streaming protocol packet at the request of the web browser;
A streaming session module for receiving the real-time transmission protocol packet through the web socket module and delivering the received real-time transmission protocol packet to the format converter;
And a client manager for requesting the streaming client to create or terminate a client module for controlling streaming of the real time transmission protocol in the streaming client at the request of the web browser,
Wherein the streaming client, the streaming session module, and the client manager are implemented as scripts that can be parsed by the web browser. 2. The method of claim 1,
And a description step of acquiring media parameters of the media stream reproducing apparatus with a specified profile. The method according to claim 1,
Wherein the web browser supports the HTML5 standard and the browser built-in player is a video tag of the HTML5 standard. The method of claim 3,
Wherein the format converter skips the format conversion of the media stream if the media stream is executable by the browser built-in player. The method of claim 3,
Wherein the streaming module and the format converter are implemented by JavaScript without a plug-in program installed separately in the web browser. The method of claim 3,
Wherein the browser built-in player parses an HTML document to determine a source location of the converted media stream. The method according to claim 1,
The in-browser player restores the voice from the converted media stream,
And an output device for synchronizing and outputting the restored voice with the restored image. The method according to claim 1,
Wherein the web socket packet comprises a web socket header, a subsequent real-time transport protocol header adjacent to the web socket header, and a subsequent media stream adjacent to the real-time transport header. The method according to claim 1,
Wherein the media server is a network camera that captures an image in real time and provides the media stream to the media stream player. delete

Images