KR102064284B1 - Device, system, and method to perform real-time communication - Google Patents

Abstract

According to an embodiment of the present invention, a method for performing real-time communication with an external web browser may include: initiating real-time communication by signaling with a web browser through a network; Packetizing based on the packet, depacketizing the packetized media data to obtain encoded media data, and performing real time communication by transmitting the encoded media data to a web browser via a network. .

Description

DEVICE, SYSTEM, AND METHOD TO PERFORM REAL-TIME COMMUNICATION}

The present invention relates to apparatus and methods for providing encoded multimedia over a network, and more particularly to providing improved encoding efficiency when communicating encoded media data in real time, for example by web real time communication. A source device, a network system, and a method are provided.

Communication service providers and enterprises can use real-time communications based on a protocol known as Web Real-Time Communication (WebRTC). Web real-time communication is an open Internet standard for implementing real-time multimedia communication functions (e.g. voice call, video chat, end-to-end file sharing, etc.) in a web browser, and enables real-time communication between source and sink browsers. Make it possible. In a device that supports a web browser, web real-time communication may use application programming interfaces (APIs) to enable the device to have real-time communication capabilities without the user having to download plug-in software. Web real-time communication establishes a connection between two web browsers and creates a channel for exchanging data between corresponding peers.

When performing web real time communication, the sink device may have sufficient hardware resources to process media data received from the source device. For example, a sink device having a high performance processor such as a portable computer, a personal digital assistant, and a smart phone may have sufficient hardware even if the sink device decodes the media data through software. With resources, media data can be played smoothly. On the other hand, the source device may have hardware resources that are not sufficient to process the media data. For example, if the source device encodes media data via software, a low performance processor may not be able to encode media data to be transmitted smoothly or efficiently. In this case, the overall operation and / or performance of the source device and the web real-time communication may be degraded, thereby reducing the quality and efficiency required for the real time communication.

The above description is only to help understand the background art of the technical idea of the present invention, and thus it cannot be understood as the content corresponding to the prior art known to those skilled in the art.

Source devices and network systems according to an embodiment of the present invention are to improve the efficiency and quality of real-time communications such as web real-time communications. Furthermore, devices and systems according to an embodiment of the present invention are to enable end-to-end web real time communication without the user of the sink device downloading plug-in software.

The efficiency and quality of real time communication can be achieved through source devices with improved encoding efficiency. For example, the web real-time communicator of the source device receives the encoded media via a streaming server.

A method for performing real time communication with an external web browser according to an embodiment of the present invention, the method comprising: signaling the web browser through a network to initiate the real time communication; After the signaling is performed, packetizing the encoded media data based on a predetermined real-time network protocol; Depacketizing the packetized media data to obtain the encoded media data; And performing the real time communication by transmitting the encoded media data to the web browser via the network.

Depacketizing the packetized media data may include obtaining the encoded media data from a payload of the packetized media data.

The method may further include determining the predetermined real time network protocol by reading setting information stored in internal storage, wherein the payload of the packetized media data may be identified according to the determined real time network protocol.

The performing of the real time communication may include packetizing the encoded media data again; And transmitting the repackaged media data to the web browser.

The method may further comprise generating the encoded media data by encoding the media source using a hardware media encoder.

The real time communication may include web real time communication.

The predetermined real time network protocol may be one of a real-time transport protocol (RTP), a real-time streaming protocol (RTSP), and a real-time messaging protocol (RTMP).

The encoded media data may be based on one of video compression standards H.263, H.264, H.265, VP8, VP9, and AV1 (AOMedia Video 1).

According to another embodiment of the present invention, a method for performing web real-time communication with an external web browser includes: initiating the web real-time communication by signaling with the web browser; Requesting packetized media data from a streaming server according to a result of the signaling, wherein the packetized media data includes media data encoded based on a predetermined media compression standard; And performing the web real time communication by transmitting the encoded media data obtained from the packetized media data to the web browser.

The method further comprises receiving real time network protocol information corresponding to the streaming server from a source device associated with the streaming server, wherein the encoded media data is obtained from a payload of the packetized media data, and The payload of packetized media data may be identified based on the received real time network protocol information.

The method further includes reading configuration information stored in internal storage to determine a real-time network protocol corresponding to the streaming server, wherein the encoded media data is obtained from a payload of the packetized media data and the packet The payload of the normalized media data may be identified based on the determined real time network protocol.

The packetized media data may be requested when the web browser supports the predetermined media compression standard as a result of the signaling.

The predetermined media compression standard may be one of image compression standards H.263, H.264, and H.265.

The method may further comprise receiving a media source when the web browser supports a media compression standard different from the predetermined media compression standard as a result of the signaling.

The method includes encoding the media source based on the different media compression standard; And performing the web real time communication by transmitting an encoded media source based on the different media compression standard.

Another aspect of the present invention relates to a source device that performs web real-time communication with an external web browser. A source apparatus according to an embodiment of the present invention includes a media encoder configured to process a media source to generate encoded media data; A streaming server configured to support packetization of the encoded media data based on a predetermined real time network protocol; And a web real time communicator configured to signal the web browser to initiate the web real time communication, and to request the packetized media data from the streaming server according to a result of the signaling. Configured to perform the web real-time communication using normalized media data.

The media encoder generates the encoded media data based on a first media compression standard, wherein the web real-time communicator is configured to send the packet to the streaming server when the web browser supports the first media compression standard as a result of the signaling. Request mediated media data.

The media encoder generates the encoded media data based on a first media compression standard, wherein the web real-time communicator supports a second media compression standard different from the first media compression standard as a result of the signaling. Receive the media source and perform the web real-time communication.

The web real time communicator may be configured to encode the received media source based on the second media compression standard and to perform the web real time communication using a media source encoded based on the second media compression standard. have.

The web real time communicator may be configured to extract the encoded media data from the payload of the packetized media data, and transmit the encoded media data to the web browser to perform the web real time communication.

According to another embodiment of the present invention, a source apparatus for performing web real-time communication with an external web browser may include a media encoder configured to process a media source to generate encoded media data; And at least one processor, the at least one processor to perform signaling with the web browser; According to a result of the signaling, packetize the encoded media data based on a predetermined real time network protocol; Depacketize the packetized media data to obtain the encoded media data for the web real-time communication.

Source devices and network systems according to an embodiment of the present invention improve the efficiency and quality of real-time communications, such as web real-time communications. Furthermore, devices and systems according to an embodiment of the present invention enable end-to-end web real-time communication without the user of the sink device downloading plug-in software.

1 is a block diagram illustrating a network system for executing web real-time communication between a source device and a sink device according to an embodiment of the present invention.
2 is a block diagram illustrating a source device according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating a method of operating a source apparatus according to an exemplary embodiment of the present disclosure.
4 is a flowchart illustrating a method of operating a web real-time communication device according to an exemplary embodiment of the present invention.
5 is a flowchart illustrating an embodiment of signaling for end-to-end web real-time communication.
6 is a flowchart illustrating an embodiment of a preparation operation for receiving and processing packetized media data.
7 is a block diagram illustrating an embodiment of a computer system for implementing embodiments of a source device.
8 is a block diagram illustrating a source apparatus according to another exemplary embodiment.
9 is a flowchart illustrating a method of operating a web real-time communication device according to another embodiment of the present invention.
10 is a block diagram illustrating a source device according to another exemplary embodiment.
11 is a flowchart illustrating a method of operating a web real-time communication device according to another embodiment of the present invention.
12 is a block diagram illustrating a network system including a source system.
13 is a block diagram illustrating a source system according to an exemplary embodiment of the present invention.
14 is a block diagram illustrating a modified embodiment of the source system of FIG. 13.

In the following description, for purposes of explanation, numerous specific details are set forth in order to assist in understanding the various embodiments. It may be evident, however, that various embodiments may be practiced without these specific details or in one or more equivalent ways. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the various embodiments.

If a device is described as "connected" to another device, this includes not only the direct connection, but also the indirect connection between other devices in between. If it is described as "directly connected" to this other part, it means that there is no other element between that part and the other part. "At least one of X, Y, and Z" and "at least one selected from the group consisting of X, Y, and Z" is X one, Y one, Z one, or two of X, Y, and Z or Any combination of the above (eg, XYZ, XYY, YZ, ZZ) can be interpreted. Here, "and / or" includes all combinations of one or more of the configurations.

Herein, terms such as first, second, etc. may be used to describe various elements, configurations, regions, layers, and / or sections, but such elements, configurations, regions, layers, And / or sections are not limited to these terms. These terms are used to distinguish one device, configuration, region, layer, and / or section from another device, configuration, region, layer, and / or section. Thus, the first element, configuration, region, layer, and / or section in one embodiment may be referred to as the second element, configuration, region, layer, and / or section in another embodiment.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Throughout the specification, when a portion "contains" a certain component, it means that it can further include other components, except to exclude other components unless specifically stated otherwise. Unless otherwise specified, terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

1 is a block diagram illustrating a network system 100 for executing web real time communication between a source device 110 and a sink device 120 according to an embodiment of the present invention. The network system 100 may include a plurality of devices, servers, and / or software components that operate to perform the methods according to the embodiments described herein. The devices and servers shown in FIG. 1 may be deployed in other ways, and the operations and services provided by such devices and / or servers may be combined or separated to perform the embodiments described herein. And may be performed by more or fewer devices and servers. One or more devices and / or servers may be run by the same or other entities, for example companies.

Referring to FIG. 1, the network system 100 includes a network 105, a source device 110, a sink device 120, and a signaling server 130. Each of source device 110 and sink device 120 includes instructions such as program code and / or data stored on computer-readable media by including one or more processors, memories, and other suitable components. And the various applications and steps described herein. For example, the instructions may be stored on one or more computers readable media inside and / or outside various components of the network system 100.

Source device 110 and sink device 120 are connected to network 105. Network 105 includes at least one of a public network, one or more private networks, a wired network, a wireless network, other suitable types of networks, combinations thereof, and such a network. These may be accessible by source device 110, sink device 120, and signaling server 130. Source device 110 and sink device 120 may include any of wired and wireless communication functions, and may communicate with each other via network 105.

The source device 110 and the sink device 120 perform end-to-end communication by real time communication such as web real time communication. The source device 110 transmits media data including at least one of audio data and video data to the sink device 120 participating in a specific media share session. In this case, the media data may include multimedia data, but is not limited thereto. The media data may be displayed on the display of the sink device 120. That is, the sink device 120 can render the received media data to its display and voice equipment.

For example, source device 110 may include at least one of an Internet Protocol (IP) camera, a network video recorder, a web camera, and the like, which sink media data through network 105. Transmit to device 120. For example, sink device 120 includes a portable computer, a PDA, a smartphone, a tablet PC, a video game device, a television, a desktop computer, a monitor, a projector, and the like, which receive received media data in its display and voice. Can be played on the device.

The source device 110 includes a web real-time communicator 115 that performs web real-time communication with a web browser 125 of the sink device 120. The web real time communicator 115 may initiate web real time communication by signaling with the web browser 125 through the signaling server 130. For example, the web real time communicator 115 may define a media sharing session for end-to-end connection of web real time communication through signaling with the web browser 125. As described later with reference to FIG. 5, during signaling, the signaling server 130 relays the messages from the source device 110 to the sink device 120, and sources the messages from the sink device 120. Relay to device 110 may be performed. During signaling, web real-time communicator 115 and web browser 125 may support media related capabilities such as supportable media compression standards, bit rates for encoding and decoding media data, resolution of media data, and the like. media capabilities can be exchanged. Further, during signaling, the web real time communicator 115 and the web browser 125 may identify candidates for identifying the web real time communicator 115 and the web browser 125 within the network 105 such as IP addresses and ports. candidates can be exchanged.

After the signaling is successfully performed, the web real-time communicator 115 captures the media data generated by the internal components of the source device 110, and in real time the web browser 125 of the sink device 120. Web real-time communication can be performed. The web browser 125 may render the received media data on the display of the sink device 120 and the voice device.

2 is a block diagram illustrating a source device 110 according to an embodiment of the present invention.

Referring to FIG. 2, the source device 110 may include a memory 111, a media encoder 112, a streaming server 113, and a web real-time communicator 115. It includes.

The memory 111 may be in the form of a memory device or system known or developed in the art, and may store a media source including any one of an image source and an audio source. The source device 110 may further include at least one camera for providing image data and at least one microphone for providing audio data. For example, the camera may include an image sensor that converts optical signals input through the lens into electrical signals to generate image data. An image sensor is a semiconductor device, such as a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like, which is known or developed in this field capable of converting optical signals into electrical signals. Can be. For example, the microphone may convert voice into electrical signals to generate voice data. The generated image data and audio data may be processed by one or more processors of the source device 110 to provide an image source and an audio source.

Media encoder 112 encodes a media source and generates encoded media data ECMD based on certain media compression standards. In embodiments, the media encoder 112 may encode the media source based on one of the video compression standards H.263, H.264, H.265, VP8, VP9, AOMedia Video 1 (AV1), and the like. For example, the media encoder 112 may be integrated into a System on Chip together with the memory 111 and encode the media source stored in the memory 111 to generate encoded media data ECMD. can do.

The streaming server 113 receives the encoded media data ECMD and packetizes the encoded media data ECMD in response to a request of the web real-time communicator 115. The streaming server 113 may include a media capture driver (MCD) 113a. The media capture driver 113a can request the media encoder 112 to encode the media source and capture the encoded media data ECMD. The streaming server 113 may packetize the encoded media data (ECMD) based on a real-time network protocol such as Real-Time Transport Protocol (RTP), Real-Time Streaming Protocol (RTSP), Real-Time Messaging Protocol (RTMP), or the like. Can be. Therefore, the packetized media data PKMD may have a specific format corresponding to a real time network protocol. Accordingly, the packetized media data PKMD may include not only the encoded media data ECMD but also a header corresponding to the real time network protocol.

The streaming server 113 can communicate with streaming client servers, such as cloud servers, connected to the network 105 of FIG. 1. As described above, the streaming server 113 may packetize the encoded media data ECMD, transmit the packetized media data PKMD to the streaming client server, and upload the corresponding video to the streaming client server.

Source device 110 further includes a web real-time communicator 115. The web real time communicator 115 provides a web real time communication function that enables the source device 110 to perform end-to-end communication with the web browser 125 of the sink device 120. The web browser 125 of the sink device 120 also provides a web real time communication function that enables the sink device 120 to perform end-to-end communication with the web real time communicator 115 without downloading and installing plug-in software. do.

According to an embodiment of the present invention, the web real time communication unit 115 receives packetized media data PKMD from the streaming server 113 and performs web real time communication using the packetized media data PKMD. . As described with reference to FIG. 1, the web real time communicator 115 may perform web real time communication after signaling with the web browser 125 is successfully performed.

The web real time communicator 115 includes a streaming client (STRC) 115a. The streaming client 115a may request the packetized media data PKMD from the streaming server 113. In response to the request, the streaming server 113 may use the media capture driver 113a to capture the encoded media data ECMD and generate the packetized media data PKMD. As such, the streaming client 115a may receive the packetized media data PKMD from the streaming server 113.

The web real-time communicator 115 may depacketize the packetized media data PKMD to obtain encoded media data ECMD. In embodiments, the web real time communicator 115 may read configuration information, for example configuration files stored in memory 111 or other internal storage of source device 110, and may use a packet using any real time network protocol. It may be determined whether normalized media data PKMD is generated. Based on the configuration information, the web real time communicator 115 may parse (or identify) a payload of the packetized media data PKMD. The web real-time communicator 115 may drop the header of the packetized media data PKMD and obtain encoded media data ECMD from the payload of the packetized media data PKMD.

The web real time communicator 115 transmits the encoded media data ECMD to the web browser 125 of the sink device 120 to perform web real time communication. The web browser 125 may decode the encoded media data (ECMD), for example, based on the media compression standard determined in the signaling, and render the decoded media data to the display and voice equipment of the sink device 120.

Media encoder 112 may be a hardware encoder. For example, the memory 111 and the media encoder 112 may be included in the hardware layer, and the streaming server 113 and the web real time communicator 115 may be included in the software layer. However, embodiments of the source device 110 are not limited thereto. For example, media encoder 112 may be implemented via software, firmware, or a combination thereof. In this case, the corresponding program codes (or software) are loaded into the memory 111 or another internal memory of the source device 110, and one or more processors optimized for encoding execute the instructions of the program codes to execute the media encoder. (112) may be implemented.

The web real time communicator 115 may be implemented through software executed by one or more processors (see 1200 in FIG. 7). Assuming the web real time communicator 115 encodes the media source directly for web real time communication, relatively many hardware resources are required to encode the media source. For example, if the source device 110 includes a processor having a relatively low driving capability and the processor executes program codes (or software) to implement the web real time communication device 115, the hardware resources of the processor. Many of these can be used to encode media sources. In this case, web real-time communication may not be performed smoothly or efficiently, and thus overall operating performances of the source device 110 may be degraded. For example, the source device 110 may be one of an IP camera, a network video recorder, a web camera, and the like including a processor having a relatively low operating performance.

The media encoder 112, on the other hand, may be implemented through software executed by one or more processors that are hardware encoders or optimized for encoding to provide improved encoding performance. For example, media encoder 112 may be integrated within a system on chip to provide functions for encoding media sources, but such system on chip may be mounted on source device 110. According to an embodiment of the present invention, the web real-time communicator 115 receives encoded media data ECMD from the media encoder 112 through the streaming server 113. Accordingly, the web real-time communicator 115 encodes the encoded media data ECMD on the assumption that the encoded media data ECMD is packetized for efficient delivery without performing the functions necessary to encode the media. Can be obtained. Thus, even if the web real time communicator 115 is implemented through software and the processor executing the software has a relatively low operating performance (or relatively less hardware resources), the web real time communicator 115 may obtain Encoded media data (ECMD) can be used to efficiently perform web real-time communication. Thus, source device 110 may have improved encoding efficiency.

The media encoder 112 may be implemented in various ways and mounted on the source device 110. For example, as described above, media encoder 112 may be implemented in software that is executed by one or more processors, or any one of a variety of hardware encoders. With this in mind, if the web real-time communicator 115 receives encoded media data (ECMD) directly from the media encoder 112 and not from the streaming server 113, the web real-time communicator 115 may use various types of media. Appropriately designed to be compatible with encoder 112, it should be possible to request and receive encoded media data ECMD directly from media encoder 112 (an example of such a system is shown in FIG. 10).

Since the source device 110 includes a streaming server 113 that can upload encoded media data ECMD from the media encoder 112 to a streaming client server, such as cloud servers, the streaming server 113 It may be used to obtain encoded media data (ECMD) to be used in web real time communication. According to an embodiment of the present invention, the web real-time communicator 115 requests the packetized media data PKMD to the streaming server 113, processes the packetized media data PKDM, and encodes the encoded media data ECMD. ). Considering that streaming server 113 packetizes encoded media data (ECMD) based on one of predetermined RTP, RTSP, and RTMP real-time network protocols, web real-time communicator 115 may select some real-time network. Only protocols may be designed in consideration, thereby improving the compatibility of the web real time communicator 115, which may mean that the web real time communicator 115 may be employed in various types of source devices. Furthermore, due to the improved compatibility of the web real time communicator 115, the development cost of the web real time communicator 115 and the source device 110 including the same may be reduced.

3 is a flowchart illustrating a method of operating a source device 110 according to an embodiment of the present invention.

2 and 3, in step S110, signaling for web real-time communication with the web browser 125 of the sink device 120 is performed. During signaling, the web real time communicator 115 negotiates with the web browser 125 of the sink device 120 a media compression standard to be applied to the media data to be sent to the web browser 125. The web real-time communicator 115 may negotiate with the web browser 125 such that a media compression standard supported by the media encoder 112 is selected.

In step S120, in the media encoder 112, the media source is encoded based on a predetermined media compression standard. The predetermined media compression standard may then be a media compression standard negotiated during signaling.

In step S130, in the streaming server 113, media data (ECMD) encoded based on the real-time network protocol is packetized. The streaming server 113 captures the encoded media data (ECMD) generated by the media encoder 112 and encodes the media based on the real time network protocol of one of the real time network protocols such as RTP, RTSP, and RTMP. Data ECMD can be packetized.

In operation S140, the packetized media data PKMD is depacketized to obtain encoded media data ECMD. The web real-time communicator 115 may perform depacketization (or packet fragmentation) based on a real-time network protocol corresponding to the streaming server 113. The web real time communicator 115 may obtain encoded media data ECMD from the payload of the packetized media data PKMD.

In operation S150, web real-time communication is performed with the web browser 125 of the sink device 120 using the encoded media data ECMD. The web real time communicator 115 may repacket the encoded media data ECMD in a specific format of the web real time communication and transmit the packetized media data to the web browser 125 of the sink device 120 again.

The web browser 125 of the sink device 120 may depacketize the packetized media data to obtain encoded media data ECMD. The web browser 125 will decode the encoded media data ECMD to play the media.

4 is a flowchart illustrating a method of operating a web real-time communication device 115 according to an embodiment of the present invention. 5 is a flowchart illustrating an embodiment of signaling for end-to-end web real-time communication. 6 is a flowchart illustrating an embodiment of a preparation operation for receiving and processing packetized media data.

2 and 4, in step S210, signaling for web real-time communication with the web browser 125 of the sink device 120 is performed. In signaling, the web real-time communicator 115 may exchange messages based on the session description protocol with the web browser 125 of the sink device 120. The signaling may be performed after both the web real time communicator 115 and the web browser 125 access the signaling server 130.

The signaling establishes a media sharing session for end-to-end communication of web real time communication. Referring to FIG. 5, the signaling may include steps S311 to S318. In operation S311, the web browser 125 may transmit an offer based on the session descriptor protocol to the signaling server 130, where the offer may include media related capabilities of the web browser 125, for example, a web browser ( 125) informs the media compression standards that can be supported and the resolution of media data. In operation S312, the signaling server 130 may relay the offer from the web browser 125 to the web real time communicator 115. In operation S313, the web real time communicator 115 may transmit an answer based on the session descriptor protocol to the signaling server 130, where the answer is media of the web real time communicator 115 and / or the source device 110. Relevant capabilities such as the web real-time communicator 115 and / or the source device 110 inform the media compression standard and the resolution of the media data and the like. The media compression standard shown in this answer may include a media compression standard corresponding to the media encoder 112 mounted in the source device 110. In operation S314, the signaling server 130 may relay the answer to the web browser 125. That is, through steps S311 to S314, the web real-time communicator 115 and the web browser 125 may negotiate media related capabilities including a media compression standard.

In operation S315, the web browser 125 may display candidates for identifying the web browser 125 in the network 105, for example, an IP address and a port for performing web real-time communication. Can be sent to. In operation S316, the signaling server 130 may relay the candidates of the web browser 125 to the web real time communicator 115. In operation S317, the web real time communicator 115 may also transmit candidates for identifying the web real time communicator 115 in the network 105. In operation S318, the signaling server 130 may relay the candidates of the web real time communicator 115 to the web browser 125. In embodiments, each of the web real-time communicator 115 and the web browser 125 may determine candidates using Session Traversal Utilities for NAT (STUN) servers. For example, STUN servers may return candidates in response to requests from web real time communicator 115 and web browser 125. Through steps S315 to S318, each of the web real-time communicator 115 and the web browser 125 may specify an IP address and a port on which web real-time communication is to be performed.

In the embodiment described above, the web browser 125 sends the offer and then the web real-time communicator 115 sends an answer in response to the offer. However, embodiments of the present invention are not limited thereto, and the order of steps S311 to S318 may be variously changed. For example, the web real-time communicator 115 may first send an offer to begin signaling, and the web browser 125 may send an answer in response to the offer.

In step S320, after the signaling is successfully performed, the web real time communication unit 115 and the web browser 125 may perform end-to-end communication based on the web real time communication.

Referring back to FIG. 4, in step S220, the streaming server 113 requests the packetized media data PKMD. Accordingly, the web real time communicator 115 receives the packetized media data PKMD including the encoded media data ECMD from the streaming server.

In operation S230, the packetized media data is depacked to obtain encoded media data ECMD. The web real time communicator 115 may parse the payload of the packetized media data PKMD based on the real time network protocol of the streaming server 113 and extract the encoded media data ECMD from the payload.

Before performing step S220 or step S230, referring to FIG. 6, the web real time communicator 115 may determine the real time network protocol of the streaming server 113 (step S410). Based on the determined real-time network protocol, the web real-time communicator 115 may identify the payload of the packetized media data (PKMD). In embodiments, the web real-time communicator 115 may transmit configuration information, such as a configuration file, to any kind of internal storage (see 1300 and 1600 in FIG. 7) or source device 110 in the source device 110. Read from any kind of external storage associated with the, and can determine the real-time network protocol of the streaming server 113 based on the read configuration information. In another embodiment, the web real-time communicator 115 may sequentially call streaming servers conforming to multiple real-time network protocols, depending on whether a response is received or not, according to the real-time network of the streaming server 113. The protocol can be determined.

Referring back to FIG. 4, in step S240, web real time communication is performed using encoded media data ECMD. The web real time communicator 115 may transmit the encoded media data ECMD to the web browser 125 of the sink device 120.

The source device 110 including the web real-time communicator 115 according to an embodiment of the present invention may have improved encoding efficiency by using the media data (ECMD) encoded by the media encoder 112 for web real-time communication. . Furthermore, by providing the encoded media data ECMD via the streaming server 113, the compatibility of the web real time communicator 115 may be improved.

7 is a block diagram illustrating an embodiment of a computer system 1000 for implementing embodiments of the source device 110.

Referring to FIG. 7, the computer system 1000 may include a bus 1100, a processor 1200, a processor, a system memory 1300, a semiconductor chip 1400, a media recorder 1500, and a storage medium. (1600, Storage Medium), and a network interface (1700, Network Interface).

The bus 1100 or other communication mechanism connects various components of the computer system 1000 for the communication of data, signals, and information. The processor 1200 may be at least one of a general purpose processor and a dedicated processor, and may control various operations of the computer system 1000. The system memory 1300 may store information and instructions to be executed by the processor 1200. The system memory 1300 may be any combination of at least one of random access memory (RAM), read only memory (ROM), and various types of computer readable media.

The semiconductor chip 1400 may be connected to the bus 1100 to provide encoded media data (see ECMD of FIG. 2). The semiconductor chip 1400 may include a media encoder 1410 that encodes a media source. The media source and / or encoded media data may be stored in the internal memory 1420 of the semiconductor chip 1400 or the system memory 1300 of the computer system 1000. The internal memory 1420 and / or the system memory 1300 may be provided as the memory 111 shown in FIG. 2.

In embodiments, media encoder 1410 may be a hardware encoder. In another embodiment, the semiconductor chip 1400 may include one or more processors that execute program codes corresponding to the media encoder 1410. In such a case, internal memory 1420 may be provided as working memory of such processors.

The system memory 1300 may store program codes that provide specific functions when executed by the processor 1200. Such program codes process the packetized media data (see PKMD in FIG. 2) to generate encoded media data (see ECMD in FIG. 2), and the encoded media data to an external web browser (see 120 in FIG. 2). Web real-time communication module 1310 (WebRTC Module) for providing the Web real-time communication, and other functions described herein to transmit to. That is, the web real time communication module 1310 may provide the functions of the web real time communicator 1150 illustrated in FIG. 2 when executed by the processor 1200. The program codes stored in the system memory 1300 may further include a streaming module 1320 for providing the functions of the streaming server 1320 illustrated in FIG. 2 when executed by the processor 1200. In addition, the program codes stored in the system memory 1300 may further include one or more additional functional modules for providing other functions. For example, the program codes stored in the system memory 1300 may further include an operating system module that provides an operating system for the computer system 1000.

The media recorder 1500 records images and / or audio to provide image data and / or audio data. Image data and / or audio data may be processed by the processor 1200 and provided to a media source. Although FIG. 7 shows that the media recorder 1500 is included in the computer system 1000, embodiments of the present invention are not limited thereto. For example, the media recorder 1500 can be separate from the computer system 1000 and can communicate with the computer system 1000 via a network. In this case, the computer system 1000 may receive image data and / or audio data from the separated media recorder 1500 through the network interface 1700.

The storage medium 1600 may be at least one of various media accessible by the processor 1200. The storage medium 1600 may include nonvolatile storage such as a memory card. For example, the storage medium 1600 may include a Compact Flash (CF) card, a Smart Media (SM) card, a memory stick, a Multi Media Card (MMC), and a Secure Digital: SD) card, Universal Flash Storage (UFS), and the like. In addition, the storage medium 1600 may include removable or non-removable storage.

Network interface 1700 (or transceiver) transmits and receives signals between computer system 1000 and other devices over a network. The network may be a single network or a combination of multiple networks. For example, the network may be a public network, one or more private networks, a wired network, a wireless network, and / or a computer system 1000 and other accessible by various components of the network system 100 shown in FIG. 1. It may include appropriate types of networks.

In embodiments, the computer system 1000 processes a user action, for example, an input for selecting keys on a keypad or an input for selecting a button and an image, and transmitting a corresponding signal to the bus 1100. It may further include an input / output component.

8 is a block diagram illustrating a source device 210 according to another exemplary embodiment.

Referring to FIG. 8, the source device 210 may include a memory 211, a media encoder 212, a streaming server 213, and a web real-time communicator 215. It includes. The memory 211, the media encoder 212, and the streaming server 213 may be configured similarly to the memory 111, the media encoder 112, and the streaming server 113 described with reference to FIG. 2. Hereinafter, for the sake of brevity, the description of these components is omitted.

When the web real-time communicator 215 performs signaling with the web browser 125 of the sink device 120, a media compression standard that does not correspond to the media encoder 212 may be selected. For example, if the web browser 125 of the sink device 120 does not support the media compression standard supported by the media encoder 212, the media compression standard supported by the media encoder 212 is the web browser 125. You can't choose to negotiate with them. In such a case, another media compression standard may be selected that does not correspond to media encoder 212.

According to an embodiment of the present invention, the web real time communication unit 215 requests the packetized media data PKMD from the streaming server 213 according to the signaling result.

When the media compression standard corresponding to the media encoder 212 is selected in the signaling, the streaming client 215a of the web real-time communicator 215 may request the packetized media data PKMD from the streaming server 213. In response to the request, the streaming server 213 captures the encoded media data ECMD provided by the media encoder 212 using the media capture driver 213a and generates the packetized media data PKMD. do. The web real time communication unit 215 may receive the packetized media data PKMD through the streaming client 215a and perform web real time communication using the packetized media data PKMD.

When another media compression standard is selected that does not correspond to the media encoder 212 in the signaling, the web real-time communicator 215 may receive the media source MS directly from the memory 211. The web real time communicator 215 may also include a media capture driver 215b. The media capture driver 215b of the web real time communicator 215 may capture the media source MS from the memory 211. The web real time communicator 215 may encode the captured media source MS based on the media compression standard selected in the signaling. The web real time communicator 215 may perform web real time communication using the encoded media data.

9 is a flowchart illustrating a method of operating a web real-time communicator 215 according to another embodiment of the present invention.

8 and 9, in step S510, signaling for web real-time communication with the web browser 125 of the sink device 120 is performed. The media compression standard to be applied to the media data in the signaling is selected.

In step S520, it is determined whether the media compression standard selected in the signaling corresponds to the media encoder 212. If so, steps S530 and S540 are performed. When the media compression standard selected in the signaling corresponds to the media encoder 212, the media source may be encoded by the media encoder 212 having a relatively high processing speed. In embodiments, the media compression standard corresponding to media encoder 212 may be at least one of image compression standards H.263, H.264, and H.265. As such, steps S530 and S540 may be performed when the media compression standard selected in the signaling corresponds to the media encoder 212. Steps S530 and S540 are described similarly to steps S220 and S230 of FIG. 4, and thus redundant descriptions are omitted.

In operation S550, the media source is captured when the media compression standard selected in the signaling does not correspond to the media encoder 212. In operation S560, the captured media source is encoded based on the selected media compression standard. For example, the media compression standard encoded by the web real time communicator 215 may be at least one of the video compression standards VP8, VP9, and AV1.

In operation S570, web real time communication is performed using the encoded media data. The web real time communicator 215 may transmit the encoded media data to the web browser 125 of the sink device 120.

According to an embodiment of the present invention, when the media compression standard selected in the signaling does not correspond to the media encoder 212, the web real time communicator 215 may directly encode a media source for web real time communication. Accordingly, a source device having a relatively high compatibility with various web browsers of the sink devices can be provided.

10 is a block diagram illustrating a source device 310 according to another exemplary embodiment.

Referring to FIG. 10, the source device 310 may include a memory 311, a media encoder 312, a streaming server 313, and a web real-time communicator 315. It includes. The memory 311, the media encoder 312, and the streaming server 313 may be configured similarly to the memory 111, the media encoder 112, and the streaming server 113 described with reference to FIG. 2. Hereinafter, for the sake of brevity, the description of these components is omitted.

Except for further receiving the encoded media data ECMD from the media encoder 312, the web real time communicator 315 is configured like the web real time communicator 215 described with reference to FIG. 8. The web real time communicator 315 may be designed to be accessible to at least one of various types of media encoders. The media capture driver 315b of the web real-time communicator 315 may capture encoded media data ECMD from the media encoder 312 when the media encoder 312 is accessible.

11 is a flowchart illustrating a method of operating a web real-time communication device 315 according to another embodiment of the present invention.

10 and 11, in step S610, signaling for web real-time communication with the web browser 125 of the sink device 120 is performed.

In step S620, it is determined whether the media compression standard selected in the signaling corresponds to the media encoder 312. If so, step S630 is performed. In step S630, it is determined whether the streaming server 313 exists. For example, as described with reference to FIG. 6, the web real time communicator 315 reads configuration information or sequentially calls streaming servers according to various real time network protocols to determine whether the streaming server 313 is present. Determine. Steps S640 and S650 are performed when the streaming server 313 is present. Steps S640 and S650 are described similarly to steps S220 and S230 of FIG. 4, and thus redundant descriptions are omitted.

When the selected media compression standard does not correspond to the media encoder 312, steps S660 and S670 are performed. Steps S660 and S670 are described in the same manner as steps S550 and S560 of FIG. 9, and overlapping descriptions are omitted.

In step S680, it is determined whether the media encoder 312 is accessible. In step S690, when the media encoder 312 is accessible, the encoded media data ECMD is captured from the media encoder 312. More specifically, the web real time communicator 315 may be designed to be accessible to at least one of the various types of media encoders. In addition, when the streaming server 313 does not exist but the media encoder 312 in the source device 310 is included in at least one of the various types of media encoders, the web real-time communicator 315 may generate the encoded media data (ECMD). ) Can be captured from the media encoder 312. For example, the web real time communicator 315 may include a media encoder that is accessible to the source device 310 by reading configuration information from internal storage of the source device 310 or from external storages associated with the source device 310. You can determine whether or not. Alternatively, the web real time communicator 315 may sequentially call one or more types of media encoders to determine whether a media encoder accessible to the source device 310 is included.

In operation S700, web real-time communication is performed using the encoded media data ECMD.

According to an embodiment of the present disclosure, the web real-time communicator 315 may capture encoded media data ECMD from the media encoder 312 when the media encoder 312 is accessible. Accordingly, even if the source device 310 does not include the streaming server 312, the source device 310 including the web real-time communicator 315 may have improved encoding efficiency, and thus various source devices The compatibility of the web real time communicator 315 with respect to is improved.

12 is a block diagram illustrating a network system 400 including a source system 410.

Referring to FIG. 12, the source system 410 may be connected to the sink device 420 and the signaling server 430 through the network 405.

The source system 410 is a web real-time communication device 415 that communicates with one or more source devices SD1 to SDm providing media sources and the source devices SD1 to SDm. time communication device). The web real time communication device 415 provides a web real time communication function to the first to m th source devices SD1 to SDm. The web real-time communication device 415 performs signaling with the web browser 425 of the sink device 420 through the signaling server 430 (signaling server).

The web real time communication device 415 may be connected to the first to m th source devices SD1 to SDm through, for example, a private network behind the routers. The web real time communication device 415 may request packetized media data from each of the first to m th source devices SD1 to SDm after performing signaling. In response to such a request, each of the first to m th source devices SD1 to SDm may provide packetized media data including the encoded media data to the web real time communication device 415.

13 is a block diagram illustrating a source system 410 according to an embodiment of the present invention.

Referring to FIG. 13, each of the first to m th source devices SD1 to SDm may include a memory, a media encoder, and a streaming server. The first source device SD1 includes a first memory 511, a first media encoder 512, and a first streaming server 513. The second source device SD2 includes a second memory 521, a second media encoder 522, and a second streaming server 523. The m-th source device (SDm) includes an m-th memory (5m1, m-th Memory), an m-th media encoder (5m2, m-th Media Encoder), and an m-th streaming server (5m3, m-th Streaming Server) do.

The web real-time communication device 415 may communicate with the first to m th streaming servers 513 to 5 m3 through the first to m th channels CH1 to CHm, respectively. The web real time communication device 415 and the source devices SD1 to SDm may include network ports for connecting to the first to m th channels CH1 to CHm.

After performing the signaling, the web real time communication device 415 may request the packetized media data from the streaming server of the corresponding source device. The media encoder of the source device that receives the request encodes a media source in memory and generates encoded media data. The streaming server packetizes the encoded media data and sends the packetized media data to the streaming client 415a of the web real time communication device 415. FIG. 13 illustrates that the first to mth media encoders 512 to 5m2 provide the first to mth encoded media data ECMD1 to ECMDm, respectively, and the first to mth streaming servers 513 to 5m3. Each of the first to m th packetized media data PKMD1 to PKMDm is illustrated as being transmitted to the web real-time communication device 415. The web real time communication device 415 processes the received packetized media data to generate encoded media data, and transmits the encoded media data to the web browser 425 of the sink device 420 to perform web real time communication. can do.

In embodiments, the web real-time communication device 415 may include the first to m th source devices SD1 to SDm from the first to m th source devices SD1 to SDm before receiving the first to m th packetized media data PKMD1 to PKMDm. Information of real-time network protocols of the m-th streaming servers 513 to 5m3 may be received.

FIG. 14 is a block diagram illustrating a modified example 600 of the source system 410 of FIG. 13.

Referring to FIG. 14, each of the first to m th source devices SD1 ˜ SDm ′ may include a memory and a media encoder. The first source device SD1 ′ includes a first memory 611 and a first media encoder 612, and the second source device SD2 ′ includes a second memory 621. Memory) and a second media encoder 622, and the m-th source device SDm 'includes the m-th memory 6m1 and m-th memory and the m-th media encoder 6m2 and m-th Media. Encoder).

The streaming server 713 may be integrated into the web real time communication device 700. The streaming server 713 of the web real-time communication device 700 may communicate with the first to m th source devices SD1 'to SDm' through the first to m th channels CH1 to CHm, respectively. The streaming server 713 receives the first to m th encoded media data ECMD1 to ECMDm through the first to m th channels CH1 to CHm, respectively, and the first to m th packetized media data. PKMD1 to PKMDm are provided to a web real-time communicator 715 of the web real-time communication device 700.

In embodiments, the source devices SD1 to SDm, SD1 'to SDm', and the web real time communication devices 415 and 700 illustrated in FIGS. 12 and 13 are the computer system 1000 described with reference to FIG. 7. It can be implemented through. In this case, the source devices SD1 to SDm and SD1 ′ to SDm ′ may optionally include a streaming module 1320 according to the embodiments shown in FIGS. 12 and 13. The web real time communication apparatuses 415 and 700 do not include the semiconductor chip 1400, and may optionally include a streaming module 1320 according to the embodiments illustrated in FIGS. 12 and 13. In embodiments, the functions of the web real time communication devices 415, 700 may be provided by a hardware device.

Although specific embodiments and application examples have been described herein, these are provided only to aid a more general understanding of the present invention, and the present invention is not limited to the above embodiments and has ordinary skill in the art to which the present invention pertains. Many modifications and variations are possible in this disclosure from the disclosure.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and all the things that are equivalent to or equivalent to the claims as well as the following claims will belong to the scope of the present invention. .

100: network system
105: network
110: source device
111: memory
112: media encoder
113: streaming server
115: web real time communicator
120: sink device
125: web browser

Claims (21)

In a method of operating a source device that performs real time communication with an external web browser:
The source device comprises a media encoder configured to perform encoding based on a predetermined media compression standard,
The method,
Signaling with the web browser via a network to initiate the real time communication;
Generating encoded media data by encoding a media source using the media encoder when the predetermined media compression standard is selected for the real time communication as a result of the signaling with the web browser;
Packetizing the encoded media data based on a predetermined real time network protocol;
Depacketizing the packetized media data to obtain the encoded media data; And
Performing the real time communication by transmitting the encoded media data to the web browser via the network. The method of claim 1,
Depacketizing the packetized media data is obtaining the encoded media data from a payload of the packetized media data. The method of claim 2,
The method may further include determining the predetermined real-time network protocol by reading setting information stored in internal storage.
The payload of the packetized media data is identified according to the determined real-time network protocol. The method of claim 1,
The performing of the real time communication,
Repacketing the encoded media data; And
Sending the repackaged media data to the web browser. The method of claim 1,
The media encoder is configured in hardware. The method of claim 1,
The real time communication comprises web real-time communication. The method of claim 1,
The predetermined real time network protocol is one of a Real-Time Transport Protocol (RTP), a Real-Time Streaming Protocol (RTSP), and a Real-Time Messaging Protocol (RTMP). The method of claim 1,
The encoded media data is based on one of video compression standards H.263, H.264, H.265, VP8, VP9, and AV1 (AOMedia Video 1). In order to perform web real time communication with an external web browser:
Signaling with the web browser to initiate the web real time communication;
Requesting packetized media data from a streaming server according to a result of the signaling, wherein the packetized media data includes media data encoded based on a predetermined media compression standard; And
Performing the web real time communication by transmitting the encoded media data obtained from the packetized media data to the web browser,
The packetized media data is requested to the streaming server when the predetermined media compression standard is selected for the web real time communication as a result of the signaling with the web browser. The method of claim 9,
Receiving real-time network protocol information corresponding to the streaming server from a source device associated with the streaming server,
The encoded media data is obtained from a payload of the packetized media data,
The payload of the packetized media data is identified based on the received real-time network protocol information. The method of claim 9,
The method may further include determining a real-time network protocol corresponding to the streaming server by reading setting information stored in an internal storage.
The encoded media data is obtained from a payload of the packetized media data,
The payload of the packetized media data is identified based on the determined real-time network protocol. delete The method of claim 9,
And the predetermined media compression standard is one of image compression standards H.263, H.264, and H.265. The method of claim 9,
Receiving a media source when a media compression standard different from the predetermined media compression standard is selected for the web real time communication as a result of the signaling with the web browser. The method of claim 14,
Encoding the media source based on the different media compression standard; And
Performing the web real time communication by transmitting an encoded media source based on the different media compression standard. In a source device that performs web real-time communication with an external web browser:
A media encoder configured to process the media source based on the first media compression standard to generate encoded media data;
A streaming server configured to support packetization of the encoded media data based on a predetermined real time network protocol; And
Initiating the web real time communication by signaling with the web browser and sending the packetized media data to the streaming server when the first media compression standard is selected for the web real time communication as a result of the signaling with the web browser. Include a web real-time communicator configured to request,
And the web real time communicator is configured to perform the web real time communication using the packetized media data. delete The method of claim 16,
And the web real time communicator is configured to receive the media source and perform the web real time communication when a second media compression standard different from the first media compression standard is selected for the web real time communication as a result of the signaling. The method of claim 18,
The web real time communicator,
Encode the received media source based on the second media compression standard,
And perform the web real time communication using a media source encoded based on the second media compression standard. The method of claim 16,
The web real time communicator,
Extract the encoded media data from the payload of the packetized media data,
And transmit the encoded media data to the web browser to perform the web real time communication. In a source device that performs web real-time communication with an external web browser:
A media encoder configured to perform encoding based on a predetermined media compression standard; And
Include at least one processor,
The at least one processor,
Perform signaling with the web browser;
Generate encoded media data by encoding a media source using the media encoder when the predetermined media compression standard is selected for the web real time communication as a result of the signaling with the web browser;
Packetize the encoded media data based on a predetermined real-time network protocol;
And depacketize the packetized media data to obtain the encoded media data for the web real time communication.

Images