WO2021132986A1

WO2021132986A1 - Method for media enhancement and bandwidth optimization on communication networks

Info

Publication number: WO2021132986A1
Application number: PCT/KR2020/018582
Authority: WO
Inventors: Williard Joshua Decena JOSE
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2019-12-22
Filing date: 2020-12-17
Publication date: 2021-07-01
Also published as: PH12019050293A1

Abstract

The present disclosure relates to a method for media enhancement and bandwidth optimization on communication networks. The method for a media enhancement on communication networks receives, from a transmitting device, a multimedia content, processes, through a media enhancement function (MEF) in the communication networks, the multimedia content, and transmits, through the MEF device to a receiving device, the processed multimedia content.

Description

METHOD FOR MEDIA ENHANCEMENT AND BANDWIDTH OPTIMIZATION ON COMMUNICATION NETWORKS

The present application relates to wireless communication technologies, and in particular, to a method for media enhancement and bandwidth optimization on communication networks.

Telecommunication networks are evolving rapidly in the last decade. Starting from 2G all the way to 5G there is an increase in the maximum throughput in the telecom networks. These improvements come from various techniques like improving the spectral efficiency, improving the coding, higher carrier frequencies, and improved antennas (e.g. MIMO). There is a prevalence of multimedia, with people naturally communicating through visual and auditory means.

There is a need to improve the Quality of Experience (QoE) of people who send and receive multimedia streams using telecom networks. Possible solutions are classical or machine learning-based compression and DSP methods to improve media quality and reduce bandwidth. However, these techniques are currently implemented on-device, which needs additional configuration and consumes computing power and battery life.

Methods in accordance with various embodiments of the present disclosure aims to deliver high-quality media over medium or low bandwidth link through a communication network without needing additional processing on the transmitting or receiving devices.

The present disclosure relates to a method for media enhancement and bandwidth optimization on communication networks. The method comprises transmitting multimedia content through a transmitting device, receiving and processing the content through a media enhancement function (MEF) device in the communication network, and transmitting the processed content to a receiving device.

In addition, the MEF optimizes the bandwidth by reducing bandwidth requirements while maintaining the same multimedia content quality.

According to the embodiment of the present disclosure, the Quality of Experience (QoE) of subscribing multimedia streams through Communications Networks can be improved by performing enhancement techniques directly on the multimedia.

According to the embodiment of the present disclosure, the size of media traffic through communication networks can be reduced while maintaining or improving the quality of the media stream.

According to the embodiment of the present disclosure, high-quality media can be delivered over medium or a low bandwidth link through the communication networks without needing additional processing on the transmitting or receiving devices with techniques which are transparent to the users.

The accompanying drawings, which are included to provide a further understanding of the disclosure, are incorporated herein to illustrate embodiments of the disclosure. Along with the description, they also serve to explain the principle of the disclosure. In the drawings:

FIG. 1 illustrates a high-level architecture diagram of the media enhancement function (MEF).

FIG. 2 illustrates the general flowchart for the MEF operation.

FIG. 3 illustrates the application of MEF on IP Multimedia Subsystem (IMS) standard as an additional component.

FIG. 4 illustrates the hardware implementation of MEF in an appliance-based deployment.

FIG. 5 illustrates the hardware implementation of MEF in a virtualized or containerized deployment.

FIG. 6 illustrates deployment of MEF in the IMS core.

FIG. 7 illustrates deployment of MEF on mobile edge of both transmitter and receiver.

FIG. 8 illustrates deployment of MEF on mobile edge of receiver only.

FIG. 9 illustrates deployment of MEF on mobile edge of transmitter only.

FIG. 10 illustrates deployment of MEF on 4G/5G/6G telecommunications network.

FIG. 11 illustrates the multimedia traffic detected by the UPF or PGW re-routed automatically to the MEF for processing.

FIG. 12 illustrates the deployment of EMF on IP Television (IPTV) broadcast over the IMS platform.

The following are definitions of terms as used in the various embodiments of the present disclosure.

The term "communications network" as used herein refers to any cellular network standardized by 3GPP and ETSI, including but not limited to 4G, 5G, or 6G. This also includes IMS systems such as Voice over LTE (VoLTE), Video over LTE (VoLTE), and Rich Communication Services (RCS).

The term "multimedia" as used herein refers to digital content including video and music streams, video conference and voice calls, photos/images, and any other similar data or combinations thereof.

The term "media enhancement" as used herein refers to improving the quality of multimedia, including but not limited to increasing resolution and frame rate.

The term "bandwidth optimization" as used herein refers to reducing the bandwidth requirements of multimedia while maintaining the same quality.

The term "feature enhancements" as used herein refers to specialized additional transformations or processing done to multimedia, including but not limited to closed-captioning of video streams.

The present disclosure relates to a method for media enhancement and bandwidth optimization using a media enhancement function (MEF) on communication networks.

FIG. 1 illustrates a high-level diagram of the MEF comprising a multimedia stream coming from a transmitting device, passing through a communications network, before being received by a receiving device according to a preferred embodiment of the present disclosure. The MEF receives multimedia streams and data sent over the communications network.

In a preferred embodiment, the MEF performs one or more of the following three functions: (1) media enhancement; (2) bandwidth optimization; and (3) special feature enhancements to multimedia.

FIG. 2 shows a diagram of the MEF operation of the method. The transmitting device (UE1) finds route in the communication network (COMNET) to a receiving device (UE2) (operation 200). Next, UE1 coordinates with COMNET to establish media stream with UE2 (operation 202). COMNET then activates media enhancement function (MEF) to intercept media stream between UE1 and UE2 (operation 204). UE1 and UE2 start media transmission through COMNET, transparently enhancing the media stream using MEF (operation 206). The method is preferably implemented on the communications network either as a standalone network appliance, especially for legacy networks (e.g. 4G), or as a virtualized network function (VNF) implemented on the service-based telecom network (e.g. 5G and 6G).

In a preferred embodiment, the MEF process the received multimedia for media enhancement, including but not limited to, video and photo upscaling (e.g. super-resolution), video frame rate-up conversion, audio de-noising and artificial bandwidth extension for high fidelity multimedia, brightness and contrast optimization, and HDR effect generation.

In another preferred embodiment, the MEF process the received multimedia for bandwidth optimization, including but not limited to, classical compression of data packets, re-encoding and decoding of media streams with more efficient codecs or machine learning-based coding, and orchestration of transmission with a lower resolution.

In yet another embodiment, the MEF process the received multimedia for feature enhancements, including but not limited to live closed-captioning of video stream, live audio translation service, photo captioning using RNNs, and real-time censorship and parental control of multimedia.

FIG. 3 shows an application of MEF on IP Multimedia Subsystem (IMS) standard as an additional component. The MEF is integrated with the media resource function (MRF). The MRF's primary function in the standard is for media stream mixing, transcoding, and conferencing features, interoperability of VoLTE with circuit-switched calls, as well as dual-tone multi-frequency (DTMF) detection. In a preferred embodiment, the MEF is added as a parallel function to MRF processor, but also controlled by the MRF controller (300). Multimedia streams needing enhancement, bandwidth optimization and/or other feature enhancements are then routed to the MEF.

FIG. 4 shows a hardware implementation of MEF in an appliance-based deployment. The MEF here is implemented as an appliance (or network entity) which has its own set of dedicated resources. For legacy 4G EPC, the MEF is implemented as a network appliance coupled to the Unified Gateway (UGW). As a network appliance (devices), this MEF embodiment has its processors, memory, storage, network interface, and possible acceleration modules. Multiple MEF appliances can be connected with each other through daisy chaining (400), with one of the appliances assigned as the master to orchestrate load-balancing among appliances (402).

The processors of the MEF device may control the overall operation of the MEF device according to an embodiment proposed by the disclosure. For example, the processors may control signal flow to perform the operation according to the drawings and the flowcharts described herein. Specifically, the processors may exchange messages or signals with another network entities (ex. the transmitting device or the receiving device). The processors may perform the operations described above with reference to FIG. 2 based on instructions stored in the memory or storage. The processors of the MEF device may transmit or receive the multimedia data through the network interface to or from another network entity. In the disclosure, the processor may be defined as a circuit, an application-specific integrated circuit, or a controller.

The memory may store a program and various types of control information required by the processors, and may further store respective information described in the disclosure. The memory may store at least one piece of information transmitted and received through the interface and information generated through the processors.

FIG. 5 shows a hardware implementation of MEF in a virtualized or containerized deployment. For 5G stand-alone deployment, the MEFD is implemented as a network function. This can either be implemented as a virtual machine, if the 5G Core Network is implemented with VM Cloud (500), or as a container, if the 5G Core Network is implemented with Container Cloud (502). In both cases, the MEF is connected to the User Plane Function (UPF) through which all data passes through. Eligible multimedia sessions are re-routed to the MEF for enhancement and/or bandwidth optimization. This re-routing is accomplished through service chaining, wherein some of the data from the UPF is chained to another service, in this case, the MEF. Due to the service-based architecture of the 5G Core, the services are virtualized, and forwarding is accomplished through software defined networking (SDN).

FIG. 6 shows the deployment of MEF in the IMS core. The MEF can be implemented as a network function of the IP Multimedia Subsystem and the 5G network. Due to the emergence of multi-access edge computing (MEC), network functions can be deployed in the network core or on the network edge. The network function can be deployed in the core when the connection between the transmitting device and the core network is congested, or when the quality of the media uploaded by the transmitting device is low. As the multimedia passes through the core network, the MEF enhances the media and/or decreases the bandwidth requirement, so that the receiving device gets higher quality media and subsequently less bandwidth requirement (600).

FIG. 7 shows the deployment of MEF on mobile edge of both transmitter and receiver (700). This can be used to optimize the core network traffic used. In this case, the MEF is very close to the end user devices, and thus, traffic in between the two MEF is reduced. Behind the MEF, the media quality is high (702).

FIG. 8 shows the deployment of MEF on mobile edge of receiver only (800). This can be used if the network bandwidth is congested, apart from the edge connection to the receiving device. This mode is useful if the transmitting device is connected to a different network which does not have MEF.

FIG. 9 shows the deployment of MEF on mobile edge of transmitter only. This mode can be used if the multimedia quality from the transmitting device is low quality due to reduced bitrate or low-quality media source. In this mode, the media is enhanced upon transmission to the mobile edge (900). This mode is useful if the receiving device is connected to a different network which does not have MEF (902). This is also useful when the core network does not have available capacity, while the edge network can still provision resources for the MEF.

FIG. 10 shows the deployment of MEF on 4G/5G/6G telecommunications network. In an alternative embodiment, the MEF will be connected logically to the User Plane Function (UPF) for 5G/6G core deployment (1004). This is the equivalent of the PGW (for 4G) or GGSN (for GPRS). The MEF receives the policy from the policy control function (PCF) (1000), and the multimedia session is tracked by the session management function (SMF) (1002). Since this is based on the 5G SA architecture, all signaling traffic is sent with HTTP/2.0.

In another embodiment, the MEF will be connected to the PDN Gateway (PGW) (1006) for 5G NSA deployment and by extension, 4G LTE EPC. The PGW receives the multimedia stream/data and determines whether a specific session needs to be routed to the MEF. A special signaling interface will be connected between the PGW and the MEF. Telecom policies will then be pulled from the PCRF to the PGW who will push these to the MEF.

FIG. 11 shows how multimedia traffic detected by the UPF or PGW are re-routed automatically to the MEF for processing on 4G/5G/6G telecommunication networks. This alternative embodiment detects multimedia flows through the session traffic itself, based on the headers (e.g. RTP) or based on dedicated network bearer information. The multimedia is detected directly by the UPF/PGW (1100), such as the network function or element through which all traffic passes through and re-routed automatically to the MEF for processing.

FIG. 12 shows the deployment of EMF on IP Television (IPTV) broadcast over the IMS platform. In this alternative embodiment, IP Television (IPTV) broadcast TV programs over the IMS platform. In this case, the user (UE) is only receiving multimedia and not transmitting. All the media comes from the media delivery function (MDF) which in turn sources the media from the broadcasting station (1200). The media sent by the MDF is enhanced by the MEF close to the UE, which is possibly co-located in the Communications Network of the UE. For example, if the broadcast is in HD, the MEF can upscale the media to 4K UHD.

According to various embodiment, a method for a media enhancement on communication networks may comprise receiving, from a transmitting device, a multimedia content, processing, through a media enhancement function (MEF) in the communication networks, the multimedia content, and transmitting, through the MEF device to a receiving device, the processed multimedia content.

According to various embodiment, the MEF may be implemented on the communication networks as a standalone network appliance.

According to various embodiment, the MEF may be implemented on the communications network as a virtualized network function (VNF).

According to various embodiment, the processing the multimedia content through the MEF may comprise at least one of upscaling image data of the multimedia content; increasing a frame rate of image data of the multimedia content through up-conversion processing; de-noising audio data of the multimedia content; performing artificial bandwidth extension; optimizing at least one of brightness and contrast of the image data; and adding at least one of additional content or a control data to at least one of audio and image data of the multimedia content.

According to various embodiment, the adding of the at least one of additional content or the control data may comprise at least one of: adding closed-captioning of the image data to the image data; adding translation of the audio data to the audio data; adding captioning data to the image data to the image data; and adding censoring data of the multimedia content.

According to various embodiment, the method may further optimize a bandwidth for transmitting the processed multimedia content.

According to various embodiment, the optimizing of the bandwidth may comprise at least one of: compressing data packets of the processed multimedia content; re-encoding and decoding media streams the processed multimedia content; and orchestrating transmission of the processed multimedia content with a lower resolution.

According to various embodiment, an electronic device for a media enhancement function (MEF) on communication networks, may comprises a memory; a network interface; and a processor electronically connected to the memory and the interface and configured to: receive, from a transmitting device, a multimedia content; process, through a media enhancement function (MEF) in the communication networks, the multimedia content; transmit, through the MEF device to a receiving device, the processed multimedia content.

According to various embodiment, the processor may process the MEF by performing at least one of: upscaling image data of the multimedia content; increasing a frame rate of image data of the multimedia content through up-conversion processing; de-noising audio data of the multimedia content; performing artificial bandwidth extension; optimizing at least one of brightness and contrast of the image data; and adding at least one of additional content or a control data to at least one of audio and image data of the multimedia content.

According to various embodiment, the processor may add the at least one of additional content or the control data by performing at least one of: adding closed-captioning of the image data to the image data; adding translation of the audio data to the audio data; adding captioning data to the image data to the image data; and adding censoring data of the multimedia content.

According to various embodiment, the processor may be further configured to optimize a bandwidth for transmitting the processed multimedia content.

According to various embodiment, the processor may be configured to further optimize the bandwidth by performing at least one of: compressing data packets of the processed multimedia content; re-encoding and decoding media streams the processed multimedia content; and orchestrating transmission of the processed multimedia content with a lower resolution.

According to various embodiment, a network interface for media enhancement function (MEF) may be configured to perform the method as described before through communicating with a network entity including at least one of a gateway, a user plane function , a control plane function, a media resource function, or a media delivery function of communication networks.

It is contemplated for embodiments described herein to extend to individual elements and concepts described herein, independently of other concepts, ideas or system, as well as for embodiments to include combinations of elements recited anywhere in this application. It is to be understood that the disclosure is not limited to the embodiments described in detail herein with reference to the accompanying drawings. As such, many variations and modifications will be apparent to practitioners skilled in this art. Illustrative embodiments such as those depicted refer to a preferred form but is not limited to its constraints and is subject to modification and alternative forms. Accordingly, it is intended that the scope of the disclosure be defined by the following claims and their equivalents.

Moreover, it is contemplated that a feature described either individually or as part of an embodiment may be combined with other individually described features, or parts of other embodiments, even if the other features and embodiments make no mention of the said feature. Hence, the absence of describing combinations should not preclude the inventor from claiming rights to such combinations.

Claims

A method for a media enhancement on communication networks comprising:

receiving, from a transmitting device, a multimedia content;

processing, through a media enhancement function (MEF) in the communication networks, the multimedia content;

transmitting, through the MEF device to a receiving device, the processed multimedia content.
The method of claim 1, wherein the MEF is implemented on the communication networks as a standalone network appliance.
The method of claim 1, wherein the MEF is implemented on the communications network as a virtualized network function (VNF).
The method of claim 1, wherein the processing the multimedia content through the MEF comprises at least one of:

upscaling image data of the multimedia content;

increasing a frame rate of image data of the multimedia content through up-conversion processing;

de-noising audio data of the multimedia content;

performing artificial bandwidth extension;

optimizing at least one of brightness and contrast of the image data; and

adding at least one of additional content or a control data to at least one of audio and image data of the multimedia content.
The method of claim 4, wherein the adding of the at least one of additional content or the control data comprises at least one of:

adding closed-captioning of the image data to the image data;

adding translation of the audio data to the audio data;

adding captioning data to the image data to the image data; and

adding censoring data of the multimedia content.
The method of claim 1, further comprising:

optimizing a bandwidth for transmitting the processed multimedia content.
The method of claim 5, wherein the optimizing of the bandwidth comprises at least one of:

compressing data packets of the processed multimedia content;

re-encoding and decoding media streams the processed multimedia content; and

orchestrating transmission of the processed multimedia content with a lower resolution.
An electronic device for a media enhancement function (MEF) on communication networks, the device comprising:

a memory;

a network interface; and

a processor electronically connected to the memory and the interface and configured to:

receive, from a transmitting device, a multimedia content;

process, through a media enhancement function (MEF) in the communication networks, the multimedia content;

transmit, through the MEF device to a receiving device, the processed multimedia content.
The device of claim 8, wherein the processor processes the MEF by performing at least one of:

upscaling image data of the multimedia content;

increasing a frame rate of image data of the multimedia content through up-conversion processing;

de-noising audio data of the multimedia content;

performing artificial bandwidth extension;

optimizing at least one of brightness and contrast of the image data; and

adding at least one of additional content or a control data to at least one of audio and image data of the multimedia content.
The device of claim 9, wherein the processor adds the at least one of additional content or the control data by performing at least one of:

adding closed-captioning of the image data to the image data;

adding translation of the audio data to the audio data;

adding captioning data to the image data to the image data; and

adding censoring data of the multimedia content.
The device of claim 8, wherein the processor is further configured to optimize a bandwidth for transmitting the processed multimedia content.
The device of claim 11, wherein the processor is configured to further optimize the bandwidth by performing at least one of:

compressing data packets of the processed multimedia content;

re-encoding and decoding media streams the processed multimedia content; and

orchestrating transmission of the processed multimedia content with a lower resolution.
A network interface for media enhancement function (MEF), the MEF is configured to perform the method according to any one of claims 1 to 7 through communicating with a network entity including at least one of a gateway, a user plane function, a control plane function, a media resource function, or a media delivery function of communication networks.