KR20210111420A - System and Method for Media Processing Based on Deep Learning - Google Patents

Abstract

Disclosed are a system and method for processing a media based on deep learning. In providing a media service, a server, in the present embodiment, transmits identification information and a parameter of a deep learning model trained in advance to be optimized for specific content to a terminal together with the specific content. By applying the specific content to the deep learning model, the terminal can generate the content having an optimized media quality.

Description

Deep Learning-based media processing system and method {System and Method for Media Processing Based on Deep Learning}

The present invention relates to a deep learning-based media processing system and method. More specifically, it relates to a media processing system and method capable of improving the quality of content provided in a video on demand (VoD) or sound source streaming service based on deep learning.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

Deep learning-based technologies are being actively applied to traditional image processing fields such as Super Resolution (SR) and de-noising. (Convolutional Neural Network)-Unet-based method for effectively reconstructing the high-pitched range of a sound source having a low bitrate has been proposed (see Non-Patent Document 1). There are various examples showing that lost information of a media signal, that is, an image or a sound source, can be effectively restored by a process such as compression and/or transmission using a processing based on deep learning technology.

The deep learning-based media processing technique is generally performed in H/E (Head End, or storage server) for reasons of complexity, and then sends the improved media signal down to the terminal. . However, this method may not be effective in terms of network infrastructure operation because it causes a surge in traffic capacity for streaming.

On the other hand, the deep learning-based processing technique is a training step of training a deep neural network (DNN) using a database (DB) for learning and inputting a derived parameter, that is, a network coefficient, to the media content. ) to obtain the desired result by applying the reasoning step. Deep learning-based processing aims to apply well to all content in the inference stage by deriving parameters using a wide range of learning DBs in the training stage. On the other hand, if a DB of too small a range is used for training, there is a problem that the optimal performance of the DNN cannot be obtained for various inputs due to overfitting.

Therefore, in deep learning-based processing of media content, a method that increases the efficiency of network infrastructure operation and can cope with the scientific learning phenomenon is required.

Non-Patent Document 1: Kuleshov, Volodymyr, S. Zayd Enam, and Stefano Ermon. "Audio super resolution using neural networks." arXiv preprint arXiv:1708.00853 (2017)

In the present disclosure, in providing a media service, the server receives identification information and parameters of a deep learning model trained in advance to be optimized for specific content. is transmitted to the terminal side with The main purpose of the terminal is to provide a media processing system and method capable of generating content with optimized media quality by applying specific content to a deep learning model.

According to an embodiment of the present invention, the content and the transceiver for receiving information of a DNN (Deep Neural Network) model trained in advance to be specialized to the content from the server; and an inference model configured based on the information of the DNN model, receiving the content and performing a media processing technique.

According to another embodiment of the present invention, in a media processing method performed by a media processing apparatus, the process of receiving content and information of a deep neural network (DNN) model trained in advance to be specialized to the content from a server ; constructing an inference model using the information of the DNN model; and performing a media processing technique by inputting the content into the inference model.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium to execute each step included in the media processing method.

As described above, according to this embodiment, in providing a media service, the server includes identification information of a deep learning model trained in advance to be optimized for specific content and There is an effect that it becomes possible to generate content with optimized media quality by providing a media processing system and method for transmitting a parameter together with a specific content to the terminal side, and for the terminal to apply the specific content to a deep learning model. .

In addition, according to this embodiment, in providing a media service, the server by providing a media processing system and method for delivering the identification information and parameters of the deep learning model trained in advance to be optimized for specific content to the terminal side along with the specific content. , effective use of the overfitting phenomenon and increase in efficiency in terms of network infrastructure operation are effective.

1 is an exemplary diagram of a media processing system according to an embodiment of the present invention.
2 is an exemplary diagram of a terminal according to an embodiment of the present invention.
3 is a flowchart of a media processing method according to an embodiment of the present invention.
4 is an exemplary diagram of a VoD service as an embodiment of the present invention.
5 is an exemplary diagram in which the media processing system according to the present embodiment is specially applied to the SR (Super Resolution) function.
6 is a conceptual diagram of a music streaming service.
7 is an exemplary diagram of a model learning process in a sound source streaming service as another embodiment of the present invention.
8 is an exemplary diagram of a music streaming service as another embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

In addition, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

This embodiment discloses the contents of a deep learning-based media processing system and method. More specifically, in providing a media service, the server receives identification information and parameters of a pre-trained deep learning model to be optimized for specific content. To provide a media processing system and method for delivering to the terminal side with

Examples of the media service may include, but are not limited to, a Video on Demand (VoD) or a music streaming service.

Since a deep learning model can be implemented as an artificial neural network, a deep learning model and a deep learning artificial neural network are used interchangeably.

Hereinafter, a media processing system according to the present embodiment will be described with reference to FIGS. 1 and 2 .

1 is an exemplary diagram of a media processing system according to an embodiment of the present invention.

When the media processing system 100 according to an embodiment of the present invention provides a media service, the server converts information and parameters of the deep learning model trained in advance to optimize the content requested by the user in the form of a bitstream. It is transmitted to the terminal side along with the content. The terminal constructs an inference model using information and parameters of the model, inputs the content to the inference model, and applies a media processing technique to generate content having optimized quality. The media processing system 100 includes all or part of a storage server (hereinafter, 'server') 101 , a wired or wireless network 110 , and a terminal 120 .

Here, the information of the deep learning model may include identification data about the shape and size of the deep learning model. In addition, the parameters of the deep learning model constitute the deep learning model, the number is determined according to the shape and size of the deep learning model, and is a coefficient to be trained.

The server 101 according to the present embodiment is a DNN (Deep Neural Network) trained in advance to be specialized in the content requested by the user. The model's identification information and parameters are transmitted to the terminal 120 along with the content. The server 101 includes a content DB (Database) 102 , a DNN DB 103 , and a model compression unit 105 . Here, the components included in the server 101 according to the present embodiment are not necessarily limited thereto. For example, a training unit (not shown) for training the DNN and a transceiver (not shown) for transmitting and receiving data to and from the terminal may be additionally provided on the server 101 . In addition, the information of the DNN model 104 may be identification data representing the type and structure of the model.

The content DB 102 includes media content that the server 101 can provide to a user. For example, in the case of a VoD service, the content may be a video, and in the case of a sound streaming service, the content may be a sound source (music), but the present invention is not limited thereto.

The content DB 102 may include target content for learning for training a deep learning model. For example, in the case of a VoD service, the content may be an FHD video, and the target content for learning may be a 4K video. In the case of a music streaming service, the content may be a low bit rate sound source, and the content for learning may be a high bit rate sound source.

The DNN DB 103 includes identification information and parameters of the DNN model 104 trained in advance to perform a media processing technique for each content. The server 101 may obtain content corresponding to the user's selection information from the content DB 102 , and obtain identification information and parameters of the DNN model 104 from the DNN DB 103 and deliver it to the terminal 120 side. .

Here, the media processing technique refers to up-scaling and/or quality improvement of content, but is not limited thereto. For example, in the case of a VoD service, the media processing technique may mean an increase in resolution and/or image quality of an image or video, and in the case of a sound source streaming service, an increase in a bit rate and/or an improvement in sound quality of a sound source.

In another embodiment of the present invention, when the DNN model 104 is fixed to one, the server 101 is a DNN The parameters of the model 104 may be transmitted to the terminal 120 side along with the content.

The model compression unit 105 compresses identification information and parameters of the trained DNN model 104 . In the case of VoD service, since the size of the DNN model 104 is tens to hundreds of MB (Megabyte), it may not be a big burden on the bandwidth of the available traffic infrastructure, but depending on the situation, the identification of the DNN model 104 is Information and parameters can be compressed and transmitted. For example, in the Moving Picture Expert Group-Neural Network Representation (MPEG-NNR), standardization studies for a technique capable of effectively compressing these parameters are in progress. The training process of the DNN will be described later.

The wired/wireless network 110 according to the present embodiment is a path for data transmission between the server 101 and the terminal 120 . Here, the data may include all or part of the user's content selection information, user terminal information, identification information of the content and/or DNN model, and parameters.

2 is an exemplary diagram of a terminal according to an embodiment of the present invention.

The terminal 120 according to this embodiment transmits the user's content selection information to the server 101 side, and obtains identification information and parameters of the content and the deep learning model from the server 101 . The terminal 120 constructs an inference model using the received model identification information and parameters, inputs the content to the DNN model, and applies a media processing technique, thereby generating content having an optimized quality. The terminal 120 includes all or part of a terminal transceiver unit 201 , an input unit 202 , an inference model 203 , and an output unit 204 .

The terminal transceiver 201 transmits the information of the terminal 120 and the user's content selection information to the server 101 using the wired/wireless network 110 . In addition, the terminal transceiver 201 receives from the server 101 identification information and parameters of the deep learning model trained in advance or in real time to be optimized for content and content in the form of a bitstream.

In another embodiment of the present invention, when the DNN model is fixed to one, the terminal transceiver 201 may receive the content and parameters of the deep learning model from the server 101 .

The input unit 202 is When the information and parameters of the DNN model 104 are compressed data, a function of decompressing the data may be performed. In addition, the input unit 202 may decode the content in the form of a bitstream and convert it to be suitable to be input to the inference model.

The inference model 203 is implemented using a DNN model. The inference model 203 may be formed using identification information and parameters for one deep learning model optimized for content to be reproduced, received from the server 101 . The terminal 120 may generate content having an optimized quality by inputting the content received from the server 101 into the inference model 203 and applying a media processing technique.

The output unit 204 converts the optimized content to fit the user interface and provides it to the user. For example, in the case of a VoD service, the output unit 204 may convert the optimized content into a visual and/or auditory form using a display and a speaker and provide it to the user. In addition, in the case of a sound streaming service, the output unit 204 may convert the content into an auditory form using a speaker and provide it to the user.

Hereinafter, learning of the DNN model 104 executed by the training unit of the server 101 will be described.

The server 101 may include a deep learning-based DNN model 104 , and the training unit may perform a training process for the provided DNN model 104 . The DNN model 104 may be any deep learning-based artificial neural network capable of executing media processing techniques. The DNN model 104 may be a pre-trained model to enable execution of a media processing technique suitable for a specific content based on a specific content of a VoD or sound streaming service.

A normal deep learning-based DNN model should be well applied to all contents in the inference stage by deriving parameters using a wide learning DB in the training stage. If the DNN model 104 is trained to be specialized for content as in the present embodiment, the optimal performance of the DNN model 104 cannot be implemented for various inputs due to overfitting. Therefore, in general, training the DNN model 104 to be specialized for a particular content should be avoided. However, in the present embodiment, after training the DNN model 104 to be optimized only for specific content, identification information and parameters of the DNN model and content are provided together. By doing this, the media processing system 100 according to the present embodiment can perform a media processing technique optimized for a specific content, thereby achieving excellent quality improvement of the content.

The training unit according to this embodiment updates the parameters of the DNN model 104 by using a loss function based on a distance metric between the output of the DNN model 104 for specific content and the target content for learning. do. Here, as the distance metric, any one capable of expressing a metric difference between two comparison objects, such as cross entropy, L1 or L2 metric, may be used.

In consideration of the diversity of the terminal 120 , the training unit may perform training on N (N is a natural number) DNN models 104 having different shapes. The media processing system 100 stores identification information and parameters of the DNN model 104 for which the loss function is minimized in combination with the content. When the user selects content later, the combined information may be provided to the terminal 120 side.

The training unit according to the present embodiment may generate identification information and parameters of the DNN model 104 by performing training on the DNN model 104 in advance.

In another embodiment of the present invention, when the computing power of the server 101 is sufficient or when the identification information and parameters of the DNN model 104 for the selected content are not prepared, the training unit for the DNN model 104 Training may be run in real time to generate identification information and parameters of the DNN model 104 .

1 and 2 are exemplary configurations according to the present embodiment, and include other components or other connections between components according to the type of server, the type of wired/wireless network, the type of terminal, and the type of the deep learning model. implementation is possible.

It is assumed that the server 101 of the media processing system 100 according to the present embodiment is mounted on the programmable system. It is assumed that the server 101 provides the information and parameters of the media content and the deep learning model requested by the user to the plurality of terminals 120 using a wired or wireless transmission method.

The terminal 120 of the media processing system 100 according to the present embodiment may be a programmable computer. The terminal 120 may be any device capable of receiving a media streaming service using a wireless or wired transmission path that is limited by a transmission band.

3 is a flowchart of a media processing method according to an embodiment of the present invention. 3 (a) is a flowchart executed by the server 101 of the media processing system 100 according to the present embodiment, and FIG. 3 (b) is a flowchart executed by the terminal 120 of the media processing system 100 It is a flowchart.

The server 101 of the media processing system 100 according to the present embodiment receives selected content information and user terminal information from the terminal 120 (S301).

The server 101 transmits information and parameters of the selected content and DNN model to the terminal (S302). The deep neural network (DNN) model 104 may be any deep learning-based neural network capable of executing media processing techniques, and may be trained in advance or in real time to be specialized to selected content. The information of the DNN model 104 may be identification data expressed in the type and structure of the model.

The server 101 may transmit identification information and parameters of the DNN model 104 in a compressed form.

In another embodiment of the present invention, when the DNN model 104 is fixed to one, the server 101 is a DNN The parameters of the model 104 may be transmitted to the terminal 120 side along with the content.

The terminal 120 of the media processing system 100 transmits the selected content information and the user terminal information to the server (S311).

The terminal 120 receives the selected content and information and parameters of the DNN model from the server (S312). The terminal 120 receives from the server 101 the identification information and parameters of the DNN model trained in advance or in real time to be specialized to the content and the content in the form of a bitstream.

In another embodiment of the present invention, when the DNN model is fixed to one, the terminal 120 may receive the content and parameters of the deep learning model from the server 101 .

The terminal 120 constructs an inference model using the information and parameters of the DNN model (S313).

When the identification information and parameters of the DNN model are compressed data, the terminal 120 may perform a function of decompressing the data. In addition, the terminal may decode the content in the form of a bitstream and convert it to be suitable to be input to an inference model (203).

The terminal 120 inputs the content to the inference model and performs a media processing technique (S314). The inference model 203 may generate content having an optimized quality by applying a media processing technique to the content.

The terminal 120 provides the optimized content to the user (S315). The terminal 120 may convert the optimized content into a visual and/or audible form to fit the user interface and provide it to the user.

As described above, according to this embodiment, in providing a media service, the server transmits identification information and parameters of a deep learning model trained in advance to be optimized for specific content to the terminal side along with the specific content, and the terminal By providing a media processing system and method for applying content to a deep learning model, there is an effect that it becomes possible to generate content having optimized media quality.

Hereinafter, an example in which the media processing system 100 according to the present embodiment is applied to a VoD service will be described with reference to FIGS. 4 and 5 .

In providing a VoD service, an Internet Protocol TV (IPTV) operator that is a service performing entity receives an original video from a CP (content provider) operator. Since the original video is usually of a large capacity, transcoding may be applied in consideration of the network bandwidth and provided to the IPTV service provider. Here, the transcoding process includes decoding and encoding of the original video. Since the encoding process is lossy compression compared to the original video, the video content of the IPTV service provider inevitably includes degradation in quality. Therefore, the IPTV operator can provide the user by combining information and parameters of the video content and the deep learning model after training a deep learning model capable of executing the media processing technique as presented in this embodiment for each video content. .

4 is an exemplary diagram of a VoD service as an embodiment of the present invention.

In providing the VoD service, the server 101 of the media processing system 100 according to the present embodiment transmits video content corresponding to the selected content information and trained DNN model information and parameters to the terminal side. After configuring an inference model using model information and parameters, the VoD service terminal STB (Settop Box) 120 may input video content to the inference model to generate video content having an optimized quality.

In providing the VoD service, as shown in FIG. 5 , the media processing system 100 according to the present embodiment may be specially applied to the SR (Super Resolution) function. For example, the training unit of the server 101 uses FHD (Full High-Definition) video content as an input, and uses 4K video content as a target video to train a DNN model to perform SR-level media processing techniques. A trained DNN model 104 can be generated. The IPTV operator transmits FHD video content and DNN model information and parameters to the STB 120 , and the STB 120 inputs the FHD video content to the DNN model to generate a 4K video with minimal image quality degradation.

Hereinafter, an example in which the media processing system 100 according to the present embodiment is applied to a sound source streaming service will be described with reference to FIGS. 6 to 8 .

In providing a sound source streaming service, the sound source for the service is stored in the DB of the server in advance. As shown in Figure 6, the streaming server encodes a 16-bit wav sound source sampled at 44.1 KHz at various bit rates, creates and stores a DB of the sound source for the service, and provides the user-selected song to the terminal side do. In the diagram of FIG. 6 , one arrow at the same position for each DB means one song.

7 is an exemplary diagram of a model learning process in a sound source streaming service as another embodiment of the present invention.

The server 101 of the media processing system 100 according to the present embodiment trains the DNN model to generate a high bitrate sound source for each sound source encoded at a low bitrate. For example, the training unit of the server 101 may use a 96 Kbps sound source as an input and a 320 Kbps sound source as a target sound source to perform training on the DNN model. The trained DNN model 104 may perform SR-level media processing techniques. The server 101 may secure the DNN model 104 trained in advance for all songs included in the sound source DB.

8 is an exemplary diagram of a music streaming service as another embodiment of the present invention.

In providing the sound source streaming service, the server 101 of the media processing system 100 according to the present embodiment transmits the low bit rate sound source content corresponding to the selected content information and the trained DNN model 104 information and parameters to the terminal 120 ) to the side. The terminal 120 receiving the streaming service constructs the inference model 203 using model information and parameters, and then inputs the low-bit-rate content to the inference model 203 to generate high-bit-rate content with optimized quality. have.

As described above, according to this embodiment, in providing a media service, a media processing system that delivers identification information and parameters of a deep learning model trained in advance so that the server is optimized for a specific content to the terminal side along with the specific content; By providing the method, it is possible to effectively use the scientific learning phenomenon and to increase the efficiency in terms of network infrastructure operation.

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein include digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. These computer-readable recording media are non-volatile or non-transitory, such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. media, and may further include transitory media such as carrier waves (eg, transmission over the Internet) and data transmission media. In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes may be stored and executed in a distributed manner.

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: media processing system 101: server
102: content DB 103: DNN DB
104: DNN model 105: model compression unit
110: wired/wireless network 120: terminal
201: terminal transceiver unit 202: input unit
203: inference model 204: output

Claims (9)

a transceiver for receiving content and information of a DNN (Deep Neural Network) model trained in advance to be specialized to the content from a server; and
An inference model configured based on the information of the DNN model and receiving the content and performing a media processing technique
Media processing apparatus comprising a. According to claim 1,
The transceiver unit,
The media processing apparatus according to claim 1, wherein the content information and terminal information on which the media processing apparatus is mounted are transmitted to the server side. According to claim 1,
The information of the DNN model is,
Media processing apparatus, characterized in that it includes identification information on the type and size of the DNN model, and parameters of the DNN model. According to claim 1,
The DNN model is
Media processing apparatus, characterized in that the parameter of the DNN model is updated to perform a media processing technique for mapping between the content and the target content by using the content and the target content for learning. According to claim 1,
The media processing technique is
Media processing apparatus comprising up-scaling and/or quality enhancement for the content. In the media processing method performed by the media processing apparatus,
Receiving content and information of a DNN (Deep Neural Network) model trained in advance to be specialized to the content from a server;
constructing an inference model using the information of the DNN model; and
A process of performing a media processing technique by inputting the content into the inference model
Media processing method comprising a. 7. The method of claim 6,
The method of claim 1, further comprising the step of transmitting the content information and terminal information on which the media processing apparatus is mounted to the server. 7. The method of claim 6,
The receiving process is
The media processing method, characterized in that receiving identification information on the type and size of the DNN model, and parameters of the DNN model. A computer program stored in a computer-readable recording medium to execute each step included in the media processing method according to any one of claims 6 to 8.

Images