KR20230089753A - Method and apparatus for live streaming - Google Patents

Abstract

A live streaming method and apparatus are disclosed. A method of operating a server according to an embodiment includes dividing a video stream for live streaming into segments having a predetermined length corresponding to each of a plurality of resolutions, and corresponding to each of the plurality of resolutions, at a resolution higher than the corresponding resolution. Updating the pre-learned parameters of the neural network based on the segment based on the segment of the corresponding resolution, compressing and storing data about the parameters of the neural network corresponding to each of a plurality of resolutions, and requesting live streaming of the video stream. The method may include transmitting data and segments related to parameters of the compressed neural network corresponding to at least one of a plurality of resolutions to the client.

Description

Live streaming method and apparatus {METHOD AND APPARATUS FOR LIVE STREAMING}

The following embodiments relate to a live streaming method and apparatus, and specifically to a server and a client of a live streaming service.

Recently, as the demand for video content increases, a deep learning-based video restoration method is being developed to improve quality of user experience (QoE) in a video streaming environment. In particular, the Super-Resolution (SR) approach that can convert Low Resolution (LR) images to High Resolution (HR) images based on CNN (Convolutional Neural Network) has improved visual quality. showed the possibility of improvement, and as a result, various neural network-based streaming pipelines using SR have been proposed. For example, Dejavu is a streaming system that enhances the quality of restored video by training images in the video conferencing category based on the fact that visual similarity between frames is relatively high in the case of video conferencing video. NAS is a video streaming system based on a CNN model that trains a pair of low-resolution and high-resolution video frames for each video episode to recover the SR version of the input video.

The following embodiments provide a technique for transmitting low-resolution images and neural network parameters for super-resolution high-resolution images in real time without inefficiently using a large amount of network bandwidth when transmitting neural network parameters for live streaming to clients. can do.

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

A method of operating a server according to an embodiment includes dividing a video stream for live streaming into segments having a predetermined length corresponding to each of a plurality of resolutions; Corresponding to each of the plurality of resolutions, updating a parameter of a neural network pre-learned based on a segment having a higher resolution than the corresponding resolution based on the segment having the corresponding resolution; compressing and storing data about parameters of a neural network corresponding to each of the plurality of resolutions; and transmitting data and segments related to parameters of a compressed neural network corresponding to at least one of the plurality of resolutions to a client requesting live streaming of the video stream.

The updating based on the segment of the corresponding resolution may include: acquiring a neural network corresponding to the first resolution by updating a parameter of a pretrained neural network based on a segment of a first resolution among the plurality of resolutions; and obtaining a neural network corresponding to the second resolution by updating a parameter of the neural network corresponding to the first resolution based on a segment of the second resolution among the plurality of resolutions. One resolution may include a resolution higher than the second resolution.

The operating method of the server is based on at least one of interpolation and inverse pixel shuffle of segments corresponding to each of the plurality of resolutions, down sampling a segment corresponding to each of the plurality of resolutions ) may further include a step of doing.

The compressing and storing may include quantizing a difference value between a parameter of a neural network corresponding to each of the plurality of resolutions and a parameter of a pretrained neural network; and storing encoding data of the quantized value corresponding to each of the plurality of resolutions.

The updating based on the segment of the corresponding resolution is performed by repeatedly performing learning iterations based on the frames included in the segment of the corresponding resolution, thereby generating the first neural network corresponding to a resolution higher than the corresponding resolution. updating parameters; and acquiring a second neural network corresponding to the corresponding resolution including the updated parameters of the first neural network, based on the end of repetitions of the learning iteration, wherein the learning iteration performs the corresponding training iteration. Back-propagation of a gradient obtained based on a loss function related to a difference between an output frame obtained by applying a frame of a segment corresponding to a resolution to the first neural network and correct answer data to the first neural network steps may be included.

Updating the parameters of the first neural network may include loading frames included in the segment of the corresponding resolution; and repeating the learning iteration based on the loaded frames.

The repetition of the learning iteration may be performed during the duration of the segment of the corresponding resolution.

The updating based on the segment of the corresponding resolution may include fixing a value of a parameter of the neural network other than a weight included in a residual block of the neural network; and updating a weight included in the residual block of the neural network.

A method of operating a client according to an embodiment includes receiving a playlist of a video stream corresponding to a selected first resolution from a server for live streaming; receiving, from the server, data about a segment of the first resolution corresponding to the video stream and a neural network corresponding to the first resolution, based on the playlist; Restoring a neural network corresponding to the first resolution based on data related to the neural network corresponding to the first resolution and parameters of a pre-trained neural network loaded in advance; and converting segments of the first resolution into segments of a second resolution based on a neural network corresponding to the reconstructed first resolution.

The restoring of the neural network corresponding to the first resolution may include obtaining a residual value of the neural network corresponding to the first resolution by decoding data related to parameters of the neural network corresponding to the first resolution obtained from the server. ; and restoring a parameter of the neural network corresponding to the first resolution by adding the obtained residual value to the parameter of the pretrained neural network.

The first resolution may include a resolution lower than the second resolution determined based on at least one of a selection input received from a user and a communication environment of a network through which the video stream is received.

The parameter of the neural network corresponding to the first resolution is set based on a segment of a first resolution among a plurality of predetermined resolutions in the server and based on a segment of a resolution higher than the first resolution among the plurality of resolutions. It can be obtained by updating parameters of the trained neural network or the pretrained neural network.

The operating method of the client may further include reproducing frames of the converted second resolution segment.

The server according to one side divides the video stream for live streaming into segments of a predetermined length corresponding to each of a plurality of resolutions, and corresponding to each of the plurality of resolutions, a resolution higher than the corresponding resolution. A client that updates a previously learned neural network parameter based on a segment based on a segment of the corresponding resolution, compresses data on the neural network parameter corresponding to each of the plurality of resolutions, and requests live streaming of the video stream. , at least one processor for transmitting data and segments related to a parameter of a compressed neural network corresponding to at least one of the plurality of resolutions.

In the update based on the segment of the corresponding resolution, the processor updates the parameters of the pretrained neural network based on the segment of the first resolution among the plurality of resolutions, thereby forming a neural network corresponding to the first resolution. obtaining a neural network corresponding to the second resolution by updating a parameter of the neural network corresponding to the first resolution based on a segment of the second resolution among the plurality of resolutions; may include a resolution higher than the second resolution.

The processor may down-sample a segment corresponding to each of the plurality of resolutions based on at least one of interpolation and inverse pixel shuffle of the segment corresponding to each of the plurality of resolutions. can

In the compression and storage, the processor quantizes a difference between a parameter of a neural network corresponding to each of the plurality of resolutions and a parameter of a pretrained neural network, and the quantized value corresponding to each of the plurality of resolutions. Encoding data of can be stored.

In updating based on the segment of the corresponding resolution, the processor repeatedly performs learning iterations based on frames included in the segment of the corresponding resolution, thereby corresponding to a resolution higher than the corresponding resolution. update the parameters of the first neural network, and obtain a second neural network corresponding to the corresponding resolution including the updated parameters of the first neural network based on the end of repetition of the learning iteration; The ration backpropagates a gradient obtained based on a loss function for a difference between an output frame obtained by applying a frame of a segment corresponding to the corresponding resolution to the first neural network and the correct answer data to the first neural network ( back-propagation).

A terminal of a client according to one side receives a playlist of a video stream corresponding to a selected first resolution from a server for live streaming, and based on the playlist, the first video stream corresponding to the video stream from the server. Receive data about a segment of the resolution and a neural network corresponding to the first resolution, and respond to the first resolution based on the data about the neural network corresponding to the first resolution and the pre-loaded parameters of the neural network. and at least one processor that restores a neural network corresponding to the restored first resolution, and converts segments of the first resolution into segments of a second resolution based on the neural network corresponding to the restored first resolution.

Through the following embodiments, a server providing a live streaming service may provide a deep learning-based live streaming pipeline that transmits low-resolution video segments that are updated and compressed in real time and parameters of a neural network.

Through the following embodiments, it is possible to provide a deep learning-based live streaming pipeline with high super-resolution accuracy and low computational complexity.

The following examples promote the inherent overfitting property of CNNs while using a small number of iterations to achieve real-time performance, and apply a multi-scale, curriculum-based learning approach to CNNs for ABR-based live streaming to improve learning efficiency. Accuracy can be increased quickly.

Through the following embodiments, if the pre-learned parameter is already stored in the client-side device, only the residual value of the updated parameter is transmitted, thereby greatly reducing the quantization range of the parameter within the range of not sacrificing the image bit rate significantly, within the limited transmission capacity. can improve QoE.

1 is a diagram showing an overview of a live streaming system according to an embodiment.
2 is a diagram illustrating a pipeline of a live streaming system according to an embodiment.
3A to 3D are diagrams illustrating the structure of a neural network according to an embodiment.
4 is a diagram for explaining improvement in learning accuracy due to a structure of a neural network according to an exemplary embodiment.
5 is a diagram illustrating a timeline of operations performed on the server side for live streaming according to an embodiment.
6 is a diagram illustrating a specific algorithm of a real-time learning operation according to an embodiment.
7 is a diagram for explaining compression performance of parameters of a neural network according to an exemplary embodiment.
8 is a flowchart of a method of operating a server according to an embodiment.
9 is a diagram illustrating a timeline of operations performed on a client side for live streaming according to an embodiment.
10 is a flowchart of a method of operating a client according to an embodiment.
11 is an exemplary view of a configuration of a device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram showing an overview of a live streaming system according to an embodiment.

Referring to FIG. 1 , a live streaming system 100 according to an embodiment may include a server 101 providing a live streaming service and a client 102 requesting a live streaming service. Live streaming is transmitting and reproducing multimedia files such as sound and images in real time. to be sent to Hereinafter, the streaming target will be described as an example of a video stream.

The system 100 according to an embodiment may follow an adaptive bitrate (ABR) streaming pipeline based on HTTP Live Streaming (HLS). In order to constantly provide a video stream to the client 102 in real time regardless of fluctuations in network bandwidth between the server 101 and the client 102, the server 101 provides an adaptive bitrate (ABR) ) algorithm, video segments obtained by fragmenting a video stream may be transmitted to the client 102 at various values of resolution and bitrate according to circumstances.

The server 101 according to an embodiment may perform real-time training of a neural network for super resolution of a low-resolution video segment transmitted to the client 102, , may transmit the low-resolution video segment file 111 and parameters 112 of the trained neural network for super-resolutioning the low-resolution video segment to the client 102 . For example, the neural network may include a convolution neural network (CNN). Hereinafter, a neural network for super-resolution in the server 101 according to an embodiment is described as an example of CNN.

The parameter 112 of the learned neural network according to an embodiment may include a weight and/or bias of at least one layer included in the neural network, and the value is determined or determined by learning. can be updated Although described in detail below, the parameter 112 of the neural network for super-resolution transmitted to the client 102 may correspond to compressed data.

According to an embodiment, on the server 101 side, the neural network may be trained in real time based on a video stream corresponding to each of a plurality of resolutions. The plurality of resolutions may include a plurality of resolutions corresponding to a predetermined low resolution, and may include, for example, resolutions of 240p, 360p, 480p, and 720p. In the real-time learning process, high-resolution video streams can be used as ground truth. A high-resolution video stream may include a resolution higher than a plurality of resolutions, for example, a resolution of 1080p. As a result of learning the neural network based on the video stream corresponding to each of the plurality of resolutions, updated parameters of the neural network corresponding to each of the plurality of resolutions may be obtained. For example, parameters of a neural network updated based on a 240p video stream and parameters of a neural network updated based on a 480p video stream may be different from each other.

According to an embodiment, the parameters 112 of the neural network corresponding to each resolution may be compressed and stored, and the compressed parameters 112 of the neural network may be transmitted to the client 102 . The client 102 may request a video segment of a specific resolution based on a communication environment or a user's input, and a file 111 of the video segment corresponding to the corresponding resolution and a parameter 112 of the compressed neural network corresponding to the corresponding resolution may be transmitted to the client 102.

The client 102 according to an embodiment may render a video in real time based on the video segment file 111 received from the server 101 and the parameter 112 of the compressed neural network. . In the process of real-time video rendering, a video segment file 111 of a specific resolution may be super-resolution based on the parameters 112 of the compressed neural network, and the video reconstructed in high resolution according to the super-resolution is displayed on the client 102 side. It can be played through the terminal of.

In the live streaming system 100 according to an embodiment, a specific operating method of the server 101 for providing a high-resolution live streaming service and a specific operating method of the client 102 for reproducing high-resolution live streaming are described below. detail

2 is a diagram illustrating a pipeline of a live streaming system according to an embodiment.

Referring to FIG. 2 , a system 200 according to an embodiment may correspond to the system 100 shown in FIG. 1 . For example, the server 201 and client 202 shown in FIG. 2 may correspond to the server 101 and client 102 shown in FIG. 1 , respectively.

The system 200 according to an embodiment may follow an adaptive bitrate (ABR) streaming pipeline based on HTTP Live Streaming (HLS). As an example, the server 201 acquires a video stream through RTMP (Real-Time Messaging Protocol) 203, and the obtained video stream converts data to a client 202 terminal by HLS encoding 210. A data format may be set to be recognized and interpreted, and may be divided into segments of a predetermined length corresponding to each of a plurality of resolutions. For example, a video stream may be divided into 0.5 second segments, and each segment may be duplicated and generated into segments 211 corresponding to resolutions of 240p, 360p, 480p, 720p, and 1080p, respectively. In other words, a segment corresponding to 0 to 0.5 seconds of the video stream may be duplicated as five segments 211 corresponding to 0 to 0.5 seconds corresponding to resolutions of 240p, 360p, 480p, 720p, and 1080p, respectively.

For learning of the neural network according to an embodiment, the video frame(s) included in the 1080p resolution segment is used as the ground truth 213 of the high resolution (HR) frame, and the remaining resolution less than 1080p Video frame(s) in segment(s) of (s) may be used as training data 212 of low resolution (LR) frames. In the real-time learning step 220, a pre-trained neural network (N _pre ) may be used, and parameters of the pre-trained neural network (N _pre ) are updated based on video segments of each resolution to correspond to the parameters of the neural network corresponding to each resolution. (221) can be obtained. For example, the pre-trained neural network may include a CNN pre-trained with an arbitrary SR (Super Resolution) data set. Parameters of the pre-trained neural network to meet the real-time condition may be updated to parameter values corresponding to each resolution through fine-tuning during the length or duration of the video segment.

According to an embodiment, the updated parameters corresponding to segments of each resolution of the video stream may be compressed by quantization and/or encoding in the compression step 230, and the parameter of the compressed neural network corresponding to each resolution may be compressed. Related data 231 may be stored. When data 231 related to parameters of the compressed neural network corresponding to each resolution is stored, a playlist file corresponding to each resolution may be updated (240).

The client 202 according to an embodiment may download a video segment file 242 of a specific resolution and data 244 related to parameters of a neural network corresponding to the resolution based on the playlist 241 . For example, since the data 244 on the parameters of the neural network uses the same file path as the path of the file 242 of a video segment with a different extension, the format of the file in which the data 244 on the parameters of the neural network is stored is standard. It may correspond to the format of the HLS playlist.

According to an embodiment, on the client 202 side, when the ABR controller module 204 selects an appropriate resolution r according to the network bandwidth, the video segment file 242 of the selected resolution r and the resolution r ) may be downloaded to the terminal of the client 202 through the network. For example, the network may include a content delivery network (CDN).

)로 복원될 수 있다. 압축된 신경망의 파라미터에 관한 데이터(244)는 클라이언트(202) 측에 저장된 사전 학습된 신경망의 파라미터(N_pre)에 기초하여 복원될 수 있다.According to one embodiment, the segment file 242 downloaded to the client 202 may be decoded into low-resolution video frame(s) 243, and the data 244 related to parameters of the downloaded compressed neural network may be By the restoration step 250, the parameters of the neural network corresponding to the corresponding resolution (r) (

) can be restored. The data 244 on parameters of the compressed neural network may be restored based on the pre-trained parameters (N _pre ) of the neural network stored on the client 202 side.

)에 기초하여, 고해상도의 비디오 프레임(들)(261)로 변환될 수 있으며, 변환된 고해상도의 비디오 프레임(들)(261)은 렌더링을 통해 클라이언트(202)의 단말에서 재생될 수 있다.According to one embodiment, in the super-resolution step 260, the video frame(s) 243 of the low resolution are the parameters of the reconstructed neural network (

), it can be converted into high-resolution video frame(s) 261, and the converted high-resolution video frame(s) 261 can be reproduced in the terminal of the client 202 through rendering.

3A to 3D are diagrams illustrating the structure of a neural network according to an embodiment.

Referring to FIG. 3A, a neural network 300 according to an embodiment includes a down sampler 310, at least one residual block (ResBlocks) 320, and a long skip connection. An up sampler 330 with 340 may be included.

In order to prevent runtime performance from being different for each resolution due to differences in the resolution of frames input to the neural network 300 according to an embodiment, frames of low resolution (e.g., 240p, 360p, 480p, 720p) are first converted into high-resolution (e.g., 1080p) frames. ) can be interpolated to the domain of For example, since the frame 301 interpolated with the same resolution (eg, 1080p) regardless of the resolution of the input frame by bicubic interpolation is input to the neural network 300, the neural network 300 for the input frame The time for performing the operation of may be the same regardless of the resolution of the input frame. The interpolated frame 301 in the high-resolution domain may be applied to the down sampler 310 to perform inverse pixel shuffle, and a feature may be generated based on at least one layer included in the down sampler 310. can be extracted. Performing inverse pixel shuffling in the downsampler 310 is to prevent the computation execution time of the neural network 300 from being longer than when a low-resolution frame is input as the high-resolution interpolated frame 301 is input, For example, the calculation time of the neural network 300 can be reduced by reducing the size of the interpolated frame 301 by 4 times through inverse pixel shuffling. For example, referring to FIG. 3B , the down sampler 310 may include a layer 311 performing inverse pixel shuffling and at least one convolution layer 312 .

The neural network 300 according to an embodiment may include at least one (eg, four) residual blocks 320 . For example, referring to FIG. 3C , a residual block 320 may include at least one convolution layer 321 and 323 and an activation layer (eg, ReLU) 322, and an output value of the residual block It may be implemented so that the input values of the residual blocks are added.

The neural network 300 according to an embodiment may include an up-sampler 330. For example, referring to FIG. 3D, the up-sampler 330 performs at least one convolution layer 331 and pixel shuffle. It may include a layer 332 to perform.

The neural network 300 according to an embodiment is a long skip connection (long skip connection) between the output of the upsampler 330 and the frame 301 interpolated to the high resolution domain, which is the input of the neural network, in order to preserve features corresponding to the high resolution of the frame. skip connection) (340). The super-resolution frame 302 obtained based on the output of the up-sampler 330 and the interpolated frame 301 in the high-resolution domain may correspond to a high-resolution frame estimated from a low-resolution frame.

The neural network 300 according to an embodiment may be trained based on a loss function 350 related to a difference between the super-resolution frame 302, which is an output of the neural network, and the high-resolution frame 330, which is correct answer data.

The structure of the neural network 300 including the down sampler 310 and the long skip connection 340 according to an embodiment can improve learning accuracy within the same iteration and calculation complexity. As an example, comparing row 4 and rows 1 to 3 of the table shown in FIG. 4, the PSNR (Peak Signal-to-noise ratio) of a neural network including a down sampler and a long skip connection is higher than that of neural networks with other structures. can be seen to appear. The PSNR is a maximum signal-to-noise ratio, which is an index for evaluating loss information for image quality. The larger the PSNR, the higher the quality.

5 is a diagram illustrating a timeline of operations performed on the server side for live streaming according to an embodiment.

In FIG. 5 , S ₀ , S ₁ , S ₂ , and S ₃ denote segments in time order, and blocks in which S ₀ , S ₁ , S ₂ or S ₃ are described denote an operation corresponding to each segment. The horizontal length of a block means the time during which a corresponding operation is performed corresponding to a corresponding segment.

Referring to FIG. 5, operations performed on the server side for live streaming include HLS encoding operation, data preparation operation, real-time training operation, and neural network compression (NN compression). Actions may be included. In a general HLS player, a playlist is created at the time (t = 0, 1, 2, 3) when HLS encoding or transcoding of each segment (S ₀ , S ₁ , S ₂ , S ₃ ) of the video stream is completed. On the other hand, the time when the playlist is updated in the server according to an embodiment is the HLS encoding operation, data preparation operation, real-time learning operation, and compression operation of the neural network for each segment (S ₀ , S ₁ , S ₂ ). It may correspond to the point of completion (t = 0, 1, 2).

For example, server operations including an HLS encoding operation, a data preparation operation, a real-time learning operation, and a neural network compression operation performed on the same segment may be sequentially performed. Although FIG. 5 illustrates that a series of operations for each segment are sequentially performed according to the time order of the segments, a series of operations for some segments may be performed in parallel according to an embodiment.

According to an embodiment, the data preparation operation divides the received video stream into segments of a predetermined length, and generates each segment into segments (eg, segments 211 of FIG. 2 ) corresponding to a plurality of resolutions. action may be included.

According to an embodiment, the data preparation operation may include an operation of preparing a low-resolution segment data set by interpolating each of the segments corresponding to a plurality of resolutions. For example, for each of the segments corresponding to a plurality of resolutions, bicubic downsampling is performed according to the ratio of the high resolution image (1080p) to the low resolution (240p 360p, 480p, 720p), Segments corresponding to each low resolution may be interpolated. The low-resolution segment data set may include frame(s) of an interpolated video segment corresponding to each of a plurality of resolutions.

A real-time learning operation according to an embodiment may include an operation of fine-tuning a pretrained neural network. As an example, the DIV2K data set can be used to learn an implicit representation of a general high-resolution natural image prior, and the pre-trained neural network can be used to learn the high-resolution natural image prior included in the DIV2K data set. It may include parameters obtained by learning an implicit representation.

According to an embodiment, the real-time learning operation may include an operation of pre-processing a low-resolution segment data set. [10 , 50], compression may be applied to each low-resolution frame based on an arbitrary compression factor (or Q level) sampled in the range. A compression method applied to a low-resolution frame may include, for example, a JPEG2000 compression method. Using compressed low-resolution frames as a data set for training can contribute to rapidly increasing the accuracy of neural network training.

A real-time learning operation according to an embodiment may include operation(s) according to a multi-scale learning method. Based on the multi-scale training method of pre-trained neural networks, images corresponding to different resolutions can be assigned to the same batch of neural networks. A multi-scale learning method according to an embodiment will be described in detail below.

According to one embodiment, data such as image cropping, batch normalization, and image augmentation for patch generation to promote the over-fitting property inherent in CNNs. The process related to generalization of can be omitted in the real-time learning operation. Since the process related to generalization of data requires additional time for data processing, the time for repetition of learning iterations is reduced compared to using the data as it is in a real-time learning environment with limited learning time, so the accuracy of learning may fall In addition, the time to load newly processed training data to the GPU for each training iteration can act as a bottleneck. Therefore, by preloading the video frames included in the downsampled low-resolution segment into the GPU and using the preloaded frames in all iterations, the time required to reload the frames for each iteration is reduced, and the learning data By increasing the number of iterations of the ration, it can contribute to increasing the learning accuracy as a result.

According to an embodiment, the real-time learning operation may include an operation of fixing some parameter values so that only some parameter values can be updated instead of updating all parameters of the neural network in order to increase learning accuracy within a limited learning time. For example, weights and bias values of other layers not included in the residual block may be fixed so that only the weights included in the residual block are updated so as not to change during the learning process. By updating only some parameters instead of updating all parameters of the neural network, the server can use computing resources for the increased number of iterations and reduce the size of the updated neural network.

According to one embodiment, the real-time learning operation is a curriculum-based multi-scale (multi-scale) that gradually increases the difficulty of the learning target in order to mitigate deterioration in inference quality while ensuring consistent runtime performance regardless of the resolution of the frame applied to the neural network. scale) learning method. The multi-scale learning method may correspond to a learning method of training a neural network based on a segment of the highest resolution among a plurality of low resolutions and using the learned neural network to learn a neural network corresponding to the next higher resolution. Multi-scale learning methods can rapidly increase learning accuracy, especially at the lowest resolution (e.g. 240p). By using a multi-scale learning method for learning a neural network corresponding to each resolution according to an embodiment, when using a neural network including the same parameters corresponding to frames of different resolutions, quality degradation of inference at low resolution can be improved. It can reduce the learning time of the neural network corresponding to each resolution in live streaming, which requires a limited learning time.

A detailed algorithm of a real-time learning operation according to an embodiment may refer to FIG. 6 . Referring to Figure 6, Algorithm 1 describes the real-time learning process shown in Figure 6. Lines 5 to 6 of Algorithm 1 may correspond to an operation of preloading frames included in a segment into the GPU in order to increase the number of training iterations. Lines 8-21 may correspond to a curriculum-based multi-scale learning operation. According to line 2, the initial iteration number n ₀ can be set to 100×d, where d is a learning time based on a specific video segment and can correspond to the duration (seconds) of the video segment. Referring to line 23, the number of iterations n _t corresponding to each resolution can be estimated based on the average value of the time T _iter of one iteration after the parameter update and the execution time of all iterations. Referring to lines 15 and 16, due to real-time constraints, if the learning time exceeds d, the learning process for updating parameters corresponding to the segment may be stopped.

According to an embodiment, parameters of the neural network corresponding to each video segment may be transmitted to the client. To maximize video quality within limited network bandwidth, the parameters of the neural network can be compressed in real time while minimizing compression loss in a live streaming environment. To this end, the server of the system according to an embodiment may perform a neural network compression operation for compressing parameters of the neural network.

A neural network compression operation according to an embodiment includes an operation of compressing a neural network parameter corresponding to each resolution using an algorithm for compressing a difference between a neural network parameter corresponding to a plurality of resolutions and a pretrained neural network parameter. can include As an example, referring to FIG. 7 , a distribution range of residual weight values (hereinafter, residual values) between a parameter of a neural network corresponding to each of a plurality of resolutions and a parameter of a pretrained neural network is a distribution range of each of a plurality of resolutions. It can be experimentally observed that it is narrower than the distribution range of the weight values of updated parameters of the neural network corresponding to . In order to reduce the quantization range, a residual value of a neural network having a relatively narrow distribution range of values may be set as a quantization target. Since the parameters (N _pre ) of the pre-trained neural network are already stored on the client side, only the updated residual value (R ^r _t ) of the neural network may be transmitted to the client side as shown in Equation 1 below.

In Equation 1, N ^r _t is a parameter of the neural network updated based on the low-resolution th video segment.

For example, the server may perform quantization (Q) and encoding (H) as in Equation 2 below to compress R ^r _t .

)는 8비트 정수로 의 값을 저장하고 16비트 부동 소수점으로 클러스터 중심(C)을 저장하며, 클라이언트 측에서의 디코딩을 위한 C의 (l × 32) 비트 허프만 사전을 저장할 수 있다. 잔차 값을 압축하여 저장하는 방법을 통해 학습 정확도의 저하가 감소될 수 있다.For example, k-means clustering of 1 cluster centers C may be used for quantization of R ^r _t , where randomly sampled 10% of neural network parameters may be used for high-speed clustering. Thereafter, Huffman-Encoding generally used for lossless data compression based on an occurrence probability may be applied to the quantized R ^r _t . Finally, the parameters of the encoded neural network (

) can store the value of C as an 8-bit integer, store the cluster center (C) as a 16-bit floating point, and store a (l × 32) bit Huffman dictionary of C for decoding on the client side. Deterioration in learning accuracy may be reduced through a method of compressing and storing residual values.

According to an embodiment, for live streaming, the server may transmit video segments and compressed neural network parameters to the client. According to the general bit rate of HLS streaming, the bit rate for video transmission may be sacrificed as much as the bit rate for transmitting parameters of the neural network.

As an example, the total bitrate of a video segment is (400-β, 800-β, 1200-β, 2400-β, 4800-β) Kbps for videos with resolutions of (240, 360, 480, 720, 1080)p , where β may correspond to the bitrate of the neural network divided by the size of the compressed neural network by the duration of the video segment. In order to select the optimal β, a tradeoff between the size of a compressed neural network parameter and compression loss may be considered. For example, as the number of repetitions of training iterations increases, the compression quality of the neural network may decrease. Considering this sensitivity, the number of quantizations l may be set differently according to the video segment duration.

8 is a flowchart of a method of operating a server according to an embodiment.

A server according to an embodiment is a server that provides a live streaming service to clients in a live streaming system, and may correspond to, for example, the server 101 of FIG. 1 or the server 201 of FIG. 2 .

Referring to FIG. 8, a method of operating a server according to an embodiment includes dividing a video stream for live streaming into segments having a predetermined length corresponding to each of a plurality of resolutions (810), each of a plurality of resolutions. Correspondingly, step 820 of updating pre-learned parameters of the neural network based on segments of a resolution higher than the corresponding resolution based on the segments of the corresponding resolution, compressing data about parameters of the neural network corresponding to each of a plurality of resolutions. and storing (830) and transmitting (840) data and segments related to parameters of a compressed neural network corresponding to at least one of a plurality of resolutions to a client that has requested live streaming of a video stream. .

A method of operating a server according to an embodiment may correspond to operations performed by the server described above with reference to FIG. 5 or operations performed by the server 201 shown in FIG. 2 . For example, step 810 may correspond to the data preparation operation described above with reference to FIG. 5 and may include an operation of generating segments 211 corresponding to a plurality of resolutions shown in FIG. 2 . Step 801 according to an embodiment may further include HLS-encoding the obtained video stream, and the HLS-encoding step may include the HLS encoding operation described above with reference to FIG. 5 or the HLS encoding operation described above with reference to FIG. 2 . (210).

For example, step 820 may correspond to the real-time learning operation described above with reference to FIG. 5 and may correspond to step 220 of real-time learning shown in FIG. 2 .

Prior to step 820 of updating based on a segment of a corresponding resolution, a method of operating a server according to an embodiment performs at least one of interpolation and inverse pixel shuffle of a segment corresponding to each of a plurality of resolutions. Based on this, the method may further include down-sampling a segment corresponding to each of a plurality of resolutions. The step of downsampling interpolates the frame of the segment of the low resolution (eg 240p, 360p, 480p, 720p) described above through FIG. This may correspond to an operation of performing inverse pixel shuffle by applying the downsampler 310 to the downsampler 301.

Step 820 according to an embodiment may include fixing a value of a parameter of the neural network other than a weight included in the residual block of the neural network and updating a weight included in the residual block of the neural network. there is. As described above, in the learning process of updating the parameters of the neural network, learning can be efficiently performed by updating only some parameters instead of updating all parameters of the neural network.

Step 820 according to an embodiment may include operations according to the above-described multi-scale learning method. For example, in step 820, based on a segment of a first resolution among a plurality of resolutions, obtaining a neural network corresponding to the first resolution by updating a parameter (eg, N _pre ) of the pretrained neural network; and The method may include acquiring a neural network corresponding to the second resolution by updating a parameter of the neural network corresponding to the first resolution based on a segment of the second resolution among the plurality of resolutions. The first resolution may include a resolution higher than the second resolution. For example, the first resolution may correspond to the highest resolution among a plurality of resolutions. The parameter of the neural network corresponding to the first resolution may include a parameter value updated by learning a pretrained neural network based on segments of the first resolution. The parameter of the neural network corresponding to the second resolution lower than the first resolution may include a parameter value updated by learning the parameter of the neural network corresponding to the first resolution based on the segments of the first resolution. According to an embodiment, when a third resolution that is lower than the second resolution among the plurality of resolutions is included, a neural network corresponding to the second resolution may be learned based on segments of the third resolution and updated by learning. Parameters of the neural network corresponding to the third resolution including the parameter values may be obtained. Parameters of a neural network corresponding to each of a plurality of resolutions may be obtained according to the multi-scale learning method.

As another example, in step 820, updating parameters of the first neural network corresponding to a resolution higher than the corresponding resolution by repeatedly performing training iterations based on frames included in the segment of the corresponding resolution, and learning data The method may include acquiring a second neural network corresponding to a corresponding resolution including updated parameters of the first neural network, based on the end of repetition of the simulation. The learning iteration backpropagates the gradient obtained based on the loss function for the difference between the output frame obtained by applying the frame of the segment corresponding to the corresponding resolution to the first neural network and the correct answer data to the first neural network. -propagation) may be included. The loss function for learning the neural network may include a loss function related to a difference between the super-resolution frame 302, which is an output of the neural network, and the high-resolution frame 330, which is the correct answer data, as described above with reference to FIG. 3A.

Updating parameters of the first neural network according to an embodiment may include loading frames included in a segment of a corresponding resolution and repeating training iterations based on the loaded frames. As described above, since the time to load the newly processed training data into the GPU for each training iteration can act as a bottleneck, the process related to data generalization is omitted and preloaded frames are used in all iterations, so that the limited time It is possible to reduce data processing time and increase the number of iterations of training iterations within

The repetition of learning iterations according to an embodiment may be performed during the duration of a segment of a corresponding resolution. In other words, the total learning time corresponding to a specific video segment may be limited to the duration (eg, d) of the corresponding segment due to the constraint condition of real time.

For example, step 830 may correspond to the neural network compression operation described above with reference to FIG. 5 and may include the compression step 230 and the playlist update step 240 shown in FIG. 2 .

The compressing and storing step 830 according to an embodiment includes quantizing a difference value between a parameter of a neural network corresponding to each of a plurality of resolutions and a parameter of a pretrained neural network, and quantizing a value corresponding to each of a plurality of resolutions. It may include storing the encoded data of the value.

As an example, step 840 may correspond to an operation of providing the video segment file 242 shown in FIG. 2 and data 244 related to parameters of the neural network to the client 202 .

9 is a diagram illustrating a timeline of operations performed on a client side for live streaming according to an embodiment.

In FIG. 9 , S ₀ , S ₁ , S ₂ , and S ₃ denote segments in time order, and blocks described as S ₀ , S ₁ , S ₂ or S ₃ denote an operation corresponding to each segment. The horizontal length of a block means the time during which a corresponding operation is performed corresponding to a corresponding segment.

Referring to FIG. 9 , operations performed on the client side for live streaming may include a download operation, a neural network decoding (NN decoding) operation, a super-resolution (SR) operation, and a playblack operation. . Operations performed by a client according to an embodiment may be performed in parallel.

According to an embodiment, a new live video segment and corresponding neural network data may be downloaded at P _t (t = 0, 1, 2...). The neural network data may include data about parameters of the neural network or data about compressed parameters of the neural network. At each point in time P _t (t = 0, 1, 2...) or after the (t-1) th video segment is downloaded, the resolution corresponding to the t th video segment can be selected. and a playlist of video streams may be received based on the selected resolution. Before the video segment corresponding to the selected resolution is downloaded, data about the neural network corresponding to the selected resolution may be downloaded and decoded.

A neural network decoding operation according to an embodiment may include an operation of reconstructing or reconstructing the t-th compressed neural network through Huffman decoding and dequantization as shown in Equation 3 below.

에 기초하여, 다운로드된 비디오 세그먼트는 초해상화될 수 있다. 복원된 CNN에서 다운로드된 비디오 세그먼트에 포함된 저해상도의 프레임들에 대응하는 고해상도의 초해상화 프레임들이 출력될 수 있으며, 초해상화 프레임들은 t번째 세그먼트의 재생을 대기하는 프레임 버퍼에 저장될 수 있다.After the tth CNN is reconstructed and the video segment is downloaded, the parameters of the reconstructed CNN

Based on , the downloaded video segment can be super-resolution. High-resolution super-resolution frames corresponding to the low-resolution frames included in the video segment downloaded from the reconstructed CNN may be output, and the super-resolution frames may be stored in a frame buffer waiting for playback of the t-th segment. .

A client according to an embodiment can output high-resolution video according to super-resolution without additional buffering, and as a result, can receive a real-time video streaming service with improved QoE.

10 is a flowchart of a method of operating a client according to an embodiment.

A client according to an embodiment is a client receiving a live streaming service from a server in a live streaming system, and may correspond to, for example, the client 102 of FIG. 1 or the client 202 of FIG. 2 .

Referring to FIG. 10, a method of operating a client according to an embodiment includes receiving a playlist of a video stream corresponding to a selected first resolution from a server for live streaming (1010), based on the playlist, Receiving (1020) data about a segment of a first resolution corresponding to a video stream and data about a neural network from It may include restoring a neural network corresponding to the resolution (1030), and converting segments of the first resolution into segments of a second resolution based on the reconstructed neural network corresponding to the first resolution (1040). . For example, the first resolution may include a resolution lower than the second resolution determined based on at least one of a selection input received from a user and a communication environment of a network through which a video stream is received.

A method of operating a client according to an embodiment may correspond to the operations performed by the client described above with reference to FIG. 9 or the operations performed by the client 202 shown in FIG. 2 . As an example, steps 1010 and 1020 may correspond to the download operation shown in FIG. 9, and the ABR controller module 204 shown in FIG. 2 selects an appropriate resolution (r) according to the network bandwidth. It may include an operation of downloading the video segment file 242 of the operation and the selected resolution (r) and the data 244 related to the parameters of the compressed neural network corresponding to the resolution (r) through the network.

)로 복원하는 동작에 대응될 수 있다. 보다 구체적으로, 단계(1030)는 서버로부터 획득된 제1 해상도에 대응하는 신경망의 파라미터에 관한 데이터를 디코딩함으로써, 제1 해상도에 대응하는 신경망의 잔차 값을 획득하는 단계 및 획득된 잔차 값을 사전 학습된 신경망의 파라미터에 더함으로써, 제1 해상도에 대응하는 신경망의 파라미터를 복원하는 단계를 포함할 수 있다. 상술한 바와 같이, 제1 해상도에 대응하는 신경망의 파라미터는 라이브 스트리밍 서비스를 제공하는 서버에서, 미리 정해진 복수의 해상도들 중 제1 해상도의 세그먼트에 기초하여, 복수의 해상도들 중 제1 해상도보다 높은 해상도의 세그먼트에 기초하여 기 학습된 신경망 또는 사전 학습된 신경망의 파라미터를 갱신함으로써 획득될 수 있다.As an example, step 1030 may correspond to the decoding operation of the neural network shown in FIG. 9, and the data 244 on the parameters of the neural network compressed by the restoration step 250 shown in FIG. 2 may be pretrained. The neural network parameter (N _pre ) corresponding to the resolution (r) based on the neural network parameter (N pre ).

) may correspond to an operation of restoring. More specifically, step 1030 includes obtaining a residual value of the neural network corresponding to the first resolution by decoding data about parameters of the neural network corresponding to the first resolution obtained from the server, and pre-preparing the obtained residual value. and restoring parameters of the neural network corresponding to the first resolution by adding them to the learned parameters of the neural network. As described above, the parameter of the neural network corresponding to the first resolution is higher than the first resolution among the plurality of resolutions based on the segment of the first resolution among the plurality of predetermined resolutions in the server providing the live streaming service. It may be obtained by updating parameters of a pre-trained neural network or a pre-trained neural network based on a segment of resolution.

)에 의해 초해상화 프레임(들)(261)로 변환하는 동작에 대응될 수 있다. For example, step 1040 may correspond to the super-resolution operation of FIG. 9, and in step 260 of super-resolution shown in FIG. parameter(

) may correspond to an operation of converting to super-resolution frame(s) 261.

The operating method of the client according to an embodiment may further include playing frames of the segment of the second resolution converted after step 1040, and playing the frames of the segment of the converted second resolution may include: It may correspond to the regeneration operation described above through FIG. 9 .

11 is an exemplary view of a configuration of a device according to an embodiment.

Referring to FIG. 11 , an apparatus 1100 may include a processor 1101 , a memory 1103 and a communication module 1105 . Apparatus 1100 according to an embodiment may include terminals of a server and a client of the live streaming system described above through FIGS. 1 to 10 . The terminal of the client is a device for playing media files, and may include, for example, a mobile phone, a personal computer, and a tablet PC.

The processor 1101 according to an embodiment may perform at least one operation of the server or client described above through FIGS. 1 to 10 .

For example, the processor 1101 of the server divides the video stream for live streaming into segments of a predetermined length corresponding to each of a plurality of resolutions, and corresponding to each of the plurality of resolutions, a resolution higher than the corresponding resolution. Updating the previously learned parameters of the neural network based on the segment based on the segment of the corresponding resolution, compressing and storing data on the parameters of the neural network corresponding to each of a plurality of resolutions, and requesting live streaming of the video stream An operation of transmitting data and segments related to parameters of the compressed neural network corresponding to at least one of a plurality of resolutions to the client may be performed.

As another example, the processor 1101 of the terminal of the client receives a playlist of a video stream corresponding to the selected first resolution from a server for live streaming, based on the playlist, a video stream corresponding to the video stream from the server. An operation of receiving segments of the first resolution and data related to the neural network, and an operation of restoring the neural network corresponding to the first resolution based on the data related to the neural network corresponding to the first resolution and the parameters of the pre-loaded pretrained neural network. and converting segments of the first resolution into segments of the second resolution based on the neural network corresponding to the reconstructed first resolution.

The memory 1103 according to an embodiment may be a volatile memory or a non-volatile memory, and may store data related to the live streaming service described above through FIGS. 1 to 10 .

For example, the memory 1103 of the server may store data generated in the course of performing a method of operating a server for providing a live streaming service or data necessary for performing a method of operating a server for providing a live streaming service. . For example, the memory 1103 may store a segment file corresponding to each resolution of the received video stream and may store parameters of a trained neural network corresponding to each resolution.

As another example, the memory 1103 of the client's terminal stores data generated in the process of performing a client operating method for receiving a live streaming service or data necessary for performing a client operating method for receiving a live streaming service. can be saved For example, the memory 1103 may store pre-trained neural network parameters, segment files corresponding to a specific resolution obtained from a server, and compressed parameters of the neural network.

The communication module 1105 according to an embodiment may provide a function for the device 1100 to communicate with other electronic devices or other servers through a network. In other words, the device 1100 may be connected to an external device (eg, a user's terminal, server, or network) through the communication module 1105 and exchange data. For example, the server may transmit/receive data with the terminal of the client and the image capture device through the communication module 1105 . Also, for example, the terminal of the client may transmit/receive data with the server through the communication module 1105, and may receive data input from the user.

According to an embodiment, the memory 1103 may store a program in which the targeting advertisement method described above through FIGS. 1 to 10 is implemented. The processor 1101 may execute a program stored in the memory 1103 and control the device 1100 . Program codes executed by the processor 1101 may be stored in the memory 1103 .

The device 1100 according to an embodiment may further include other components not shown. For example, the device 1100 may further include an input/output interface including an input device and an output device as means for interfacing with the communication module 1105 . Also, for example, the device 1100 may further include other components such as a transceiver, various sensors, and a database.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. may be permanently or temporarily embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. A computer readable medium may store program instructions, data files, data structures, etc. alone or in combination, and program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

Separating a video stream for live streaming into segments of a predetermined length corresponding to each of a plurality of resolutions;
Corresponding to each of the plurality of resolutions, updating a parameter of a neural network pre-learned based on a segment having a higher resolution than the corresponding resolution based on the segment having the corresponding resolution;
compressing and storing data about parameters of a neural network corresponding to each of the plurality of resolutions; and
Transmitting data and segments related to parameters of a compressed neural network corresponding to at least one of the plurality of resolutions to a client requesting live streaming of the video stream.
including,
How the server works.
According to claim 1,
The step of updating based on the segment of the corresponding resolution
obtaining a neural network corresponding to the first resolution by updating a parameter of a pretrained neural network based on a segment of a first resolution among the plurality of resolutions; and
Acquiring a neural network corresponding to the second resolution by updating a parameter of the neural network corresponding to the first resolution based on a segment of a second resolution among the plurality of resolutions.
including,
The first resolution includes a higher resolution than the second resolution,
How the server works.
According to claim 1,
Down sampling a segment corresponding to each of the plurality of resolutions based on at least one of interpolation and inverse pixel shuffle of the segment corresponding to each of the plurality of resolutions.
Including more,
How the server works.
According to claim 1,
The step of compressing and storing
quantizing a difference between a parameter of a neural network corresponding to each of the plurality of resolutions and a parameter of a pretrained neural network; and
storing the encoded data of the quantized value corresponding to each of the plurality of resolutions;
including,
How the server works.
According to claim 1,
The step of updating based on the segment of the corresponding resolution
updating a parameter of a first neural network corresponding to a resolution higher than the corresponding resolution by repeatedly performing learning iterations based on frames included in the segment of the corresponding resolution; and
Acquiring a second neural network corresponding to the corresponding resolution including the updated parameter of the first neural network based on the end of the iteration of the learning iteration;
including,
The learning iteration is
A gradient obtained based on a loss function for a difference between an output frame obtained by applying a frame of a segment corresponding to the corresponding resolution to the first neural network and the correct answer data is back-propagated to the first neural network. stage of propagation
including,
How the server works.
According to claim 5,
Updating the parameters of the first neural network
loading frames included in the segment of the corresponding resolution; and
repeating the learning iteration based on the loaded frames;
including,
How the server works.
According to claim 5,
The repetition of the learning iteration is performed during the duration of the segment of the corresponding resolution,
How the server works.
According to claim 1,
The step of updating based on the segment of the corresponding resolution
fixing a value of a parameter of the neural network other than a weight included in a residual block of the neural network; and
Updating a weight included in the residual block of the neural network
including,
How the server works.
Receiving a playlist of video streams corresponding to a selected first resolution from a server for live streaming;
receiving, from the server, data about a segment of the first resolution corresponding to the video stream and a neural network corresponding to the first resolution, based on the playlist;
Restoring a neural network corresponding to the first resolution based on data related to the neural network corresponding to the first resolution and parameters of a pre-trained neural network loaded in advance; and
converting a segment of the first resolution into a segment of a second resolution based on a neural network corresponding to the reconstructed first resolution;
including,
How the client behaves.
According to claim 9,
Restoring the neural network corresponding to the first resolution
obtaining a residual value of the neural network corresponding to the first resolution by decoding data about parameters of the neural network corresponding to the first resolution obtained from the server; and
restoring a parameter of the neural network corresponding to the first resolution by adding the obtained residual value to the parameter of the pretrained neural network;
including,
How the client behaves.
According to claim 9,
The first resolution is
Including a resolution lower than the second resolution determined based on at least one of a selection input received from a user and a communication environment of a network in which the video stream is received.
How the client behaves.
According to claim 9,
The parameter of the neural network corresponding to the first resolution is
In the server, based on a segment of a first resolution among a plurality of predetermined resolutions, a pre-learned neural network or a parameter of the pre-trained neural network based on a segment of a resolution higher than the first resolution among the plurality of resolutions Obtained by updating
How the client behaves.
According to claim 9,
Reproducing frames of the converted second resolution segment
Including more,
How the client behaves.
A computer program stored in a medium to execute the method of any one of claims 1 to 13 in combination with hardware.
Dividing a video stream for live streaming into segments of a predetermined length corresponding to each of a plurality of resolutions;
Corresponding to each of the plurality of resolutions, updating a parameter of a neural network pre-learned based on a segment having a higher resolution than the corresponding resolution based on the segment having the corresponding resolution;
Compressing data on parameters of a neural network corresponding to each of the plurality of resolutions;
Transmitting data and segments related to parameters of a compressed neural network corresponding to at least one of the plurality of resolutions to a client requesting live streaming of the video stream;
at least one processor
including,
server.
According to claim 15,
the processor,
In updating based on the segment of the corresponding resolution,
Obtaining a neural network corresponding to the first resolution by updating a parameter of a pretrained neural network based on a segment of a first resolution among the plurality of resolutions;
Obtaining a neural network corresponding to the second resolution by updating a parameter of the neural network corresponding to the first resolution based on a segment of a second resolution among the plurality of resolutions;
The first resolution includes a higher resolution than the second resolution,
server.
According to claim 15,
the processor,
Down sampling a segment corresponding to each of the plurality of resolutions based on at least one of interpolation and inverse pixel shuffle of the segment corresponding to each of the plurality of resolutions,
server.
According to claim 15,
the processor,
In the compression and storage,
Quantizing a difference between a parameter of a neural network corresponding to each of the plurality of resolutions and a parameter of a pretrained neural network,
Storing encoding data of the quantized value corresponding to each of the plurality of resolutions,
server.
According to claim 15,
the processor,
In updating based on the segment of the corresponding resolution,
Updating a parameter of a first neural network corresponding to a resolution higher than the corresponding resolution by repeatedly performing learning iterations based on frames included in the segment of the corresponding resolution;
Obtaining a second neural network corresponding to the corresponding resolution including the updated parameters of the first neural network based on the end of the iteration of the learning iteration;
The learning iteration is
A gradient obtained based on a loss function for a difference between an output frame obtained by applying a frame of a segment corresponding to the corresponding resolution to the first neural network and the correct answer data is back-propagated to the first neural network. stage of propagation
including,
server.
Receiving a playlist of video streams corresponding to a selected first resolution from a server for live streaming;
Based on the playlist, receiving data about a segment of the first resolution corresponding to the video stream and a neural network corresponding to the first resolution from the server;
Restoring a neural network corresponding to the first resolution based on data related to the neural network corresponding to the first resolution and parameters of a pre-loaded pretrained neural network;
Converting segments of the first resolution into segments of a second resolution based on a neural network corresponding to the reconstructed first resolution;
at least one processor
including,
client terminal.

Images