KR102130077B1 - System to improve the resolution based on pattern information of grid generation - Google Patents

Abstract

An objective of the present invention is to process image data in real-time without occupying a large amount of memory. According to an embodiment of the present invention, a system for improving resolution based on grid generation pattern information comprises: a server to perform a service for transmitting requested video data to a user device in response to a request of the user device side, generate a universal neural network file required for an artificial neural network algorithm operation for improving the resolution of image information based on possessed video data, and transmit the generated universal neural network file and low-quality video data with resolution changed below a preset level to the user device; and a user device to perform a calculation operation on low-quality video data received from the server based on an artificial neural network algorithm using the received universal neural network file, improve the resolution of the low-quality video data in accordance with the calculation operation, and play the video data with the improved resolution.

Description

System to improve the resolution based on pattern information of grid generation

The present invention relates to a system for improving resolution based on grid generation pattern information. More specifically, the present invention relates to a system for extracting information on a grid generated when a size of an image is reduced, removing a grid of low quality image data based on the information, and restoring the resolution to high-quality image data.

In order to improve the resolution of image data, various approaches such as interpolation techniques and machine learning techniques have been used. Recently, research using deep learning techniques has been actively conducted in the field of pattern recognition such as image recognition/analysis. In addition, it has been proved that the technique using deep learning is superior to the learning-based machine learning technique in the upscaling technique, which is one of the fields of image quality improvement.

An object of the present invention is to process image data in real time without taking up a large amount of memory.

An object of the present invention is to make it possible to easily improve resolution regardless of content type through a deep learning based general purpose neural network file.

A system for improving resolution based on grid generation pattern information according to an embodiment of the present invention performs a service for transmitting requested video data to a user device in response to a request from a user device, but an image based on the retained video data A server that generates a general purpose neural network file required for operation of an artificial neural network algorithm for improving the resolution of information, and transmits the generated general purpose neural network file and low-definition video data in which the resolution is changed to a predetermined level or less to the user device; And performing an operation operation based on an artificial neural network algorithm applying the received general-purpose neural network file to the low-quality video data received from the server, improving the resolution of the low-definition video data according to the operation, and improving the resolution of the video. And user equipment for reproducing data.

According to an embodiment of the present invention, even when the image quality data is transmitted from the side that provides the image data, when reproducing the image, the resolution can be improved based on the neural network file, so that data is transmitted and received with low quality data instead of high quality data. Can provide an environmental foundation. Accordingly, the present invention has an effect of solving the problem of a decrease in transmission speed that occurs when transmitting high-capacity high-definition image data.

In addition, since the present invention enables to convert high-quality images through neural network files at the time of video execution even if only low-quality images are retained, it has an effect of occupying relatively little memory capacity.

1 is a view showing a resolution improvement system according to an embodiment of the present invention.
2 is a diagram illustrating an example of an image quality improvement operation according to an embodiment of the present invention.
3 is a block diagram showing the configuration of a server according to an embodiment of the present invention.
4 is a view showing the configuration of a data processing unit according to an embodiment of the present invention.
5 is a view showing the configuration of a neural network learning unit according to an embodiment of the present invention.
6 is a diagram illustrating an example of a size change operation performed by a size change unit according to an embodiment of the present invention.
7 is a view for explaining an example of a deep learning learning operation according to an embodiment of the present invention.
8 is a diagram showing the configuration of a user equipment according to an embodiment of the present invention.
9 is a diagram illustrating a process of generating and transmitting a neural network file for improving image quality according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

When a component is said to be'connected' or'connected' to another component, it is understood that other components may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be'directly connected' or'directly connected' to another component, it should be understood that no other component exists in the middle.

The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as'include' or'have' are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

1 is a view showing a resolution improvement system according to an embodiment of the present invention.

As illustrated in FIG. 1, a system for improving resolution according to an embodiment of the present invention may include a server 100 and a user device 200.

According to an embodiment, the server 100 may include a server that provides a VOD service to the user device 200.

The server 100 may transmit video data to provide a VOD service to the user device 200. In addition, according to an embodiment of the present invention, the server 100 may calculate a neural network file required for improving the image quality of video data and transmit it to the user device 200 together. Accordingly, the user device 200 can improve the image quality based on the neural network file receiving the video data provided from the server 100.

In addition, the user device 200 may select video data to be transmitted to the server 200 and request transmission. In addition, the user device 200 may calculate user viewing pattern information calculated based on selection information and playback information of its own video data and transmit it to the server 200.

Reference will be made to FIG. 2 to briefly describe an image quality improvement operation performed in the user device 200.

2 is a diagram illustrating an example of an image quality improvement operation according to an embodiment of the present invention.

As illustrated in FIG. 2, the user device 200 may generate a video file with improved resolution through a neural network file. At this time, according to an embodiment of the present invention, the neural network file is a general-purpose file that can be used regardless of the type of video file. Accordingly, the neural network file can be combined with any video file transmitted to the user device 200 to improve resolution.

In addition, according to various embodiments, the user device 200 may mount the general-purpose neural network file as embedded software, and receive a video file, which is a target for image quality improvement, from the server 100 (eg, a video streaming server).

3 is a block diagram showing the configuration of a server according to an embodiment of the present invention.

As illustrated in FIG. 3, the server 100 according to an embodiment of the present invention may include a communication unit 110, a storage unit 120, and a control unit 130. In addition, the control unit 130 may include a data processing unit 131, a neural network learning unit 132, a result evaluation unit 133, and a usage pattern calculation unit 134.

First, the communication unit 110 may use a network to transmit and receive data between a user device and a server, and the type of the network is not particularly limited. The network may be, for example, an Internet Protocol (IP) network that provides a large data transmission/reception service through an Internet Protocol (IP) or an All IP network that integrates different IP networks. In addition, the network is a wired network, a Wibro (Wireless Broadband) network, a mobile communication network including WCDMA, a High Speed Downlink Packet Access (HSDPA) network and a mobile communication network including a Long Term Evolution (LTE) network, LTE advanced (LTE-A) ), 5G (Five Generation), one of the mobile communication network, satellite communication network and Wi-Fi (Wi-Fi) network, or may be made by combining at least one of them.

The communication unit 110 according to an embodiment of the present invention may perform data communication with an external web server and a plurality of user devices. For example, the communication unit 110 may receive content data (pictures and videos) including images from other web servers or user devices (including administrator devices). In addition, as the server 100 includes a server that provides a VOD service, VOD content corresponding to a user request may be transmitted to the user device 200.

According to various embodiments, the communication unit 110 may receive and transmit a VOD file for a VOD service, but is not limited thereto, and the communication unit 110 collects training data required to generate a neural network file for resolution improvement To perform the communication function.

The neural network file may contain information necessary to restore the resolution of the damaged image data similar to the original data through the artificial neural network algorithm, and may include information on various parameters to be selected when the artificial neural network algorithm is driven. have.

The storage unit 120 may include, for example, an internal memory or an external memory. The internal memory includes, for example, volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (eg, OTPROM (one time programmable ROM (PROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash, etc.), hard drives, Or it may include at least one of a solid state drive (SSD).

The external memory may be a flash drive, such as compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (SD-SD), extreme digital (XD), It may further include a multi-media card (MMC) or a memory stick. The external memory may be functionally and/or physically connected to the electronic device through various interfaces.

The storage unit 120 according to an embodiment of the present invention is a processing data (original image data) reduced by processing a certain amount of image data (eg, photo and video data) received from a user device (administrator device) or another web server. Data, or data enlarged to the original data size after reduction) and corresponding original data may be matched and stored. The original data and the processed data can be used to extract information on the lattice phenomenon that occurs when the resolution decreases, respectively.

In addition, the storage unit 120 may store a neural network file that is data for restoring the resolution to the original image level by removing the grid existing in the processed data through an artificial intelligence algorithm (eg, SRCNN) after extracting information on the grid phenomenon.

The controller 130 may also be referred to as a processor, a controller, a microcontroller, a microprocessor, or a microcomputer. Meanwhile, the control unit may be implemented by hardware or firmware, software, or a combination thereof.

In the case of implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code can be stored in the memory and driven by the control unit. The memory may be located inside or outside the user terminal and the server, and may exchange data with the control unit by various means already known.

The controller 130 according to an embodiment of the present invention may generate a general purpose neural network file, which is a file required to improve the resolution of image data through artificial neural network-based calculation.

The control unit 130 may include a data processing unit 131, a neural network learning unit 132, a result evaluation unit 133, and a usage pattern calculation unit 134.

First, the data processing unit 131 may collect and process learning data necessary for calculating a general purpose neural network file required for improving the quality of video data. The data processing unit 131 may perform a primary change (reduction change) and a secondary change (re-enlarge the reduced image data) of the collected data. More specific details of the operation performed by the data processing unit 131 will be described with reference to FIG. 4.

The neural network learning unit 132 may perform a neural network learning operation based on artificial intelligence based on the processed data calculated after collecting and processing the data performed by the data processing unit 131. The neural network learning unit 132 may perform a parameter setting operation and a neural network file calculation operation required for the learning process. A more detailed description of the neural network learning unit 132 will be described with reference to FIG. 5.

The result evaluation unit 133 may perform an operation of evaluating the result value applied to the general purpose neural network file calculated by the neural network learning unit 132 in the user device 200.

Specifically, the result evaluation unit 133 may determine a degree of improvement in the resolution of data as a result of applying a neural network file in the user device 200. In addition, the result evaluation unit 133 may determine an error rate between the result data and the original data after applying the neural network file. In this case, the comparison unit of the result data and the original data may be each frame constituting the image, or may be a fragment unit divided for transmission of the image.

Alternatively, according to various embodiments, each image data may be set of a plurality of frames based on image identity (eg, when an image is displayed over 100 frames, the 100 frames may be set as a single frame bundle) It may also be divided into multiple). Accordingly, the comparison unit of the result data and the original data may be a unit of division divided based on image identity.

Furthermore, when it is determined that the error rate between the result data and the original data is greater than or equal to a reference value, the result evaluation unit 133 may request correction of a weight, bias value, etc. constituting a neural network file. That is, the result evaluation unit 133 may determine whether to modify the parameters constituting the neural network file by comparing the original data and the result data.

According to an embodiment, the result evaluation unit 133 may calculate the importance of each image object required to understand an image from original data. In addition, the error rate of 1 unit (for example, 1 frame, 1 frame bundle, etc.) of the original data and the result data (data improved by applying a neural network file from the user device 200) is equal to or greater than a preset value, and is applied to the 1 unit. When it is determined that an image object having a value greater than or equal to a preset value is included, it is possible to request correction of the weight and bias values constituting the neural network file.

The importance of each image object may be calculated based on a size ratio occupied by one image object in one frame and a repetition ratio of the image object.

According to various embodiments, the result evaluation unit 133 may calculate the importance of each image object based on the content characteristics of the video data according to various embodiments. The result evaluation unit 133 may first check the content characteristics of the video data, and at this time, the content characteristics of the private video data may be calculated by the data processing unit 131. For example, the content characteristics of the video data include information on the path where the video data was uploaded to the server 100 (eg, the folder name selected when the user or the administrator uploaded the video file to the server 100), the user or the administrator When the video file is uploaded to the server 100, it can be calculated based on the content genre, field, etc. entered at the time. In addition, the content characteristics of the calculated video data may be managed as metadata of the corresponding video data.

Accordingly, the result evaluation unit 133 can check the content characteristic information of each video data extracted and stored at the time of uploading to the server 100, thereby calculating the importance of each image object based on the content characteristic information. For example, the result evaluation unit 133 may classify items of image objects into facial objects, text objects (eg, subtitles), object objects, and the like, and each object item (eg, drama content) that matches content characteristic information. Correspondingly, it is possible to determine a face object of a person can be matched).

The usage pattern calculation unit 134 may calculate a usage pattern for a user's VOD service based on a user's streaming request record, download record, and user information calculated and transmitted from the user device 200.

The usage pattern calculator 134 may calculate items such as a user's preferred genre, preferred content characteristics, main streaming request time zone, and main viewing device as a usage pattern. The usage pattern calculator 134 may recommend a screen mode suitable for the user based on the calculated usage pattern information of the user. According to various embodiments, a neural network file for image quality improvement may be used universally, but according to a screen mode (eg, a movie size mode, a subtitle-oriented mode, a specific color (RGB) enhancement mode, a person-oriented resolution improvement mode, etc.) It can also be calculated as a separate file. That is, the user may perform resolution improvement by downloading one general purpose neural network file suitable for his use pattern among a plurality of neural network files for each screen mode available from the server 100 to the user device 200. To this end, the usage pattern calculation unit 134 may calculate usage patterns for each user and separately manage information on the usage patterns. Thereafter, when the user requests streaming, a neural network file in a mode suitable for the user's usage pattern may be generated and provided.

According to various embodiments, a general purpose neural network file suitable for a user's usage pattern may be automatically set based on the user's usage pattern information calculated by the usage pattern calculator 134 of the server 100, or based on a screen mode selected by the user. It may be set.

4 is a view showing the configuration of a data processing unit according to an embodiment of the present invention.

As illustrated in FIG. 4, the data processing unit 131 according to an embodiment of the present invention may include a size changing unit 131a, an image division unit 131b, and a feature region extraction unit 131c.

In order to calculate a neural network file for improving the image quality of video data, the input data to be input to the input layer or feature values of the data to be input must be prepared in the neural network learning unit, and the data to be input to the input layer in the data processing unit 131 and Data can be prepared.

First, the size change unit 131a performs a first change operation to reduce the size of the image constituting the video data by a predetermined value from the original size, and a second change operation to enlarge the result image of the first change corresponding to the original size. can do. The size change operation performed by the size change unit 131a will be described with reference to FIG. 6.

6 is a diagram illustrating an example of a size change operation performed by a size change unit according to an embodiment of the present invention.

As shown in FIG. 6, the size changing unit 131a may perform the operation a, which is the first change operation for reducing the original image 605 at a predetermined ratio, and the reduced image 610 calculated as a result of the operation a same size as the original image The second change operation, which is enlarged to, may be performed. The image 615 generated after the processing operation (1st change operation (a) and 2nd change operation (b)) has a lower resolution than the original image 605, which means that the pixels constituting the image are enlarged only in size without increasing the number. It is due to what has become.

When comparing the image 615 (the same resolution as that of the image 610) with the original image 605, it can be seen that the image 615 has an increased pixel size, and thus a mosaic-like grid is generated.

The server 100 according to an embodiment of the present invention may perform neural network learning based on the processed image 615 and the original image 605 to convert the resolution level from the low-quality image 615 to the original image 605. To this end, the size changing unit 131a of the data processing unit 131 reduces the size of the original image by a predetermined value and a second change that enlarges the reduced image generated by the first change operation to the same size as the original image. The change operation can be performed. In addition, the data processing unit 131 may perform an operation of extracting the original image and the processed image generated by the first change operation and the second change operation as learning data.

Before performing neural network learning according to various embodiments, the data processing unit 131 extracts pattern information (location information, color information, etc.) of the grid generated on the processed image (the image enlarged after reduction in size) and learns the data about the grid information. It can be used as input data for.

The image division unit 131b may perform an operation of dividing video data held by the server 100 based on a preset condition. In this case, the image division unit 131b may perform an operation of dividing video data based on the number of frames. Alternatively, the image splitter 131b may divide video data into bundles (chunks) by bundling frames having a matching rate higher than or equal to a preset value based on a matching rate of an image object. For example, in the division unit, when the same person is photographed and the image division unit 131b provides a streaming service from the server 100 to the user device 200, video data may be divided into a plurality of chunks for each unit delivered to the user device 200.

The chunks segmented by the image segmentation unit 131b may be used when evaluating AI neural network learning and resolution enhancement results.

The feature region extracting unit 131c may perform an operation of extracting a feature region including a feature image based on each frame or division unit of video data. The feature region extracting unit 131c may determine whether an image region having a predetermined feature region requirement exists in each frame or division unit. The feature area requirement may include, for example, an area in which an image object judged to have an importance level equal to or greater than a preset value. For example, the feature region extracting unit 131c may set a high image object importance for the face image of the main character in the drama content, and accordingly, the feature region may be a region in which the face image of the main character is displayed (eg, quilted region; background and Object display area).

The feature region extracting unit 131c may perform an operation of extracting not only a feature region in an image, but also a specific frame or a specific division unit among all frames or images of video data.

For the feature region extracted by the feature region extraction unit 131c, a learning importance weight may be set to increase the number of learning iterations.

5 is a view showing the configuration of a neural network learning unit according to an embodiment of the present invention.

As illustrated in FIG. 5, the neural network learning unit 132 according to an embodiment of the present invention may include a parameter setting unit 132a and a neural network file calculation unit 132b.

In addition, the neural network learning unit 132 may perform a deep learning learning process based on an artificial neural network, and accordingly may generate a neural network file, which is a file required for improving the quality of video data having low resolution.

Referring to FIG. 7 for a brief description of the deep learning learning operation based on the artificial neural network.

7 is a view for explaining an example of a deep learning learning operation according to an embodiment of the present invention.

Referring to FIG. 7, a perceptron that is a neural network model including an input layer, a hidden layer, and an output layer is disclosed. According to an embodiment, the neural network learning disclosed in the present invention may be performed using a multilayer perceptron implemented to include at least one hidden layer. Basically, the perceptron can receive a plurality of signals and output one signal.

The weight and bias required in the calculation process using the artificial neural network model can be calculated as an appropriate value through a back propagation technique. The artificial neural network learning process according to an embodiment of the present invention extracts appropriate weight data and bias data through such a back propagation technique. In the present invention, the neural network file calculated through the artificial neural network in order to perform resolution improvement may include information on extracted appropriate weight data and bias data.

Since the learning method and parameter modification through the artificial neural network performed through the back propagation technique are known techniques, detailed description thereof will be omitted.

Preferably, the neural network learning unit 132 according to an embodiment of the present invention may perform learning using a convolution neural network (CNN) model among artificial neural network models. In the case of CNN, there are features such as maintaining the shape of input/output data of each layer, effectively recognizing features with adjacent images while maintaining spatial information of the image, and extracting and learning features of images with multiple filters. .

The basic operation method of CNN may be a method of learning an image by scanning a portion of a region one by one through a filter and finding a value for it. At this time, the goal of CNN is to find a filter with an appropriate weight value. In CNN, filters can be generally defined as square matrices such as (4,4) or (3,3). The set value of the filter for CNN according to an embodiment of the present invention is not limited. The filter can compute the composite product while traversing the input data at a specified interval.

The parameter setting unit 132a according to an embodiment of the present invention may set initial parameter values for performing a learning process for image data through the CNN.

According to various embodiments of the present disclosure, the parameter setting unit 132a may determine a frame size of the original data, a reduction ratio set in the original data when generating the processed data, and set initial parameters corresponding thereto.

Also, according to various embodiments, the parameter setting unit 132a may specify a type of image data required for learning an artificial neural network, and may request that the corresponding image data be input as training data. For example, the parameter setting unit 132a may determine whether a learning importance weight for each genre is set, and may further request an image in which the learning importance weight is set before the corresponding learning operation is completed. Specifically, in the case of content having a high proportion of people, such as a drama or a movie, the parameter setting unit 132a may request frame information including an associated image object as learning data in order to perform repetitive learning on a main character.

The neural network file calculation unit 132b may perform an operation of inputting and learning data processed by the data processing unit 131 to a preset artificial neural network model. At this time, the neural network file calculating unit 132b inputs original data and processed (reduced to a predetermined ratio) data into the CNN algorithm to obtain information about the grid generated in the process of changing from original data to processed data (lattice generation pattern information). Can be extracted. More specifically, the grid generation pattern information calculated by the neural network file calculation unit 132b may be calculated based on a difference between original data and processing data, and may include pattern information on a grid position and color change of the grid. have.

The neural network file calculator 132b may generate a neural network file required to restore an image to original data by erasing the grid from the processed data based on the calculated grid generation pattern information. The neural network file calculation unit 132b inputs low-quality video (processed data) data as input data to the artificial neural network algorithm, and when the resolution of the output data shows a matching rate of more than a preset value with the original data, 1 data of You can end the learning process. In a similar manner, the neural network file calculator 132b may repeatedly perform an operation of inputting a myriad of different types of processed data into an input layer and determining a matching rate with original data as a result of artificial neural network calculation.

According to various embodiments of the present disclosure, the neural network file calculator 132b may calculate grid generation pattern information generated when an image of a specific size is reduced by inputting various types of original data and processing data. Accordingly, the neural network file calculator 132b may calculate grid generation pattern information that is commonly generated when an image is reduced in various images as well as in a specific image.

The neural network file calculating unit 132b inputs the processed data into the input layer and when the matching rate between the output data and the original data is greater than or equal to a preset value, parameters (weight, bias, learning rate) set in the corresponding artificial neural network algorithm Etc.) and layer-specific activation functions.

That is, when the neural network file calculated by the neural network file calculating unit 132b is delivered to the user device 200, the user device 200 may receive it and perform low-quality video data based on the information based on the neural network file, thereby performing an artificial neural network test. Accordingly, it is possible to perform a function of improving the resolution of video data.

8 is a diagram showing the configuration of a user equipment according to an embodiment of the present invention.

As illustrated in FIG. 8, the user device 200 according to an embodiment of the present invention may include a communication unit 210, a storage unit 220, an input unit 230, a display unit 240, a camera unit 250, and a control unit 260. In addition, the control unit 260 may include a video playback unit 261, a resolution conversion unit 262, and a user information collection unit 263.

The communication unit 210 according to an embodiment of the present invention may perform a communication function for receiving neural network files and video data from the server 100. Furthermore, the communication unit 210 may perform a communication operation for transmitting feedback information collected from the user device 200 to the server 100.

The storage unit 220 may store neural network files and video data received from the server 100 according to an embodiment of the present invention. According to various embodiments of the present disclosure, the storage unit 220 may store or temporarily store result data (data with improved resolution) which is a result of driving an artificial neural network algorithm operation by applying a neural network file to low-quality data having a resolution below a preset reference value. have.

The storage unit 220 may store the generated feedback information. Alternatively, the storage unit 220 may store information required to calculate feedback information. For example, when one frame is extracted to provide as feedback information among result data (resolution-enhanced data) generated as a result of computation of an artificial neural network algorithm, reference information for the extraction (eg, a user detected during video playback) It is possible to save whether the facial distortion is detected and the contents of extracting the frame at the time when the facial distortion is detected.

The input unit 230 may receive user selection information regarding a content genre, a content name, and the like.

When the video data received from the server 100 or the result data after the resolution improvement operation of the video data is played, the display unit 240 may display a playback screen of the video.

The camera unit 250 may take pictures and videos in response to a user request. The camera unit 250 may upload image information such as photographs and videos taken to the server 100 or other web servers. Alternatively, image information photographed through the camera unit 250 may be transmitted to another user device.

When taking image information such as a photo or video, the camera unit 250 may first determine a resolution based on a user request. According to an embodiment of the present disclosure, the camera unit 250 may lower the resolution of the photographed picture or video to a level below a predetermined level and store it based on the presence or absence of a general purpose neural network file for improving image quality.

According to various embodiments of the present disclosure, the camera unit 250 may operate a camera that regularly photographs the user's face according to a preset reference interval while reproducing data having improved resolution. It is possible to determine whether the user has a facial expression or a frown, and corresponding feedback information can be extracted.

The control unit 260 may perform resolution conversion and reproduction of a video file downloaded from the server 100. Specifically, the control unit 260 may include a video playback unit 261, a resolution conversion unit 262, and a user information collection unit 263.

First, the video playback unit 261 according to an embodiment of the present invention can be controlled to play a video file and display it on the display unit 240. The video playback unit 261 may determine the resolution of the video data requested to be output. When the resolution of the video data requested to be executed is determined to require a resolution improvement below a predetermined level, the video playback unit 261 may request the resolution conversion unit 262 to improve the resolution. Thereafter, the video playback unit 261 may play a file with improved resolution through the resolution conversion unit 262.

The resolution converter 262 may determine the resolution level of the current image data (picture and video) and the target resolution level requested by the user. At this time, the resolution converter 262 determines whether or not the neural network file is retained, and if the neural network file does not exist, requests the server 100 to download the general-purpose neural network file from the server 100. Thereafter, the resolution converter 262 may perform an operation of converting low-quality data into a desired level of resolution by driving an artificial neural network algorithm based on the general-purpose neural network file.

The user information collection unit 263 may collect user information for feedback. The user information collection unit 263 may select a frame to be used as feedback information among result data after resolution improvement is performed based on an artificial neural network algorithm and store it. For example, the user information collection unit 263 may acquire the user's facial information while the user plays the video data with improved resolution. The video frame information being displayed can be collected.

In addition, the user information collection unit 263 may collect content information such as items and genres of content reproduced above a reference value. For example, the user information collection unit 263 may determine a refresh rate of animation compared to a documentary (based on a photographic image) and a refresh rate of content without subtitles compared to content with subtitles, and collect information about the animation. The reproduction information collected by the user information collection unit 263 may be provided to the server 100, and the server 100 may calculate usage pattern information of the user based on this.

9 is a diagram illustrating a process of generating and transmitting a neural network file for improving image quality according to an embodiment of the present invention.

As illustrated in FIG. 9, the server 100 according to an embodiment of the present invention may generate a general purpose neural network file for resolution improvement and transmit it to the user device 200.

Specifically, first, the server 100 may perform operation 705 of processing the acquired video data. The operation 705 is an operation for generating data for training in an artificial neural network algorithm, and processing to reduce the image size of video data may be performed to generate data suitable for learning.

According to an embodiment, the operation 705 may be a processing operation (reduction in size) for each frame constituting a video file. Alternatively, the operation 705 may be an operation of selecting a frame to be input for artificial neural network learning through sampling for each unit of division of a video file, and then processing (reducing the size by multiplying a predetermined ratio) with respect to the corresponding frame. For example, in the case of a video file having a total number of frames of 2400, assuming that 24 chunks are composed of 100 chunks as one unit, the server 100 samples 1 frame per video division unit, and total 100 frames are used as training data. It can be processed.

After operation 705, the server 100 may perform operation 710 for obtaining grid generation pattern information based on the processed video data. The processed video data may refer to data obtained by reducing the size of the original data (designated as learning original data only for data having a predetermined resolution level or higher) at a preset ratio.

When the size of one image is reduced, the number of pixels to be displayed in the same area decreases, and the resolution automatically decreases. Accordingly, when the image size is reduced, even if the image size is enlarged again later, the reduced resolution level is maintained, and thus a grid phenomenon occurs.

The server 100 may obtain pattern information on which a grid is generated by comparing the original image with the processed image where the grid is generated. The obtained lattice generation pattern information may be used to restore resolution by removing the lattice from the image where the lattice phenomenon is generated later.

After operation 710 of acquiring grid generation pattern information, the server 100 may perform operation 715 of generating a neural network file for improving image quality based on the grid generation pattern information. The neural network file may be generated to be universally applicable regardless of the content type, and for this purpose, various types of images and content may be used for training. In addition, the server 100 may generate a neural network file by calculating artificial neural network algorithm information (activation function applied to each layer, weight, bias, etc.) required to remove the grid from the generated low-quality image data and restore the original image. .

Factors such as weight and bias provided as a result value may be determined based on a matching ratio between the final result (image quality improved data) and the original image data. When the coincidence rate is equal to or greater than a predetermined level, the server 100 may determine weight and bias information applied when calculating the artificial neural network as information to be included in the general purpose neural network file.

Thereafter, the server 100 may perform operation 720 to confirm that a streaming request (or download request) for video data is received from the user device 200. In response to a user request, the server 100 may perform operation 725 of transmitting a low quality version of the requested video data together with a neural network file for improving image quality generated in the user device 200. Accordingly, since the user device 200 receives the low quality version of the video, the content can be easily received without limitation by a relatively network environment, and the user through the general neural network file received together is applied to the received low quality version of video data. Is able to reproduce the high-definition video of the desired level.

Although not shown in the figure, according to various embodiments, the user device 200 may transmit feedback information on the state of video data that has been played back or converted to the server 100. Accordingly, the user device 200 may calculate playback-related information for each user, such as a content genre, a content characteristic, and a main playback request time, which are reproduced at a frequency equal to or greater than a reference value, and may be transmitted to the server 100.

Furthermore, the user device 200 may provide a frame sample of the result data after the resolution improvement operation is completed to the server 100 according to a preset period. Accordingly, the server 100 may compare the result data frame calculated after the resolution improvement operation received from the user device 200 and the original data frame of the same content. The transmitted frame information may include transmission location information in the content together, and accordingly, the server 100 may search for a frame image to be compared in the original data.

The server 100 may compare the image frame provided for feedback from the user device 200 with the original image frame of the corresponding content, and determine a matching rate. In addition, when it is determined that the matching rate is equal to or less than a preset reference value, a re-learning operation for updating a neural network file may be requested, and a re-learning operation may be performed accordingly.

Meanwhile, the neural network file generated according to various embodiments of the present invention may be compressed as necessary. In one embodiment, the server 100 may compress the neural network file in consideration of the performance of the user device 200, and transmit the compressed neural network file to the user device 200.

The neural network file may be compressed using at least one of Pruning, Quantization, Decomposition, and Knowledge Distillation. Pruning is one of compression techniques that delete weights and biases that have no significant effect on output values or weights and biases of neural network files. Quantization is one of compression techniques that quantizes each weight into predetermined bits. Decomposition is a compression technique that reduces the size of a weight by linearly decompositioning a weight matrix or tensor, which is a set of weights. Knowledge Distillation is one of the compression techniques that generate and train a student model smaller than the original model using the original model as the teacher model. At this time, the Student model may be generated through the above-described Pruning, Decomposition or Quantization.

At this time, the degree of compression according to the performance of the user device 200 may be determined through various methods. In one embodiment, the degree of compression of the neural network file may be determined based on a simple specification of the user device 200. That is, the degree of compression may be collectively determined by the processor specifications and memory specifications of the user device 200.

In another embodiment, the degree of compression of the neural network file may be determined based on the use state of the user device 200. Specifically, the server 100 may receive usage state information from the user device 200 and obtain available resource information of the user device 200 according to the received use state information. The server 100 may determine the degree of compression of the neural network file based on available resource information of the user device 200. In this case, the available resource information may be information related to an application in which the user device 200 is running, CPU or GPU share determined according to the running application, and information related to a memory capacity that can be stored in the user device 200.

Although the present invention has been described in detail with reference to the above-mentioned examples, those skilled in the art can make modifications, alterations, and modifications to the examples without departing from the scope of the present invention. In short, in order to achieve the intended effect of the present invention, it is not necessary to separately include all the functional blocks shown in the drawings or to follow all the orders shown in the drawings in the order shown, and even if not, any number of technical aspects of the present invention described in the claims Note that it can fall within the scope.

100: server
110: communication unit
120: storage unit
130: control unit
131: Data processing unit
132: neural network learning department
133: result evaluation unit
134: usage pattern calculation unit
131a: size change unit
131b: image division
131c: feature region extraction unit
132a: parameter setting unit
132b: Neural network file calculator

Claims (1)

In response to a request from the user device, a service for transmitting the requested video data to the user device is performed, and a general purpose neural network file required for the operation of the artificial neural network algorithm for improving the resolution of image information is generated based on the retained video data, A server that transmits the generated general-purpose neural network file and low-definition video data having a resolution lower than a predetermined level to the user device;
Perform an operation based on an artificial neural network algorithm applying the received general-purpose neural network file to the low-quality video data received from the server, improve the resolution of the low-definition video data according to the operation, and improve the resolution of the video data User equipment to play; and
The server
Includes a control unit for generating a general-purpose neural network file, which is a file required to improve the resolution of the image data through artificial neural network-based operation.
The control unit
A data processing unit that collects and processes learning data necessary to calculate a general purpose neural network file required for improving the quality of video data; And
Neural Network Learning Department;
The data processing section
A size changing unit for reducing the size of the image constituting the video data from the original size to a preset ratio to generate processed data in which grids are generated according to a resolution decrease;
An image segmentation unit for dividing frames in video data based on a match rate of an image object; And
Includes a feature region extractor for extracting a feature region for each segment of video data, but extracting a region in which the corresponding image object is displayed as a feature region when the importance of the image object included in the frame is greater than or equal to a predetermined value.
The neural network learning unit
A parameter setting unit for setting initial parameters required for the calculation of the artificial neural network algorithm; And
Comparing the original data and the processed data with a reduced resolution by reducing the size of the original data, calculates the generation pattern information of the grid generated in the process of reducing the size of the original data, and removes the grid on the processing data based on the grid generation pattern information. And a neural network file calculator for calculating neural network parameters required to generate neural network files including the calculated parameters.

Images