KR20210115710A - Image processing method, video playback method and apparatuses thereof - Google Patents

Abstract

The present invention relates to an image processing method, an image reproducing method and a device thereof. According to an embodiment of the present invention, an image processing method and the device are provided to receive a first image including a plurality of frames, acquire importance information indicating the importance of at least one area included in the plurality of frames, determine at least one mask indicating a scaling degree corresponding to at least one area of the first image based on the importance information, generate a second image by encoding the first image according to at least one mask, and output at least one mask and the second image. According to one embodiment of the present invention, it is possible to provide a high-quality video streaming service in a network environment having a low transmission capacity by providing streaming of an original streaming image at twice or lower resolution, and by providing an image reproducing device that maintains the original image through artificial neural network-based super-resolution-based restoration.

Description

Image processing method, image playback method, and apparatuses thereof

The following embodiments relate to an image processing method, an image reproducing method, and apparatuses thereof.

In order to provide streaming, a method based on a user's point of view, a method based on content, and a method based on a neural network may be used. The method based on the user's point of view is a method of streaming with high quality encoding only the area the user looks at, that is, the area corresponding to the user's point of view. In the method based on the user's viewpoint, when the user suddenly changes the viewpoint, latency of image quality change may occur. In addition, when multi-encoding of one content is differently performed for each viewpoint in a method based on a user's viewpoint, image capacity and calculation overload may occur. In addition, a method based on a neural network does not always guarantee high quality when a single network is used. In addition, when a content-based neural network is used, it takes about tens of hours for modeling, so it is not suitable for live streaming, and there is a problem that a high GPU (Graphic Processing Unit) specification is required for high-definition streaming.

According to an embodiment, the image processing device provides streaming of the original streaming image at a resolution of 2 times or more, and the image reproducing device maintains the original image through super-resolution based restoration based on an artificial neural network. It is possible to provide a high-quality video streaming service in a network environment having a low transmission capacity.

According to an embodiment, the original resolution of the important region of each frame of the original image is maintained as much as possible, and the remaining region except for the important region is down-sampled to reduce the capacity of the image for the live streaming service.

According to an embodiment, it is possible to reduce the cost of using a HTTP Live Streaming (HLS) server by performing streaming at a capacity four times or more lower in the server for streaming.

According to an embodiment, it is possible to reduce data transmission cost by transmitting and receiving image data with a low capacity through mask-based scaling.

According to an embodiment, an area that is not well restored in the original image by super-resolution (SR) technology that restores a low-resolution image to a high-resolution image based on an artificial neural network maintains the original resolution as much as possible, and the rest of the restoration is performed. By transmitting the resolution of the area that works well through downscaling, it is possible to preserve the image quality of the important area in the original image or the area where the quality of the ultra-high resolution is poor during restoration.

According to an embodiment, an image processing method includes: receiving a first image including a plurality of frames; obtaining importance information indicating the importance of at least one region included in the plurality of frames; determining at least one mask indicating a scaling degree corresponding to at least one region of the first image based on the importance information; generating a second image by encoding the first image according to the at least one mask; and outputting the at least one mask and the second image.

In the determining of the at least one mask, the resolution of at least one first area of the first image is maintained based on the importance information, and resolutions of the remaining areas except for the first area are down-sampled (down-sampling). sampling), and determining the at least one mask and mask information corresponding to the at least one mask.

The obtaining of the importance information may include: receiving, from a producer terminal of the first image, first importance information set corresponding to at least one region of each frame of the first image; and corresponding to at least one region of each frame of the first image based on an accuracy map according to a result of reconstructing at least one region of each frame of the second image by a pre-trained neural network. The method may include at least one of obtaining the determined second importance information.

The obtaining of the second importance information includes a result of restoring at least one region of each frame of the second image by the pre-trained neural network and a correct answer corresponding to at least one region of each frame of the second image. generating the accuracy map based on a difference between images; and determining second importance information corresponding to at least one region of each frame of the first image based on the accuracy map.

The determining of the at least one mask may include: sorting the second importance information; and determining at least one mask corresponding to at least one region of each frame of the first image based on the aligned second importance information.

In the determining of the at least one mask based on the aligned second importance information, the first resolution of the first region in the first image according to the aligned second importance information is the original resolution of the first image. determining, as a first mask, a mask corresponding to the first area to be set to ; determining a mask corresponding to the second region as a second mask so that a second region excluding the first region is down-sampled to a second resolution lower than the first resolution according to the aligned second importance information; and a third mask corresponding to the third area so that the remaining third areas except for the first area and the second area are down-sampled to a third resolution lower than the second resolution according to the aligned second importance information. It may include at least one of determining the mask.

The neural network restores the second image based on a combined image in which at least one of a first mask, a second mask, and a third mask corresponding to the second image is concatenated. You can create a reconstructed image.

The determining of the at least one mask may include determining at least one mask for the first region of the first image according to the first importance information, regardless of the second importance information; and determining at least one mask for regions other than the first region according to the second importance information.

In the determining of the at least one mask, the plurality of frames of the first image based on the importance information so that the capacity of the second image is maintained equal to the capacity obtained by downscaling the first image by a predetermined ratio. The method may include determining at least one mask of at least one region included in the fields.

The determining of the at least one mask may include: determining the at least one mask for each frame of the first image based on the importance information; and determining the at least one mask for each chunk including a plurality of frames of the first image based on the importance information.

The obtaining of the importance information may include: dividing each of the plurality of frames into grids corresponding to at least one region of each of the plurality of frames; and obtaining importance information of regions corresponding to the plurality of grids.

The first image is divided into at least one of a first area, a second area, and a third area based on the at least one mask, and the generating of the second image includes a first area corresponding to the first area. encoding the first region based on one mask; encoding the second region based on a second mask corresponding to the second region; and encoding the third region based on a third mask corresponding to the third region.

The first image may include live streaming contents.

According to an embodiment, an image reproducing method includes: acquiring image information including an image having a plurality of regions having a plurality of resolutions and at least one mask corresponding to the plurality of regions of the image; reconstructing the image using a pre-trained neural network based on the image information; and reproducing the restored image.

The restoring of the image may include: extracting the at least one mask from the image information; and reconstructing the image using the neural network based on the extracted at least one mask.

The neural network may be trained to generate the reconstructed image corresponding to the image based on the image and at least one mask corresponding to the image.

According to an embodiment, an image processing apparatus includes: a communication interface for receiving a first image including a plurality of frames; and at least one mask that obtains importance information indicating importance of at least one region included in the plurality of frames, and indicates a scaling degree corresponding to at least one region of the first image based on the importance information. and a processor configured to generate a second image by determining , and encoding the first image according to the at least one mask, wherein the communication interface outputs the mask and the second image.

According to an embodiment, an image reproducing apparatus includes: a communication interface for acquiring image information including an image having a plurality of regions including a plurality of resolutions and at least one mask corresponding to the plurality of regions of the image; a processor for reconstructing the image using a pre-trained neural network based on the image information; and a display for reproducing the restored image.

According to one side, the image processing device provides streaming of the original streaming video at a resolution that is two times or more lower, and the video playback device maintains the original video through the artificial neural network-based Super-Resolution-based restoration. It is possible to provide a high-quality video streaming service in a network environment having a transmission capacity.

According to one side, it is possible to reduce the capacity of an image for a live streaming service by maintaining the original resolution as much as possible for an important region of each frame of the original image and down-sampling the remaining region except for the important region.

According to one side, it is possible to reduce the cost of using an HLS (HTTP Live Streaming) server by performing streaming at a lower capacity of 4 times or more in the server for streaming.

According to one side, data transmission cost can be reduced by transmitting and receiving image data with a low capacity through mask-based scaling.

According to one side, by super-resolution (SR) technology that restores a low-resolution image to a high-resolution image based on an artificial neural network, the original resolution is maintained as much as possible in the area that cannot be restored well in the original image, and the rest can be restored well. The resolution of the region can be transmitted at a lower resolution through downscaling, thereby preserving the image quality of an important region in the original image or a region with poor quality of ultra-high resolution during restoration.

1 is a diagram for explaining the configuration and operation of a video streaming system according to an embodiment.
2 is a flowchart illustrating an image processing method according to an embodiment.
3 is a diagram for describing a method of generating a rescaled image based on a second image, according to an exemplary embodiment;
4 is a diagram illustrating an example of a second image according to an embodiment;
5 is a diagram for describing a configuration and operation of an image processing apparatus according to an exemplary embodiment;
FIG. 6 is a view for explaining the structure and operation of the reconstructed neural network shown in FIG. 5;
7 is a flowchart illustrating an image reproducing method according to an embodiment.
8 is a block diagram of an image processing apparatus according to an exemplary embodiment;
9 is a block diagram of an image reproducing apparatus according to an exemplary embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for description purposes only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a diagram for explaining the configuration and operation of a video streaming system according to an embodiment. Referring to FIG. 1 , an apparatus for processing an image (hereinafter, 'image processing device') 110, a streaming server 130, and an apparatus for playing an image (hereinafter, 'image playback device') according to an embodiment A video streaming system 100 including 150 is shown.

The image processing apparatus 110 according to an embodiment receives the original image 111 from a broadcast transmitter or a producer terminal. The original image 111 may be, for example, live streaming content or 360-degree content image transmitted through various streaming protocols. The streaming protocol is a protocol used to stream audio, video, and other data over the Internet, and may include, for example, Real Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS). can The original image 111 may be, for example, an image having a size of width (w) x height (h). In this case, the width w may correspond to a size occupied by all columns in the width direction, and the height h may correspond to a size occupied by all rows in the height direction. In this specification, for convenience of description, the original image 111 may be referred to as a 'first image'. Hereinafter, 'original image' and 'first image' may be understood to have the same meaning.

The image processing apparatus 110 includes a real-time down-scaling module 120 . The image processing apparatus 110 obtains importance information on the original image by a neural network included in the real-time downscaling module 120 , and determines based on the importance information, at least one of the original image 111 . A scaled image, ie, a second image, may be generated by encoding the first image according to the mask corresponding to the region. Hereinafter, for convenience of description, an operation in the case of processing a live image 'in real time' will be described. However, the embodiments described below may be substantially equally applied to processing a VOD image. In this case, the real-time downscaling module 120 may be referred to as a downscaling module.

In some cases, the image processing apparatus 110 may further include an important region designation module 115 . The important region designation module 115 may directly receive importance information ('first importance information') corresponding to at least one region of each frame of the original image 111 from a broadcaster or producer. Here, the 'importance information' may be information indicating the importance of region(s) included in a plurality of frames of the original image 111 . The importance information may include, for example, information indicating the importance of a region set as important by a producer, ie, an important region, such as a main character, a major event, an important object, etc. in the original image 111 . For example, the producer may provide the image processing apparatus 110 with importance information indicating the importance of at least one region by designating a mask on the important region of the original image 111 . The producer may set a mask for at least one region through, for example, an operation such as a mouse click and/or dragging on the original image 111 . Hereinafter, for convenience of description, the importance information set by the producer through the important region designation module 115 will be referred to as 'first importance information'.

Unlike the important region designation module 115 , the real-time down-scaling module 120 automatically sets importance information corresponding to at least one region of each frame of the original image 111 by a pre-trained neural network. do.

The real-time downscaling module 120 may receive the original image 111 through an image collection server such as a Real Time Messaging Protocol (hereinafter, 'RTMP') server 113 . The RTMP server 113 may collect broadcasts (eg, the original image 111 or the source video) transmitted through a live stream protocol, for example.

The real-time downscaling module 120 automatically obtains importance information ('second importance information') for the broadcast image (eg, the original image 111) collected from the RTMP server 113 by a pre-trained neural network. can be set. At this time, the importance information is, for example, the accuracy according to the result of reconstructing at least one region of each frame of the image ('second image') downscaled by the real-time downscaling module 120 by a pre-trained neural network. It may be determined based on an accuracy map. As will be described in detail below, the accuracy map may be understood as a map indicating the restoration accuracy of at least one region of a downscaled image when the neural network reconstructs the region. In this specification, for convenience of description, the image downscaled by the real-time downscaling module 120 may be referred to as a 'second image'. Hereinafter, the scaled image (or the rescaled image) and the second image may be understood to have the same meaning.

For example, when the restoration accuracy of the corresponding region by the neural network is high, the real-time downscaling module 120 may set the importance of the corresponding region to be low. Also, when the restoration accuracy of the corresponding region by the neural network is low, the real-time downscaling module 120 may set the importance of the corresponding region to be high. The importance information may be set to different values according to the restoration accuracy of the corresponding region as described above. Hereinafter, for convenience of description, the importance information determined by the real-time downscaling module 120 will be referred to as 'second importance information'.

The real-time downscaling module 120 may include, for example, a real-time importance estimation module 121 , a real-time importance-based downscaling module 123 , and an encoding module. The encoding module may include, for example, an HTTP Live Streaming (HLS) Encoding module 125 .

The real-time importance estimation module 121 may estimate or determine importance information of the original image 111 received from the RTMP server 113 . The real-time importance estimation module 121 may acquire importance information of the original image 111 in real time. For example, when the original image 111 is a live image, the real-time importance estimating module 121 determines the importance level corresponding to at least one region of the corresponding image based on a result of reconstructing the second image by a pre-trained neural network. Information can be obtained in real time. The neural network may be, for example, a neural network trained in advance to generate a reconstructed image corresponding to the second image obtained by downscaling the original image 111 . The neural network may be, for example, a deep neural network including a convolution layer. The neural network may be, for example, the reconstructed neural network 610 shown in FIG. 6 below.

The real-time importance-based downscaling module 123 determines a mask indicating a scaling degree corresponding to at least one region of the original image based on the importance information ('second importance information') determined by the real-time importance estimation module 121 . can Here, the scaling degree may also be referred to as a compression ratio or resolution of each region of the original image. The scaling degree may include, for example, 1x, 1/2x, and 1/4x of the original image, but is not limited thereto. For example, the first mask is a mask representing a scaling degree of 1 times the original image, the second mask is a mask representing a scaling degree of 1/2 times of the original image, and the third mask is 1/4 times of the original image. may be a mask indicating a scaling degree of . The mask may be, for example, a mask file.

The real-time importance-based downscaling module 123 may set the mask information of the first mask of the corresponding region to, for example, '1' when the corresponding region in the original image 111 is determined as the important region according to the importance information. have. Alternatively, the real-time importance-based downscaling module 123 sets the mask information of the second mask or the third mask of the corresponding region to '1' according to the importance information when the corresponding region does not correspond to the important region in the original image according to the importance information. Alternatively, it can be set to '0'.

The HLS encoding module 125 may generate a scaled image by encoding the original image 111 according to the mask information of the mask set in the real-time importance-based downscaling module 123 . A scaled image is, for example, an image for a streaming service encoded in different resolutions (eg, 1080p, 720p, 480p, etc.) according to a mask set in the real-time importance-based downscaling module 123 . can be Here, the streaming service may include, for example, a streaming service for live broadcasting, a streaming service for VOD playback, and the like, but is not necessarily limited thereto. The second image may be, for example, a low-resolution video image in which a scaling degree or resolution of regions of an image frame is set differently according to importance information, as in the second image 400 of FIG. 4 below.

A process in which the real-time downscaling module 120 generates a rescaled image based on the downscaled second image from the first image will be described in detail with reference to FIG. 3 below. In addition, the operation of each module of the real-time downscaling module 120 will be described in more detail with reference to FIG. 5 below.

The image processing apparatus 110 according to an embodiment may provide a high-quality video streaming service even in a low network environment through downscaling that maintains the original resolution of an important region in the original image 111 . More specifically, the image processing apparatus 110 maintains the original resolution as much as possible for an important region of each frame of the original image 111 and down-samples the remaining regions except for the important region to increase the capacity of the image for the live streaming service. can be reduced For example, when an image producer transmits a 360-degree image through a live stream protocol, the image processing apparatus 110 may perform down-scaling in real time to preserve the resolution of an important area in the content as much as possible. .

The streaming server 130 may transmit at least one mask and the second image received from the image processing apparatus 110 to the image reproducing apparatuses 150 . The streaming server 130 may include, for example, a plurality of virtual machines (Virtual Machines) for load balancing. The streaming server 130 may drive a virtual server (or virtual machine)(s) when necessary, and may provide a multi-channel live streaming service by increasing the number of virtual server(s) as desired.

The streaming server 130 may, for example, adjust the number of virtual machines according to the number of viewers who watch the video, that is, the number of video reproducing devices 150 . Each virtual machine can, for example, act as a server that processes HTTP requests. The video distributed to the video playback device 150 through the streaming server 130 is delivered to the video playback device 150 through, for example, a Content Delivery Network (CDN), thereby providing a live streaming service to the user. can be used to The mask may be set in chunks for streaming. Of course, when there is a change in a scene within a single chunk, one or more masks may be set in the corresponding chunk. Alternatively, a smoothing technique may be applied so that the mask does not change rapidly over time.

The image reproducing apparatus 150 may restore the second image using the pre-trained neural network 151 based on image information including at least one mask and the second image received through the streaming server 130 . have. In this case, the at least one mask may be, for example, a mask file corresponding to a plurality of regions of the second image. The image reproducing apparatus 150 may reproduce the restored image through the display apparatus 153 . The image reproducing apparatus 150 according to an embodiment may be a single or a plurality of user apparatuses including, for example, an artificial neural network-based video player or a video player. The operation of the image reproducing apparatus 150 will be described in detail with reference to FIG. 7 below.

The neural network used in the real-time downscaling module 120 of the server and the neural network used in the image reproducing apparatus 150 of the client may be the same. The neural network may receive a downscaled image and a mask and perform an operation of restoring (eg, upscaling) the downscaled image. In this case, the server or the client may input the downscaled image and the mask to the neural network by concatenating them.

2 is a flowchart illustrating an image processing method according to an exemplary embodiment. Referring to FIG. 2 , the image processing apparatus according to an embodiment receives a first image including a plurality of frames ( 210 ).

The image processing apparatus acquires importance information indicating the importance of at least one region included in the plurality of frames ( S220 ). In operation 220 , the image processing apparatus may receive, for example, first importance information set corresponding to at least one region of each frame of the first image from a producer terminal of the first image. Alternatively, the image processing apparatus may include, for example, at least one of each frame of the first image based on an accuracy map according to a result of reconstructing at least one region of each frame of the second image by a pre-trained neural network. The second importance information determined corresponding to the area of . Here, the neural network corresponds to the second image based on the second image and the combined image in which at least one mask of first mask information, second mask information, and third mask information corresponding to the second image is concatenated. It may be a neural network trained to generate a reconstructed image. For example, the neural network may be trained to minimize a loss function between a ground truth image corresponding to the second image and a reconstructed image.

A method of obtaining the second importance information by the image processing apparatus according to an embodiment is as follows. The image processing apparatus may, for example, based on a difference between a result of reconstructing at least one region of each frame of the second image by a pre-trained neural network and a correct answer image corresponding to at least one region of each frame of the second image. You can create an accuracy map. The image processing apparatus may determine second importance information corresponding to at least one region of each frame of the first image based on the accuracy map.

For example, the image processing apparatus divides each of the plurality of frames into grids corresponding to at least one region of each of the plurality of frames as shown in FIG. 4 below, and divides each of the plurality of frames into grids corresponding to the plurality of grids. It is possible to obtain the importance information of the area.

The image processing apparatus determines at least one mask indicating a scaling degree corresponding to at least one region of the first image based on the importance information obtained in operation 220 (operation 230). The image processing apparatus maintains the resolution of at least one first area of the first image and down-samples the resolutions of the remaining areas except for the first area based on the importance information, including at least one mask and Mask information of at least one mask may be determined.

When the first importance information is received in operation 220 , the image processing apparatus sets at least one mask for the first region of the first image according to the first importance information, regardless of the second importance information, for example. can decide The image processing apparatus may determine the mask information of the first mask for the first region as, for example, a first value ('1'). Alternatively, when the second importance information is obtained in operation 220 , the image processing apparatus selects masks (eg, the first mask, the second mask, and the third mask) for regions other than the first region and the corresponding masks. Mask information of the mask may be determined according to the second importance information. The image processing apparatus may determine a mask for an area other than the first area as any one of the first mask, the second mask, and the third mask according to the second importance information, and set mask information of the determined mask .

When the second importance information is obtained in step 220 , the image processing apparatus sorts the second importance information, for example, and based on the sorted second importance information, the image processing apparatus of each frame of the first image At least one mask corresponding to at least one area may be determined. More specifically, the image processing apparatus determines the mask corresponding to the first region as the first mask so that the first resolution of the first region in the first image is set to the original resolution of the first image according to the aligned second importance information. can The image processing apparatus determines the mask for the second region as the second mask so that the second region excluding the first region from the first image is down-sampled to a second resolution lower than the first resolution according to the aligned second importance information. can The image processing apparatus generates a mask for the third region so that the remaining third region excluding the first region and the second region in the first image is down-sampled to a third resolution lower than the second resolution according to the aligned second importance information. You can decide with 3 masks.

The image processing apparatus may be configured to adjust the size of the first image based on the importance information so that, for example, the capacity of the second image remains the same as the capacity obtained by downscaling the first image by a predetermined ratio (eg, x1/2). At least one mask of at least one region included in the plurality of frames may be determined. For example, the image processing apparatus sets one area to x1/2->x1 and sets four areas to x1/2->x1/4, so that all areas are uniformly downscaled to x1/2. capacity can be maintained.

In operation 230 , the image processing apparatus may determine at least one mask for each frame of the first image, for example, based on importance information. Alternatively, the image processing apparatus may determine at least one mask for each chunk including a plurality of frames of the first image, based on the importance information. A chunk may include, for example, 512 frames.

The image processing apparatus generates the second image by encoding the first image according to at least one mask ( S240 ). The first image may be divided into, for example, at least one of a first area, a second area, and a third area based on at least one mask. For example, the image processing apparatus may encode the first region based on the first mask corresponding to the first region. The image processing apparatus may encode the second region based on the second mask corresponding to the second region. Also, the image processing apparatus may encode the third region based on the third mask corresponding to the third region.

The image processing apparatus outputs the at least one mask determined in operation 230 and the second image generated in operation 240 (operation 250).

FIG. 3 is a diagram for explaining a method of generating a rescaled image based on a second image according to an embodiment, and FIG. 4 is a diagram illustrating an example of a second image according to an embodiment. The rescaled image may be input to the neural network together with the mask.

The image processing apparatus according to an embodiment downscales, for example, the original image 310 to a size of 1x (x1), 1/2x (x1/2), and 1/4x (x1/4), respectively. Downscaling may generate down-scaled images 320 , 340 , and 360 . The image processing apparatus may use at least one of various known techniques (eg, a bilinear technique and a bicubic interpolation technique) to downscale the original image 310 . . The by-reiner technique may downscale an image by, for example, allocating an average value of four pixel values around an interpolation point as a new pixel value. Also, the bicubic interpolation method can be performed using, for example, a Lagrange polynomial, cubic spline or cubic convolution algorithm as an extension of cubic interpolation to interpolate data points in a two-dimensional rule grid. .

Thereafter, the image processing apparatus upscales each of the downscaled images 320, 340, and 360 to a size of 1x (x1), 2x (x2), and 4x (x4) again. Up-scaled images 325 , 345 , and 365 may be generated. The image processing apparatus upscales the downscaled images 340 and 360, at least among various known techniques (eg, a bilinear technique and a bicubic interpolation technique). one is available Although shown in FIG. 3 for convenience of explanation, the 1x downscaled image 320 and 1x upscaled image 325 are the same as the original image 310, and the 1x downscaled image 320 and A separate operation for generating the 1x upscaled image 325 may not be performed.

The image processing apparatus determines at least one mask corresponding to each region based on the importance information corresponding to the original image 310 in the process described above, and uses the determined mask as up-scaled images 325 . , 345, 365) may be multiplied to generate a rescaled image 380 .

More specifically, the image processing apparatus may multiply the upscaled image 325 by the first mask 330 and the upscaled image 345 by the second mask 350 according to the determination based on the importance information. . Also, the image processing apparatus may multiply the upscaled image 365 by the third mask 370 . In this case, the mask information (eg, the pixel value of the mask file) of each mask may have a value of, for example, 0 or 1. Also, only one of the mask information may have a value of 1, and the remaining mask information may have a value of 0, corresponding to pixels of the same coordinates. For example, as Mask 1(x, y) + Mask 2(x, y) + Mask 3(x, y) = 1, scaling of pixels corresponding to at least one region of each frame of the original image 310 degree can be indicated. Here, (x, y) may correspond to the coordinates of the mask information corresponding to the coordinates of the pixels of the original image 310 .

Accordingly, the rescaled image 380 is downscaled to 1/2 of the first region 450 partially having the resolution of the original image, for example, as in the second image 400 shown in FIG. 4 below. It may have a shape including a second region 430 having a high resolution, and a third region 410 having a resolution downscaled to 1/4. In this case, the mask corresponding to each region may be determined by, for example, importance information (eg, first importance information) of a region set as an important region set by the image producer, or a neural network trained in advance for the region. It may be determined by importance information (eg, second importance information) obtained based on an accuracy map according to a result restored by .

In this way, each region of the second image 400 is scaled differently according to importance information of the corresponding region, and the corresponding scaling information may be indicated by a mask.

5 is a diagram for explaining the configuration and operation of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 5 , the structure and operation of the real-time downscaling module 500 of the image processing apparatus 110 according to an exemplary embodiment is illustrated.

The real-time downscaling module 500 may include a real-time accuracy estimation module 510 , an accuracy-based mask generation module 530 , and an HLS encoding module 550 .

The real-time accuracy estimation module 510 may include a mask-based reconstruction network 513 for reconstructing the downscaled second image. The restoration network 513 may also be referred to as a 'reconstruction neural network'.

The restoration network 513 may restore the image 511 downscaled to have a resolution of x1/2 compared to the original image as the restored image 514 . In this case, the mask may have all values of 1 corresponding to x1/2, and may have values of both 0 corresponding to x1 and x1/4. Also, the reconstruction network 513 may restore the downscaled image 517 to have a resolution of x1/4 compared to the original image as the restored image 518 , for example. In this case, the mask may have all values of 1 in correspondence with x1/4, and may have values of both 0 in correspondence of x1 and x1/2. The reconstruction network 513 may be one single neural network.

The real-time accuracy estimation module 510 is an accuracy map (x2) 516 based on the difference between the reconstructed image 514 reconstructed through the reconstructed network 513 and the correct answer image 515 corresponding to the downscaled image 511 ) can be created. Here, the correct answer image 515 may correspond to the original image. Here, the accuracy map (x2) 516 may be understood as an accuracy map indicating the accuracy of an image obtained by reconstructing (or upscaling) the downscaled image 511 to x2 using the reconstruction network 513 . The real-time accuracy estimation module 510 may estimate second importance information corresponding to at least one region downscaled to have a resolution of x1/2 in the original image based on the accuracy map (x2) 516 .

In addition, the real-time accuracy estimation module 510 is an accuracy map (x4) based on the difference between the reconstructed image 518 reconstructed through the reconstructed network 513 and the correct answer image 519 corresponding to the downscaled image 517. 520 may be created. Here, the correct answer image 519 may correspond to the original image. Here, the accuracy map (x4) 520 may be understood as an accuracy map generated by an image obtained by reconstructing (or upscaling) the downscaled image 517 to x4 using the reconstruction network 513 . The real-time accuracy estimation module 510 may estimate the second importance information corresponding to at least one region downscaled to have a resolution of x1/4 in the original image based on the accuracy map (x4) 516 . The structure and operation of the reconstruction network 513 will be described in more detail with reference to the reconstruction neural network 610 of FIG. 6 below.

The accuracy-based mask generation module 530 may sort 531 the estimated second importance information based on each accuracy map. The accuracy-based mask generating module 530 may determine a mask corresponding to at least one region of each frame of the first image based on the aligned second importance information.

For example, the accuracy-based mask generating module 530 may configure the mask for the first region such that the first resolution of the first region in the first image is set as the original resolution of the first image according to the aligned second importance information. 1) (533) can be determined. The accuracy-based mask generating module 530 may determine a mask (Mask 2) 535 for the second region so that the second region excluding the first region is down-sampled to a second resolution lower than the first resolution. In addition, the accuracy-based mask generation module 530 may perform a third region for the third region to be down-sampled to a third resolution lower than the second resolution, except for the first region and the second region, according to the aligned second importance information. A mask (Mask 3) 537 may be determined.

For example, when the mask information of the first mask for the region A in the image is '1', the mask information of the second mask and the mask information of the third mask of the corresponding region may be '0'. As described above, the accuracy-based mask generating module 530 may set mask information such that only one mask is applied to a certain region of the image.

The accuracy-based mask generation module 530 may determine that, for example, a difference value between the accuracy maps (x2) and 516 of one first grid region is an accuracy map (x4) of four second grid regions. If it is greater than the average of the difference values of 520, the mask information of the mask 1 533 of the first grid area is set to '1', and the mask information of the mask 3 of the second grid area 537 is set to '1'. Mask information can be set to '1'. The first grid area and the second grid area correspond to different areas. The accuracy-based mask generation module 530 repeatedly performs the above process, and then masks the remaining grid area through Mask 2(x, y) = 1 - Mask 1(x,y) - Mask 3(x,y). (Mask 2) 535 may be determined.

The real-time downscaling module 500 utilizes the masks 533, 535, 537 and the original image 539 corresponding to each region set by the accuracy-based mask generation module 530. Rescaled Image. can create The real-time downscaling module 500 may perform rescaling on the original image 539 to generate, for example, the rescaled image 380 as described above with reference to FIG. 3 .

The HLS encoding module 550 may perform HLS encoding of the rescaled image sequence. The HLS encoding module 550 may encode the rescaled image according to mask information of each region set by the accuracy-based mask generation module 530 . As a result, for example, an image for a streaming service in which each region of a frame is encoded in different resolutions (eg, 1080p, 720p, 480p, etc.) according to mask information may be generated. The operation of the HLS encoding module 550 may correspond to the HLS encoding module 125 described above with reference to FIG. 1 .

FIG. 6 is a diagram for explaining the structure and operation of the reconstructed neural network shown in FIG. 5 . Referring to FIG. 6 , a learning method of a reconstructed neural network based on super resolution (SR) according to an embodiment is illustrated.

The image processing apparatus receives a training image and at least one mask corresponding to the training image. At least one mask used in the learning process may be randomly set. The image processing apparatus may learn the reconstructed neural network 610 through, for example, a skip connection (or shortcut connection) using the residual blocks 613 of the reconstructed neural network 610 . The reconstructed neural network 610 may be, for example, Residual Networks (ResNet), which is one of Convolutional Neural Networks (CNN). Residual blocks 613 in the reconstructed neural network 610 allow the activation of a layer to be transferred from the neural network to a deeper layer quickly, and a much deeper neural network can be trained through this simple adjustment. In the case of learning the reconstructed neural network 610 by applying the skip connection, if the identity mapping(x) is differentiated in the back propagation process, a value of at least 1 is obtained, so it is stable even if the layer of the reconstructed neural network 610 deepens. learning is made possible

More specifically, a first mask 603 , a second mask 605 , and a third mask corresponding to the (re)scaled image 601 and the (re)scaled image 601 in the reconstructed neural network 610 ( It is assumed that a combined image 609 in which at least one mask of 607) is concatenated is applied.

The reconstructed neural network 610 may be trained to generate a reconstructed image 630 corresponding to the (re)scaled image 601 based on the combined image 609 . At this time, the reconstructed neural network 610 is a loss function ( loss function). The reconstructed neural network 610 may be trained in a direction to minimize the loss function.

In an embodiment, the reconstructed neural network 610 may be trained in a low dimension through, for example, the Inv Pixel Shuffle module 611 to improve speed. Inv Pixel Shuffle module 611 is, for example, F (Feature Channel Size) * W (Width) * H (Height) Dimension F * 4 (Feature Channel Size) * W/2 (Width) * H/2 ( Height) can be calculated by lowering it to Dimension. Through this, the computation time can be reduced from approximately W*H to (W/2) * (H/2).

When the reconstruction neural network 610 is trained, a mask and mask information (eg, pixel values of a mask file) corresponding to the mask may be randomly generated.

The image processing apparatus may acquire second importance information determined to correspond to at least one region of each frame of the original image during actual streaming based on the learned reconstruction neural network 610 as shown in FIG. 6 .

7 is a flowchart illustrating a method of reconstructing an image according to an exemplary embodiment. Referring to FIG. 7 , the image reproducing apparatus according to an embodiment acquires an image having a plurality of regions including a plurality of resolutions and image information including mask information corresponding to the plurality of regions of the image ( 710 ). .

The image reproducing apparatus reconstructs an image using a pre-trained neural network based on the image information obtained in operation 710 (operation 720). The image reproducing apparatus may extract at least one mask from the image information. The image reproducing apparatus may reconstruct an image using a neural network based on the extracted at least one mask. In this case, the neural network may be, for example, a neural network trained to generate a reconstructed image corresponding to an image based on an image and at least one mask corresponding to the image.

The image reproducing apparatus reproduces the image restored in step 720 (step 730). The image reproducing apparatus may reproduce the image restored in step 720 through, for example, a display (refer to 970 of FIG. 9 ).

8 is a block diagram of an image processing apparatus according to an exemplary embodiment. Referring to FIG. 8 , an image processing apparatus (hereinafter, 'image processing apparatus') 800 according to an exemplary embodiment includes a communication interface 810 and a processor 830 . The image processing apparatus 800 may further include a memory 850 . The communication interface 810 , the processor 830 , and the memory 850 may communicate with each other via the communication bus 805 .

The communication interface 810 receives a first image including a plurality of frames. The communication interface 810 outputs at least one mask determined by the processor 830 and the second image generated by the processor 830 .

The processor 830 obtains importance information indicating the importance of at least one region included in the plurality of frames. The processor 830 determines at least one mask indicating a scaling degree corresponding to at least one region of the first image based on the importance information. The processor 830 encodes the first image according to at least one mask.

The memory 850 may at least temporarily store the first image in order to process the first image received through the communication interface 810 . The memory 850 may store importance information obtained by the processor 830 corresponding to at least one region of the first image. Also, the memory 850 may at least temporarily store the mask information determined by the processor 830 and/or the second image generated by the processor 830 .

Also, the processor 830 may perform the at least one method described above with reference to FIGS. 1 to 6 or an algorithm corresponding to the at least one method. The processor 830 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The processor 830 may execute a program and control the image processing apparatus 800 . The program code executed by the processor 830 may be stored in the memory 850 .

The memory 850 may store various pieces of information generated in the process of the above-described processor 830 . In addition, the memory 850 may store various data and programs. The memory 850 may include a volatile memory or a non-volatile memory. The memory 850 may include a mass storage medium such as a hard disk to store various data.

9 is a block diagram of an apparatus for reconstructing an image according to an exemplary embodiment. Referring to FIG. 9 , an apparatus for reconstructing an image (hereinafter, referred to as an 'image reproducing apparatus') 900 according to an exemplary embodiment includes a communication interface 910 , a processor 930 , and a display 970 . The image reproducing apparatus 900 may further include a memory 950 . Communication interface 910 , processor 930 , memory 950 , and display 970 may communicate with each other via communication bus 905 .

The communication interface 910 obtains an image having a plurality of regions including a plurality of resolutions and image information including mask information corresponding to the plurality of regions of the image.

The processor 930 reconstructs an image using a pre-trained neural network based on the image information.

The memory 950 may store image information including image and mask information acquired through the communication interface 910 . The memory 950 may store the image restored by the processor 930 .

The display 970 reproduces the image restored by the processor 930 .

Also, the processor 930 may perform at least one method described above with reference to FIGS. 1 and 7 or an algorithm corresponding to the at least one method. The processor 930 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The processor 930 may execute a program and control the image reproducing apparatus 900 . The program code executed by the processor 930 may be stored in the memory 950 .

The memory 950 may store various pieces of information generated in the process of the above-described processor 930 . In addition, the memory 950 may store various data and programs. The memory 950 may include a volatile memory or a non-volatile memory. The memory 950 may include a mass storage medium such as a hard disk to store various data.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

100: video streaming system
110: image processing device
130: streaming server
150: video playback device

Claims (19)

Receiving a first image including a plurality of frames;
obtaining importance information indicating the importance of at least one region included in the plurality of frames;
determining at least one mask indicating a scaling degree corresponding to at least one region of the first image based on the importance information;
generating a second image by encoding the first image according to the at least one mask; and
outputting the at least one mask and the second image
containing,
Image processing method. According to claim 1,
The step of determining the at least one mask includes:
The at least one mask and the at least one mask so that the resolution of the at least one first area of the first image is maintained and resolutions of the remaining areas except for the first area are down-sampled based on the importance information; determining mask information corresponding to the at least one mask;
containing,
Image processing method. According to claim 1,
The step of obtaining the importance information is
receiving, from a producer terminal of the first image, first importance information set corresponding to at least one region of each frame of the first image; and
At least one region of each frame of the second image is determined corresponding to at least one region of each frame of the first image based on an accuracy map according to a result of reconstructing at least one region of each frame of the first image by a pre-trained neural network. obtaining the second importance information
comprising at least one of
Image processing method. 4. The method of claim 3,
The step of obtaining the second importance information
The accuracy map is calculated based on a difference between a result of restoring at least one region of each frame of the second image by the pre-trained neural network and a correct answer image corresponding to at least one region of each frame of the second image. generating; and
determining second importance information corresponding to at least one region of each frame of the first image based on the accuracy map
containing,
Image processing method. 4. The method of claim 3,
The step of determining the at least one mask includes:
sorting the second importance information; and
determining at least one mask corresponding to at least one region of each frame of the first image based on the aligned second importance information;
containing,
Image processing method. 6. The method of claim 5,
Determining the at least one mask based on the aligned second importance information may include:
determining a mask corresponding to the first region as a first mask so that a first resolution of a first region in the first image is set to an original resolution of the first image according to the aligned second importance information;
determining a mask corresponding to the second region as a second mask so that a second region excluding the first region is down-sampled to a second resolution lower than the first resolution according to the aligned second importance information; and
A mask corresponding to the third region is used as a third mask so that the remaining third regions except for the first region and the second region are down-sampled to a third resolution lower than the second resolution according to the aligned second importance information. step to decide
comprising at least one of
Image processing method. 4. The method of claim 3,
The neural network is
A reconstructed image corresponding to the second image is obtained based on a combined image in which the second image and at least one of a first mask, a second mask, and a third mask corresponding to the second image are concatenated. image) to create
Image processing method. 4. The method of claim 3,
The step of determining the at least one mask includes:
determining at least one mask for the first region of the first image according to the first importance information regardless of the second importance information; and
determining at least one mask for regions other than the first region according to the second importance information;
containing,
Image processing method. According to claim 1,
The step of determining the at least one mask includes:
At least one area of at least one region included in the plurality of frames of the first image based on the importance information so that the capacity of the second image is maintained equal to the capacity obtained by downscaling the first image by a predetermined ratio Steps to determine one mask
containing,
Image processing method. According to claim 1,
The step of determining the at least one mask includes:
determining the at least one mask for each frame of the first image based on the importance information; and
determining the at least one mask for each chunk including a plurality of frames of the first image based on the importance information
including any one of
Image processing method. According to claim 1,
The step of obtaining the importance information is
dividing each of the plurality of frames into grids corresponding to at least one region of each of the plurality of frames;
obtaining importance information of regions corresponding to the plurality of grids;
containing,
Image processing method. According to claim 1,
The first image is
divided into at least one of a first region, a second region, and a third region based on the at least one mask;
generating the second image
encoding the first region based on a first mask corresponding to the first region;
encoding the second region based on a second mask corresponding to the second region; and
encoding the third region based on a third mask corresponding to the third region;
containing,
Image processing method. According to claim 1,
The first image is
including live streaming contents;
Image processing method. obtaining image information including an image having a plurality of regions including a plurality of resolutions and at least one mask corresponding to the plurality of regions of the image;
reconstructing the image using a pre-trained neural network based on the image information; and
Playing the restored image
containing,
How to play video. 15. The method of claim 14,
Restoring the image
extracting the at least one mask from the image information; and
Restoring the image using the neural network based on the extracted at least one mask
containing,
How to play video. 15. The method of claim 14,
The neural network is
learned to generate the reconstructed image corresponding to the image based on the image and at least one mask corresponding to the image,
How to play video. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 16. a communication interface for receiving a first image including a plurality of frames; and
At least one mask indicating a degree of scaling corresponding to at least one region of the first image is obtained based on the importance information indicating the importance of at least one region included in the plurality of frames. a processor that determines and encodes the first image according to the at least one mask to generate a second image
including,
The communication interface is
outputting the mask and the second image,
image processing device. a communication interface for obtaining image information including an image having a plurality of regions including a plurality of resolutions and at least one mask corresponding to the plurality of regions of the image;
a processor for reconstructing the image using a pre-trained neural network based on the image information; and
A display that reproduces the restored image
containing,
video playback device.

Images