KR102273113B1 - Apparatus for transmitting image - Google Patents

Abstract

An image transmitting apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a small user and a high-end terminal. Based on the first coding unit for coding the low-resolution image, and when receiving the low-resolution image converted by the down-sampling unit, the super-resolution unit pre-learned to output a processed high-resolution image lower in quality than the original high-resolution image and a residual map providing unit that calculates the difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map, and a second coding scheme that can be decoded by the high-end terminal, the residual and a second coding unit for coding the map.

Description

Image Transmission Device {APPARATUS FOR TRANSMITTING IMAGE}

The present invention relates to an image transmitting apparatus.

In general data communication, distortion is not allowed when compression is performed, whereas a certain amount of distortion is allowed for multimedia signals such as video and audio, thereby increasing compression efficiency.

At this time, there is a demand for a technique for allowing a certain amount of distortion and performing compression according to each of the cases where the receiving terminal is a small user as well as a high-spec terminal (big user).

Republic of Korea Patent Publication No. 10-2017-0047489, publication date May 08, 2017

SUMMARY OF THE INVENTION An object of the present invention is to provide a technology related to effectively transmitting images to terminals having different screen resolutions while having different numbers of antennas.

However, the problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

An image transmission apparatus according to an embodiment includes a down-sampling unit for down-sampling an original high-resolution image and converting it into a low-resolution image, and a first coding technique that can be decoded by each of a low-spec terminal (small user) and a high-spec terminal (big user). Based on the first coding unit for coding the low-resolution image, and when receiving the low-resolution image converted by the down-sampling unit, the super-resolution unit pre-learned to output a processed high-resolution image lower in quality than the original high-resolution image and a residual map providing unit that calculates the difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map, and a second coding technique that can be decoded by the high-end terminal, the residual and a second coding unit for coding the map.

Also, the first coding technique may be an Alamouti technique.

In addition, the second coding scheme may be a Golden coding scheme.

In addition, the super-resolution unit may include a plurality of residual block units configured to include a convolution layer and a ReLU layer but not include a batch normalization layer.

Also, the residual map forming unit may provide the residual map based on a convex optimization technique.

According to an embodiment, optimal performance is possible from an end-to-end point of view, and in particular, terminals having different numbers of antennas and different screen resolutions, such as a low-end terminal and a high-end terminal, are in the same area. Even if it is within, the image can be effectively transmitted to each of these terminals.

1 is a configuration diagram schematically illustrating a configuration of an image transmitting apparatus according to an exemplary embodiment.
2 is a configuration diagram schematically illustrating a configuration of a high-spec terminal (big user) according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 is a configuration diagram schematically illustrating a configuration of an image transmission apparatus 100 according to an exemplary embodiment.

Referring to FIG. 1 , the image transmission apparatus 100 includes a downsampling unit 110 , an image source coding unit 120 , a first preprocessing unit 130 , a first coding unit 140 , and a first mux unit 150 . ), a super-resolution unit 160 , a residual map providing unit 170 , a second pre-processing unit 180 , a second coding unit 190 , and a second mux unit 191 . However, the configuration of the image transmitting apparatus 100 illustrated in FIG. 1 is merely exemplary. For example, according to an embodiment, the image transmitting apparatus 100 may be implemented or configured to be different from that shown in FIG. 1 .

First, the image transmission apparatus 100, and the components included in each of these image transmission apparatus 100, a memory for storing instructions implemented to perform a function to be described below, and a microcontroller for executing the instructions stored in the memory It can be implemented by the processor.

First, the image transmitting apparatus 100 receives an original high resolution (HR) image 10 . Then, the received original high-resolution image 10 is converted into a low-resolution (LR) image 11, and then is coded based on the Alamouti technique or the like. Here, the coding technique for the low-resolution image is not limited thereto.

In addition, the above-described original high-resolution image 10 received by the image transmission device 100 is converted into a low-quality processed high-resolution image, and then represents the difference between the processed high-resolution image and the original high-resolution image 10 . A residual map is generated. The generated residual map is coded based on a Golden code technique or the like. Here, the coding technique for the residual map is not limited thereto.

Then, the low-resolution image 11 coded based on the Alamauti technique, etc. and the residual map coded based on the golden code technique, etc. as described above, are transmitted to an image receiving device not shown in FIG. 1, for example, a low-spec terminal or a high-spec terminal. .

Here, a small user refers to a terminal having one antenna and a screen supporting a low resolution. On the other hand, a high-spec terminal (big user) refers to a terminal having a plurality of (eg, two) antennas and a screen supporting high resolution.

In the high-end terminal, an image as close as possible to the original high-resolution image, that is, a restored high-resolution image, is generated using the low-resolution image 11 and the residual map received in this way. Here, in the process of generating the reconstructed high-resolution image, the aforementioned Alamauti technique or the golden code technique may be used.

On the other hand, in the low-end terminal, the image is restored and generated using only the low-resolution image 11 received in this way. In this restoration process, the aforementioned Alamauti technique may be used.

Hereinafter, the above-described image transmitting apparatus 100 itself will be described in more detail.

The down-sampling unit 110 is configured to down-sample the original high-resolution image 10 . That is, the original high-resolution image 10 is converted into the low-resolution image 11 by the down-sampling unit 110 .

Here, although not shown in FIG. 1 , the down-sampling unit 110 may be implemented to additionally include an anti-aliasing low-pass filter, and in this case, the down-sampling unit 110 generates the original high-resolution image 10 . ) as well as anti-aliasing low-pass filtering function may be performed.

The image source coding unit 120 is configured to perform source coding on the low-resolution image 11 . Here, since the source coding operation for the image is a known technique, a detailed description thereof will be omitted. In addition, the image source coding unit 120 may not be included in the image transmitting apparatus 100 according to an embodiment. In this case, the low-resolution image 11 from the down-sampling unit 110 may be transmitted to the super-resolution unit 160 and the first pre-processing unit 130 to be described later. However, in the following description, it is assumed that the image source coding unit 120 is included in the image transmitting apparatus 100 .

The first pre-processing unit 130 is configured to receive the low-resolution image 11 on which the source coding has been performed from the image source coding unit 120 and perform a predetermined pre-processing. Here, the preprocessing may include, for example, error detection and correction coding, and may also include constellation symbol mapping. However, the first preprocessor 130 may not be included in the image transmission apparatus 100 according to an embodiment. In this case, the low-resolution image 11 from the image source coding unit 120 may be transmitted to the first coding unit 140 to be described later. However, in the following description, it is assumed that the first preprocessor 130 is included in the image transmission apparatus 100 .

The first coding unit 140 is configured to receive the low-resolution image 11 on which the above-described pre-processing has been performed from the first pre-processing unit 130 and to code the low-resolution image 11 based on the first coding technique. Here, the first coding technique may be, for example, an Alamouti technique, but is not limited thereto.

On the other hand, the super-resolution unit 160 may be configured to receive the low-resolution image 11 on which the source coding has been performed from the image source coding unit 120 to achieve super-resolution, that is, to increase the resolution. . For the result of super-resolution, hereinafter, it will be referred to as a 'processed high-resolution image'.

Here, the super-resolution unit 160 may be pre-learned to output a processed high-resolution image when receiving a low-resolution image. Hereinafter, the process of learning such a super-resolution unit 160 will be looked at in more detail.

First, learning data for learning the super-resolution unit 160 is provided. The training data may be, for example, a well-known DIV2K dataset. The DIV2K dataset contains multiple original high-resolution images.

Then, although not shown in FIG. 1 , each of the plurality of original high-resolution images included in the DIV2K dataset is down-sampled by a separate down-sampling module. In this case, according to an embodiment, each of the plurality of original high-resolution images may pass through an anti-aliasing low-pass filter. Thereafter, each of the plurality of down-sampled original high-resolution images may be compressed by a separate compression module according to various different compression rates. For example, the compression ratio at this time may be 0.05, 0.1, 0.15, ... 2.0 bpp, but is not limited thereto. As a result, a plurality of low-resolution images compressed with different compression ratios may be provided.

After that, by a separate learning module, by taking the plurality of original high-resolution images prepared as described above as the correct answer, and inputting a plurality of low-resolution images compressed at different compression rates, the super-resolution unit 160 can be learned have. Here, 'learning' may be machine learning, for example, deep learning, but is not limited thereto.

In this case, the low-resolution images used for learning may be compressed with different compression ratios as described above. For example, when there are 100 original high-resolution images, 100 low-resolution images compressed with 2.0 bpp, 100 low-resolution images compressed with 1.95 bpp, ... 100 low-resolution images compressed with 0.05 bpp. have.

In addition, in the learning process, learning can be performed on the super-resolution unit 160 by 300 epochs with the low-resolution image compressed to 2.0 bpp and the corresponding original high-resolution image, and when the learning is finished, it is compressed to another value of bpp. Learning may be performed on the super-resolution unit 160 by 150 epochs with the low-resolution image and the corresponding high-resolution image. Here, the compression ratio for the low-resolution image used when learning is performed and the value of the epoch mentioned in each are merely exemplary.

Meanwhile, the above-described super-resolution unit 160 may include a plurality of residual block units. In this case, each of the plurality of residual block units may be configured to include a convolution layer and a ReLU layer, but may not include a batch normalization layer. By configuring each of the plurality of residual block units in this way, the memory footprint can be smaller and the performance can be further improved.

The residual map providing unit 170 is configured to provide a residual map. The residual map refers to a map in the form of a matrix indicating the difference between the above-described processed high-resolution image and the above-described original high-resolution image 10 . Such a residual map may be provided based on a convex optimization technique, but is not limited thereto. Here, since convex optimization itself is a known technique, a description thereof will be omitted, but in an embodiment, restrictions on the residual map so that a reconstructed high-resolution image closer to the original high-resolution image can be generated by such a residual map. (constraint) is introduced.

To explore these limitations, we first call the original high-resolution image U ₀ , the processed high-resolution image U ₁ , and the residual map R, where each of these U ₀ , U ₁ and R is a matrix of _{N HR} x N _{HR .} Let's assume Here, N _HR x N _HR may mean a resolution for each of _{U 0} , U _{1 , and R .} In this case, if the total number of elements contained in R as N _HR ^2, elements with non-zero values of these elements are up to L may be not only information of (L is a natural number less than N _HR ²⁾ dog. In this case, the value of L may have a value based on a compression rate or a bit budget.

The second preprocessor 180 is configured to receive the residual map from the residual map providing unit 170 and perform predetermined preprocessing. Here, the preprocessing may include, for example, error detection and correction coding, and may also include constellation symbol mapping. However, the second preprocessor 180 may not be included in the image transmitting apparatus 100 according to an embodiment. However, in the following description, it is assumed that the first preprocessor 130 is included in the image transmission apparatus 100 .

The second coding unit 190 is configured to receive the residual map on which the aforementioned preprocessing has been performed from the second preprocessor 180 and to code the residual map based on the second coding technique. The second coding scheme herein may be, for example, a Golden coding scheme, but is not limited thereto.

The first mux unit 150 receives the low-resolution image 11 coded based on the first coding technique from the first coding unit 140 , and also receives the low-resolution image 11 coded based on the first coding technique from the second coding unit 190 based on the second coding technique. After receiving the coded residual map, it is configured to perform a multiplexing operation on the received ones. The result of the multiplexing operation may be transmitted to the image receiving apparatus through the antenna shown in FIG. 1 .

Similarly, the second mux unit 191 receives the low-resolution image 11 coded based on the first coding technique from the first coding unit 140 , and also receives the second coding technique from the second coding unit 190 . After receiving the coded residual map based on it, it is configured to perform a multiplexing operation on the received ones. The result of the multiplexing operation may be transmitted to the image receiving apparatus through the antenna shown in FIG. 1 .

That is, according to an embodiment, in the image transmitting apparatus 100 , a low-resolution image is generated based on the original high-resolution image, and a low-quality processed high-resolution image is generated based on the generated low-resolution image, and a low-quality processed high-resolution image and A residual map representing the difference between the above-described original high-resolution images is generated. In addition, the generated residual map and the above-described low-resolution image are each coded based on a predetermined coding technique and transmitted to the image receiving apparatus, respectively.

Hereinafter, such an image receiving apparatus will be described. As described above, the image receiving apparatus may include a high-spec terminal and a low-spec terminal. Hereinafter, a high-spec terminal will be first described.

2 is a configuration diagram schematically showing the configuration of the high-spec terminal 200 according to an embodiment.

Referring to FIG. 2 , the high-end terminal 200 includes a first demux unit 210 , a first decoding unit 220 , a first post-processing unit 230 , an image source decoding unit 240 , and a super-resolution unit 250 . ), a second demux unit 211 , a second decoding unit 260 , and a second post-processing unit 270 . However, the configuration of the high-spec terminal 200 shown in FIG. 2 is merely exemplary. For example, according to an embodiment, the high-spec terminal 200 may be implemented or configured to be different from that shown in FIG. 2 .

First, the high-spec terminal 200, and the components included in each of these high-spec terminal 200, are a memory for storing instructions implemented to perform a function to be described below, and a microprocessor for executing the instructions stored in the memory. can be implemented by

First, the high-end terminal 200 receives the above-described two types of data from the image transmitting apparatus 100 through the two antennas shown in FIG. 2 . One is a residual map and the other is a low-resolution image. Here, the residual map is coded by the first coding scheme as described above, and the low-resolution image is coded by the second coding scheme.

The first demux unit 210 and the second demux unit 211 provide the first decoding unit 220 and the second decoding unit 260, respectively, the residual map coded by the first coding technique, and the second coding unit. It provides a low-resolution image coded by the technique.

Then, the first decoding unit 220 performs decoding based on the received ones. In decoding, an Alamauti decoding technique may be employed, but the present invention is not limited thereto.

The result decoded by the first decoding unit 220 is transmitted to the first post-processing unit 230 . The first post-processing unit 230 may be configured to perform constellation symbol mapping, and may also be configured to perform error detection and correction.

The post-processed result by the first post-processing unit 230 is transmitted to the image source decoding unit 240 . The image source decoding unit 240 may be configured to output a low-resolution image by, for example, performing source decoding on the result received from the first post-processing unit 230 .

Meanwhile, the second decoding unit 260 performs decoding on the items received from the first demux unit 210 and the second demux unit 211 . In decoding, a golden coding technique may be employed, but is not limited thereto.

The result decoded by the second decoding unit 260 is transmitted to the second post-processing unit 270 . The second post-processing unit 270 may be configured to perform constellation symbol mapping, and may also be configured to perform error detection and correction.

The super-resolution unit 250 generates a reconstructed high-resolution image 21 based on the residual map, which is the result from the low-resolution image 20 and the second post-processing unit 270, the result from the image source decoding unit 240. . Here, the process of generating the reconstructed high-resolution image by the super-resolution unit 250 is shown in Equation 1 below.

[Equation 1]

U ₂ = U ₁ +GR

Here, U2 denotes a reconstructed high-resolution image generated by the super-resolution unit 250, U ₁ denotes a processed high-resolution image processed from a low-resolution image, R denotes a residual map, and G denotes a predefined two-dimensional convolution kernel. . Here, G may be a Gaussian convolution filter as in Equation 2 below, but is not limited thereto.

Hereinafter, with reference to FIGS. 1 and 2 , after the original high-resolution image is input to the image transmitting apparatus 100 , a process of outputting the restored high-resolution image from the high-end terminal 200 will be described below.

The original high-resolution image 10 is down-sampled by the down-sampling unit 110 . At this time, as described above, the original high-resolution image 10 may pass through an anti-aliasing low-pass filter according to an embodiment.

The down-sampling unit 110 outputs the low-resolution image 11 in response to the input of the original high-resolution image 10 . Then, the image source coding unit 120 performs predetermined source coding on the result output by the down-sampling unit 110 .

Then, the first pre-processing unit 130 performs a predetermined pre-processing on the result output by the image source coding unit 120 , and the first coding unit 140 processes the result output by the first pre-processing unit 130 . 1 Coding based on the coding technique.

On the other hand, the result output by the image source coding unit 120 is also delivered to the super-resolution unit (160). Then, the super-resolution unit 160 outputs a low-quality high-resolution image corresponding thereto. Then, the residual map providing unit 170 provides a residual map representing the difference in the form of a matrix based on the difference between the original high-resolution image 10 and the result of the super-resolution unit 160 and the low-quality high-resolution image.

The second pre-processing unit 180 performs a predetermined pre-processing on the residual map, and the second coding unit 190 pre-processes the result of the second pre-processing unit 180 based on the second coding technique.

Then, the first mux unit 150 receives the result of the first coding unit 140 and the second coding unit 190 and performs a multiplexing operation, and the second mux unit 191 also receives the result of the second coding unit 190 . A multiplexing operation is performed by receiving the result of 140 and the result of the second coding unit 190 . Each of the multiplexed results is transmitted to the high-end terminal 200 through the antenna shown in FIG. 1 .

Next, a low-spec terminal will be described. 3 is a configuration diagram schematically illustrating a configuration of a low-spec terminal 300 according to an embodiment.

Referring to FIG. 3 , the low-end terminal 300 includes a first demux unit 310 , a first decoding unit 320 , a first post-processing unit 330 , and an image source decoding unit 340 . However, the configuration of the low-spec terminal 300 shown in FIG. 3 is merely exemplary. For example, according to an embodiment, the low-end terminal 300 may be implemented or configured to be different from that shown in FIG. 3 .

First, the low-spec terminal 300, and the components included in each of these low-spec terminal 300, a memory for storing instructions implemented to perform a function to be described below, and a microprocessor for executing the instructions stored in the memory can be implemented by

First, the low-end terminal 300 receives a low-resolution image from the image transmitting apparatus 100 through one antenna shown in FIG. 3 . Here, the low-resolution image is coded by the second coding technique described above.

The first demux unit 310 provides the coded low-resolution image coded by the second coding technique to the first decoding unit 320 .

Then, the first decoding unit 320 performs decoding based on the received ones. In decoding, an Alamauti decoding technique may be employed, but the present invention is not limited thereto.

The result decoded by the first decoding unit 320 is transmitted to the first post-processing unit 330 . The first post-processing unit 330 may be configured to perform constellation symbol mapping, and may also be configured to perform error detection and correction.

The result post-processed by the first post-processing unit 330 is transmitted to the image source decoding unit 340 . The image source decoding unit 340 may be configured to output a low-resolution image by, for example, performing source decoding on the result received from the first post-processing unit 330 .

As described above, according to one embodiment, optimal performance is possible from an end-to-end point of view, and in particular, terminals having different numbers of antennas and different screen resolutions, such as low-spec terminals, Even if a high-end terminal and a high-end terminal are in the same area, an image can be effectively transmitted to each terminal.

Meanwhile, according to an embodiment, the image transmission method may be performed by the image transmission apparatus 100 according to the embodiment, and the steps that may be included in each of these methods are instructions programmed to perform these steps. It may be stored in a computer-readable recording medium storing the.

Combinations of each block in the block diagram attached to the present invention and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment may be configured in the respective blocks in the block diagram or in the flowchart. Each step creates a means for performing the described functions. These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. The instructions stored in the block diagram may also produce an item of manufacture containing instruction means for performing the functions described in each block in the block diagram or in each step in the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for carrying out the functions described in each block of the block diagram and each step of the flowchart.

Further, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in blocks or steps to occur out of order. For example, it is possible that two blocks or steps shown one after another may in fact be performed substantially simultaneously, or that the blocks or steps may sometimes be performed in the reverse order according to the corresponding function.

The above description is merely illustrative of the technical idea, and various modifications and variations will be possible without departing from the essential characteristics by those skilled in the art to which the present invention pertains. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea, but to explain, and the scope of the technical idea is not limited by these embodiments. The scope of protection should be construed by the following claims, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights.

Claims (7)

A down-sampling unit that down-samples the original high-resolution image and converts it into a low-resolution image;
A first coding unit for coding the low-resolution image based on the Alamounti technique that can be decoded by each of a small user and a large user;
When the low-resolution image converted by the down-sampling unit is received, a super-resolution unit previously learned to output a processed high-resolution image lower in quality than the original high-resolution image;
a residual map providing unit that calculates the difference between the original high-resolution image and the processed high-resolution image and provides it in the form of a residual map;
a second coding unit for coding the residual map based on a Golden coding scheme that can be decoded by the high-end terminal;
In the learning of the super-resolution unit,
A low-resolution image prepared by being compressed according to a different compression rate is used,
The residual map providing unit generates the residual map based on a convex optimization technique,
When the image receiving device of the high-end terminal generates a reconstructed high-resolution image using the low-resolution image and the residual map, the limitation on the residual map so that the difference between the reconstructed high-resolution image and the original high-resolution image can be minimized Introduce a constraint,
The residual map is in the form of a matrix having N (N is a natural number) elements,
The limitation is that, among the N elements, at most L (L is a natural number less than N) elements have a non-zero value, and L is a value based on the compression ratio.
Image sending device. delete delete The method of claim 1,
The super-resolution unit,
A method comprising a plurality of residual block units configured to include a convolution layer and a ReLU layer but not a batch normalization layer.
Image sending device. The method of claim 1,
The super-resolution unit,
It is learned using the original high-resolution image included in the DIV2K dataset and the low-resolution image generated by downsampling the original high-resolution image included in the DIV2K dataset.
Image sending device. delete delete

Images