KR102051597B1 - Apparatus and method for retargeting images based on content-awareness - Google Patents

Abstract

An image resizing apparatus operated by at least one processor, comprising: receiving a source image, extracting a high level feature of the original image through a plurality of convolution layers for encoding, and decoding the high level feature An encoder-decoder outputs a first attention map including location information of a meaningful region by decoding into the layers, and generates a second attention map in which the first attention map is scaled to a target aspect ratio. And a shift map generating a shift map for moving pixels of the original image to an output image based on the second state map, and shifting the original image to the shift map to output a retargeting image. do.

Description

Content-based image resizing device and method {APPARATUS AND METHOD FOR RETARGETING IMAGES BASED ON CONTENT-AWARENESS}

The present invention relates to a retargeting method for adjusting an image size.

Until now, various methods of image retargeting have been studied, and they can be classified into seam carving based method and warping based method. Seam piece based retargeting method changes the aspect ratio of an image by repeatedly removing or inserting seams of non-essential parts. The warping-based retargeting method continuously converts an input image into an image having a target size. The warping-based retargeting method iteratively calculates the optimal scaling factor for the local region and updates the warping image supported by the edge map and the saliency map.

As such, prior art uses prominent maps using low-level hand-crafted features, such as edge maps, that are manually created to find important and semantic areas within an image. Obtain a saliency map and adjust the image size based on this. However, low-level handmade features can't accurately determine meaningful regions in the image, which eventually leads to retargeting without significant semantic parts. For example, FIG. 1 is a diagram illustrating an image retargeting result using low-level features, in which a key part of an image is missing or distorted. This is a fundamental limitation of the technique of resizing images with low level features.

The problem to be solved by the present invention is to extract the high-level features of the input image through the neural network, and to use the high-level features to create an attention map containing the location information on the meaningful region The present invention provides an apparatus and method for adjusting the size of an input image using a shift map generated based on a state map.

Another object of the present invention is to provide an apparatus and method for calculating content loss and structure loss of a scaled output image and learning convolution weights related to output image generation to minimize the loss of the output image. will be.

An image resizing apparatus operated by at least one processor according to an embodiment, the apparatus comprising: receiving an original image, extracting a high level feature of the original image through a plurality of convolution layers for encoding, and extracting the high level feature An encoder-decoder that decodes a plurality of convolution layers for decoding and outputs a first attention map including location information of a meaningful region, and a size of adjusting the first attention map to a target aspect ratio. A map of two states is generated, a shift map for moving pixels of the original image to an output image is generated based on the second state map, and the retargeting image is warped by warping the original image to the shift map. Contains a shifter to output.

The shifter generates a third state map of uniform shape along a column by 1D Duplicate Convolution of the second state map, and adds the third state map and the second state map by weighted summation. The final state map may be generated, and the shift map may be generated by cumulative normalization of the final state map.

The shifter may convolve the second state map into a column vector to generate a one-dimensional row vector and repeatedly copy the row vector to generate the third state map.

The column vector may have weights learned to minimize the loss of the retargeting image.

The plurality of convolution layers for decoding may be antisymmetric to the plurality of convolution layers for encoding.

The plurality of convolution layers for decoding may have weights that are learned to minimize the loss of the retargeting image.

The image resizing device may further include a learning device for calculating a loss of the retargeting image and learning the convolution weights related to the retargeting image output at the encoder-decoder and the shifter to minimize the loss. have.

The loss may include content loss, and the learning apparatus may calculate the content loss based on a classification score obtained by inputting the retargeting image to an image classifier.

The loss may further include a structure loss, and the learning apparatus may calculate the structure loss by comparing shapes around pixels corresponding to each other in the retargeting image and the original image.

An image resizing apparatus operated by at least one processor according to another embodiment, comprising: receiving an original image, extracting a high level feature of the original image through a plurality of convolution layers for encoding, and extracting the high level feature An encoder-decoder for decoding a plurality of convolutional layers for decoding and outputting a first attention map including location information of a meaningful region, a target aspect ratio, and receiving the first attention map as the target map; A shifter for generating a second attention map scaled to an aspect ratio, a retargeting image for scaling the original image to the target aspect ratio based on the second attention map, and calculating a loss of the retargeting image And learning the weights of the plurality of convolution layers for decoding to minimize the loss. And a humidifying device.

The loss includes a content loss and a structure loss, and the learning apparatus calculates the content loss based on a classification score obtained by inputting the retargeting image to an image classifier, and calculates the content loss from each other in the retargeting image and the original image. The structure loss can be calculated by comparing the shapes around the corresponding pixels.

The image classifier may be composed of a plurality of convolution layers for encoding.

The learning apparatus inputs each of the original image and the retargeting image as a plurality of convolution layers for encoding, and then outputs a feature of the original image output from a lower convolution layer of the plurality of convolution layers for encoding. The structure loss may be calculated by comparing features of the retargeting image.

An image resizing method of an image resizing device operated by at least one processor, according to another embodiment, the first attention map including location information of a meaningful area of an original image and a target aspect ratio Receiving an input, generating a second attention map in which the first attention map is scaled to the target aspect ratio, and performing a 1D duplicate convolution of the second attention map to have a uniform shape along a column. Generating a third state map, generating a shift map using a final state map obtained by weighting the third state map and the second state map, and using the shift map. And outputting a retargeting image by adjusting the size of the image.

The generating of the shift map may generate the shift map by cumulative normalization of the final attention map.

The generating of the third state map may include generating a one-dimensional row vector by convolving the second state map into a column vector, and repeatedly generating the third state map by repeatedly replicating the row vector.

The first attention map may be a result of decoding a high level feature of the original image with a plurality of convolution layers for decoding.

The image resizing method may include calculating content loss of the retargeting image based on a classification score obtained by inputting the retargeting image to an image classifier, and surrounding the pixels corresponding to each other in the retargeting image and the original image. Comparing the shape, calculating the structure loss of the retargeting image, and learning the convolution weights related to the retargeting image output to minimize the content loss and the structure loss.

According to an exemplary embodiment of the present invention, while preserving an area considered to be important in an image, a relatively less important area is increased or reduced to change the size of the image, thereby minimizing loss of main content and objects in the image. According to an embodiment of the present invention, it can be applied to various systems requiring image size conversion.

1 is a result of image retargeting using a conventional low step feature.
2 is a block diagram of an image size adjusting apparatus according to an embodiment.
3 is a detailed configuration of an image size adjusting apparatus according to an embodiment.
4 is a diagram for describing a method of generating a final attention map using one-dimensional replication convolution according to an embodiment.
5 is a diagram illustrating the performance of a learning method according to an embodiment.
6 is a flowchart illustrating an image resizing method according to an embodiment.
7 is a flowchart of a learning method according to an exemplary embodiment.
8 to 11 are comparative views illustrating the performance of the present invention.
12 is a result of evaluating image classification accuracy according to image size change.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise. In addition, the terms "… unit", "?", "Module", etc. described in the specification means a unit for processing at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

2 is a block diagram of an image resizing device according to an embodiment, FIG. 3 is a detailed configuration of an image resizing device according to an embodiment, and FIG. 4 is a diagram illustrating a one-dimensional replication convolution according to an embodiment. It is a figure explaining the method of generating the final state map.

2 and 3, the image size adjusting apparatus 10 receives an original image and a target aspect ratio, and converts the original image into a target aspect ratio to output a retargeting image. An image in which the original image is scaled to a target aspect ratio is called a retargeting image. The original image is the input image of the image resizing device 10, and the retargeting image is the output image of the image resizing device 10.

The image resizing device 10 is operated by at least one processor, the encoder-decoder 100 generating an attention map for the original image, and the attention map as the target aspect ratio. A shifter 200 to convert.

The learning apparatus 300 learns the convolution weights related to the output image generation so that the retargeting image output from the shifter 200 is naturally converted while preserving a meaningful region. Learning Device 300 The learning device 300 may be included in the image resizing device 10, but will be described separately in order to separate the image resizing and learning.

The encoder-decoder 100 receives an original image and outputs an attention map including location information on a semantic region using high-level features of the original image. The encoder-decoder 100 gradually upgrades the features output from the encoder 110 and the encoder 110 to extract high-level features of the original image using a neural network. It consists of a decoder 130 that samples and generates an attention map.

The encoder 110 may be a neural network composed of a plurality of convolutional layers and pooling layers that pool them. The neural network may be a deep convolutional network, and may use a pre-learned deep learning model (eg, VGG16). In this case, it is assumed that the weights of the plurality of convolution layers constituting the encoder 110 are already fixed.

 The decoder 130 includes convolution layers anti-symmetrically with the encoder, and generates attention maps from the features output from the encoder 110 through upsampling. The decoder 130 has an antisymmetric structure with the encoder 110, but some layers may be changed as necessary. For example, rectified linear unit (ReLU) layers of the encoder 110 may be changed from the decoder 130 to exponential linear unit (ELU) layers. Meanwhile, the weights of the convolution layers constituting the decoder 130 are learned by the learning apparatus 300.

The shifter 200 receives an attention map generated from the high level feature of the original image from the decoder 130. Accordingly, the shifter 200 may generate a shift map of content-aware using the attention map including the location information of the meaningful area. This allows the shifter 200 to accurately determine a meaningful area, thereby preserving the area considered to be important in the image, and retargeting a relatively less important area (eg, a background). .

The shifter 200 generates a shift map based on the attention map of the original image output from the decoder 130, and outputs a retargeting image by warping the original image based on the shift map. The shift map S (x, y) is a map defining how much each pixel should be shifted from the WxH size input image I to the W'xH size output image. For explanation, the case where only the horizontal size of the image is changed will be described as an example, but the vertical size is also performed in the same manner as the horizontal size. The relationship between the input image I and the output image O through the shift map S (x, y) is shown in Equation 1. In Equation 1, (x, y) is a spatial coordinate of the output image. When only the horizontal size of the image is changed, S is determined within a range of 0 to the horizontal changing size (W-W ').

Since the attention map A _d output from the decoder 130 includes location information on a meaningful region, it may be directly used for content retargeting of content-aware. However, to maintain the overall shape of the image, pixels in similar columns must have similar shift values, or the overall shape of the image may be distorted. Therefore, the shifter 200 uses the 1D Duplicate Convolution to reshape the state of the attention map instead of directly retargeting the attention map A _d of the original image output from the decoder 130. Use a state map that is uniformly restricted along the columns.

To this end, the shifter 200 includes a shift map generator 210 and a warping unit 230.

First, the shift map generator 210 converts the attention map A _d output from the decoder 130 into a retargeting image size. The shift map generator 210 may convert the attention map A _d into a retargeting image size based on the target aspect ratio. The attention map A _r converted to the retargeting image size is expressed as in Equation 2. In Equation 2, A _r is the attention map adjusted to the size of the retargeting image, A _d is the attention map output from the decoder 130, and R is the output image size, that is, the size of the retargeting image.

Next, the shift map generation unit 210 corrects the distortion (warping) by performing a one-dimensional replica convolution of the converted sized attention map A _r to have a uniform shape along the column (vertical).

Referring to FIG. 4, the shift map generation unit 210 convolves the transformed attention map Ar _r (H x W) into a H-dimensional column vector H x 1 to obtain a one-dimensional row vector ( 1 x W). The weights of the H-dimensional column vectors are learned by the learning apparatus 300. The shift map generator 210 repeatedly replicates the one-dimensional row vector H times to generate the attention map A _1D (H x W) by one-dimensional replication convolution. The one-dimensional replication convolution is expressed as in Equation 3.

는 H차원의 열 벡터(H dimensional column vector)와의 컨볼루션을 의미한다.

는 1차원 행 벡터를 H번 반복시키는 연산을 의미한다.In Equation 3, A _r is a state map converted to the size of the retargeting image, and A _1D is a state map uniformly generated along a column (vertical) in one-dimensional replication convolution.

Means convolution with an H dimensional column vector.

Denotes an operation of repeating the one-dimensional row vector H times.

Since the attention map A _1D generated by the one-dimensional replication convolution is filled with only uniform values along the column (vertical), the shift map generator 210 generates the attention map generated by the one-dimensional replication convolution. A final state map A is generated by weighted summing the (A _1D ) and the state map A _r converted to the size of the retargeting image. The final attention map A is expressed as in Equation 4. In Equation 4, λ is a weight variable.

이고, W는 입력 영상의 가로 크기이고, W'는 출력 영상의 가로 크기이다.The shift map generator 210 generates a shift map S (x, y) by cumulative normalization of the final attention map A. FIG. The shift map S (x, y) may be calculated as in Equation 5. In Equation 5,

Where W is the horizontal size of the input image and W 'is the horizontal size of the output image.

가 곱해 짐으로써, 시프트 지도 값의 범위가 0~

가 된다. 최종 주의 지도의 값이 낮은 부분에서는 시프트 지도의 증가량이 적기 때문에, 출력 영상에서 많은 영역을 차지하게 되며, 반대로 최종 주의 지도의 값이 높은 부분은 시프트 지도의 증가량이 커서 출력 영상에 적은 영역을 차지하게 된다.The shift map generation unit 210 performs cumulative normalization so that the shift map monotonically increases. The input image is warped to the output image by the shift map. At this time, if the order of the pixels of the input image is changed, the output image is not output properly, so the order of the pixels should not be reversed. That is, the value of the shift map is cumulative normalized to monotonously increase along the spatial axis (x, y direction). Cumulative normalization converts the value between 0 and 1.

Is multiplied so that the shift map values range from 0 to

Becomes In the lower part of the final attention map, the amount of shift map increase is small, so it occupies a large area in the output image. On the contrary, in the part of the high value of the final attention map, the increase in the shift map occupies a small area in the output image. Done.

The warping unit 230 generates a retargeting image (output image) by warping the original image (the input image) by using the shift map generated by the shift map generator 210. Retargeting is performed according to equation (1).

Since the shift map has subpixel precision, linear interpolation is performed using four neighboring pixels. Differential loss can be applied to a linearly warped image. Retargeting images, on the other hand, are used to train neural networks. To this end, the retargeting image may be zero-padded to fit the input size of the neural network model.

The learning apparatus 300 calculates the loss of the output image output from the shifter 200 and learns the convolution weights related to the output image generation to minimize the loss of the output image. The convolution weights are weights used for the convolution operation in the encoder-decoder 100 and the shifter 200, and are weights of the plurality of convolution layers constituting the decoder 130, the H-dimensional column vector of the shifter 200. Can be .

The learning apparatus 300 uses a loss of the output image, and the loss of the output image may include a content loss and a structure loss. Content loss contributes to preserving important content of the input video. Structural loss (E _s ) contributes to making pixels in similar columns on the state map have similar values.

The content loss is information indicating whether the main object in the image is well preserved in the output image, and the training data for learning that the main object in the original image is well preserved in the output image. If the output image size-converted in the original image is well classified by the image classifier, it can be estimated that the main part of the original image is well preserved in the output image. Accordingly, the learning apparatus 300 inputs the output image of the shifter 200 to the image classifier and defines the content loss based on the class score output from the image classifier. Higher classification scores (i.e. better classification) may result in less outlet loss (i.e., the main portion of the original image is well preserved in the output image). The learning apparatus 300 may use a pre-trained deep learning model (eg, VGG16) as an image classifier. This may be the same as the model for outputting the feature of the original image from the encoder (110).

와

는 정답 라벨과 분류기의 시그모이드(sigmoid) 출력이다. The learning apparatus 300 may calculate the content loss E _c of the output image as shown in Equation 6. In Equation 6, C and N are the number of classes and the number of learning samples of the image classifier,

Wow

Is the answer label and the sigmoid output of the classifier.

Structure loss is information for determining whether the scaled output image is unnatural and is learning data for learning to minimize an unnatural portion of the scaled output image. If the shapes around the corresponding pixels in the retargeting image and the original image are similar, it can be assumed that the size conversion is natural.

는 미리 학습된 딥 러닝 모델(예를 들면, VGG16)의 첫 번째 컨볼루션 레이어

의 출력이다. 첫 번째 컨볼루션 레이어

의 출력은 경계 지도(edge map)와 같은 낮은 단계의 특징을 표현한다.The learning apparatus 300 may calculate a structural loss E _s of the output image as shown in Equation 7 below. The learning apparatus 300 may calculate a structure loss E _s of the output image by comparing the low-level features of the output image and the input image. In Equation 7,

Is the first convolutional layer of a pretrained deep learning model (eg VGG16).

Is the output of. First convolution layer

Outputs represent low-level features such as edge maps.

The learning apparatus 300 learns convolution weights related to output image generation such that content loss E _c and structure loss E _{s of the} output image are minimized.

5 is a diagram illustrating the performance of a learning method according to an embodiment.

Referring to FIG. 5, (a) is an input image, and (b) to (d) are a retargeting image (right) in which the attention map (left) and the input image of the input image are size-converted. At this time, (b) is a retargeting image output from the model trained only by content loss, (c) is a retargeting image output from the model trained by content loss and structure loss, (d) is a content loss and structure loss A retargeting image obtained from a state map generated using a 1D duplicated convolution layer.

Comparing (b) and (c), it can be seen that the result of learning using more structural loss is better than the result of learning only by content loss. Comparing (c) and (d), it can be seen that the best retargeting image is output when the attention map is generated using the 1D replica convolutional layer.

In other words, content loss contributes to preserving important content of the input image. In the present invention, the structure loss and the 1D overlap convolutional layer contribute to reducing the artifacts of the image by making the pixels in the columns in the image have similar values.

6 is a flowchart illustrating an image resizing method according to an embodiment.

Referring to FIG. 6, the image size adjusting device 10 receives an original image and a target aspect ratio (S110).

The image resizing apparatus 10 extracts a high step feature by encoding an original image through a neural network composed of a plurality of convolutional layers (S120). The plurality of convolution layers used for encoding may be a pre-learned deep learning model.

The image resizing apparatus 10 outputs an attention map A _d including position information on a meaningful region by decoding a feature of an original image through a neural network composed of a plurality of convolutional layers (S130). The plurality of convolution layers used for decoding may have an antisymmetric structure with the plurality of convolution layers used for encoding.

The image resizing apparatus 10 generates the attention map A _r by converting the attention map A _d into a retargeting image size based on the target aspect ratio (S140).

The image resizing apparatus 10 generates a state map A _1D having a uniform shape along a column (vertical) by performing a one-dimensional copy convolution of the converted state map A _{r in} step S150.

The image resizing apparatus 10 generates a final attention map A by weighted summing the attention map A _1D generated by the one-dimensional replication convolution and the attention map A _r converted into the size of the retargeting image. (S160).

The image resizing apparatus 10 generates a shift map using the final attention map A (S170). The shift map is a map defining how far each pixel should be shifted from the input image to the output image which is scaled. The image resizing apparatus 10 may generate a shift map by accumulating and normalizing the final attention map A as shown in Equation 5 below.

The image resizing apparatus 10 outputs the retargeting image by warping the input original image using the shift map (S180).

As described above, the image resizing apparatus 10 retargets a shift map, which is a pixel-by-pixel mapping between the input image and the output image, and maps the state map obtained through the high-level feature to the content-aware shift map. This allows you to preserve important content while increasing or decreasing less important backgrounds.

7 is a flowchart of a learning method according to an exemplary embodiment.

Referring to FIG. 7, the learning apparatus 300 calculates a loss of the output image output from the image scaling apparatus 10 and trains the image scaling apparatus 10 to minimize the loss of the output image. Through the learning by the learning apparatus 300, the convolution weights related to the output image generation are determined by the image resizing apparatus 10.

The learning apparatus 300 receives an original image input to the image resizing apparatus 10 and a retargeting image that is converted in size and output from the image resizing apparatus 10 (S210).

The learning apparatus 300 calculates content loss based on the classification score obtained by inputting the retargeting image to the image classifier (S220).

The learning apparatus 300 calculates a structure loss by comparing shapes around pixels corresponding to each other in the retargeting image and the original image (S230). The learning apparatus 300 is a low-level feature such as a boundary map outputted from a lower convolution layer in a neural network composed of a plurality of convolutional layers extracting a feature of an image, as shown in Equation 7 below. ) To calculate the structural loss.

The learning apparatus 300 learns the convolution weights in the image resizing apparatus 10 to minimize content loss and structure loss of the retargeting image (S240).

8 to 11 are comparative views illustrating the performance of the present invention.

Referring to FIG. 8, (a) is an input image, and (b) is an image obtained by linear interpolation of the horizontal size of the image to the horizontal size of the retargeting image. (c) is a retargeting image according to the present invention. Even if the horizontal size of the image is reduced to the horizontal size of the retargeting image, it can be seen that the main content (car) of the image is almost maintained.

Referring to FIG. 9, the vertical length of the input image (the leftmost side) is fixed, and the retargeting image is continuously reduced in aspect ratio from 0.9 to 0.3.

(a) is an input image. (b) is retargeting images by linear interpolation, and since the horizontal size is linearly reduced from 0.9 to 0.3 without considering the content, it can be confirmed that important content (person) of the image is not preserved. . (c) shows the retargeting images according to the present invention, even though the horizontal size of the image is reduced, the main content of the image is almost maintained.

Referring to FIG. 10, the vertical length of the image is fixed and the enlarged retargeting image.

(a) is an input image. (b) is an image magnified 1.5 times by linear interpolation, and (c) is an image magnified 1.5 times by the present invention. (d) is an image magnified 1.7 times by linear interpolation, and (e) is an image magnified 1.7 times by the present invention.

Like the image reduction, the present invention can confirm that the background is increased even though the image is enlarged, while the main content of the image is almost maintained.

Referring to FIG. 11, (a) shows input images. (b) shows retargeting images by linear interpolation. (c) shows retargeting images by manual cropping. (d) shows retargeting images by seam carving, which is a conventional retargeting method. (e) are retargeting images according to the present invention.

As such, even when the vertical size of the image is reduced, the retargeting images according to the present invention can be confirmed that the main contents are almost preserved.

12 is a result of evaluating image classification accuracy according to image size change.

Referring to FIG. 12, the horizontal axis represents a ratio of reducing the size of the input image, and the vertical axis represents a ratio of the classification result of the retargeting image to the classification result of the original image.

The present invention shows that even if the size is reduced, the amount of reduction in the classification accuracy of the retargeting image is significantly smaller than that of other methods. That is, the present invention can be seen that the main part of the image is well preserved even after retargeting.

The embodiments of the present invention described above are not only implemented through the apparatus and the method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (18)

An image resizing device operated by at least one processor,
Receives a source image, extracts a high level feature of the original image through a plurality of convolutional layers for encoding, and decodes the high level feature into a plurality of convolution layers for decoding to obtain positional information about a meaningful region. An encoder-decoder for outputting a first attention map comprising, and
Generate a second attention map in which the first attention map is scaled to a target aspect ratio, and generate a shift map for moving pixels of the original image to an output image based on the second attention map; A shifter for outputting a retargeting image by warping an original image to the shift map,
And the plurality of convolutional layers for decoding are antisymmetric to the plurality of convolutional layers for encoding. In claim 1,
The shifter is
1D Duplicate Convolution of the second state map to generate a third state map of uniform shape along a column, weighted sum of the third state map and the second state map, and a final state map And generating the shift map by cumulative normalization of the final attention map. In claim 2,
The shifter is
And convolving the second state map into a column vector to generate a one-dimensional row vector, and repeatedly replicating the row vector to generate the third state map. In claim 3,
And the column vector has weights that are learned to minimize the loss of the retargeting image. delete In claim 1,
And the plurality of convolution layers for decoding have weights that are learned to minimize the loss of the retargeting image. In claim 1,
A learning apparatus for calculating a loss of the retargeting image and learning the convolution weights related to the retargeting image output at the encoder-decoder and the shifter to minimize the loss.
Image resizing apparatus further comprising. In claim 7,
The loss includes loss of content,
The learning device
And calculating the content loss based on a classification score obtained by inputting the retargeting image to an image classifier. In claim 8,
The loss further comprises a structure loss,
The learning device
And calculating the structure loss by comparing shapes around pixels corresponding to each other in the retargeting image and the original image. An image resizing device operated by at least one processor,
Receives a source image, extracts a high level feature of the original image through a plurality of convolutional layers for encoding, and decodes the high level feature into a plurality of convolution layers for decoding to obtain positional information about a meaningful region. An encoder-decoder for outputting a first attention map comprising:
Receiving a target aspect ratio, generating a second attention map in which the first state map is scaled to the target aspect ratio, and retargeting the image in which the original image is scaled to the target aspect ratio based on the second state map. Shifter to output, and
A learning apparatus for calculating a loss of the retargeting image and learning weights of the plurality of convolution layers for decoding to minimize the loss,
The loss includes loss of content and loss of structure;
The learning device
Calculating the content loss based on a classification score obtained by inputting the retargeting image to an image classifier, and calculating the structure loss by comparing shapes around pixels corresponding to each other in the retargeting image and the original image; Image resizing device. delete In claim 10,
And the image classifier is comprised of a plurality of convolutional layers for encoding. In claim 12,
The learning device
After inputting each of the original image and the retargeting image into the plurality of convolution layers for encoding, the characteristics of the original image and the retargeting image output from the lower convolution layers of the plurality of convolution layers for encoding Comparing the features of the to calculate the structure loss, the image resizing device. An image size adjusting method of an image size adjusting device operated by at least one processor,
Receiving a first attention map and a target aspect ratio including location information of a meaningful region of an original image,
Generating a second state map in which the size of the first state map is adjusted to the target aspect ratio;
Generating a third state map having a uniform shape along a column by performing 1D Duplicate Convolution on the second state map;
Generating a shift map using a final state map obtained by weighting the third state map and the second state map, and
Outputting a retargeting image by adjusting the size of the original image using the shift map;
Image scaling method comprising a. The method of claim 14,
Generating the shift map
And cumulative normalization of the final attention map to generate the shift map. The method of claim 14,
Generating the third state map may include
And convolving the second state map into a column vector to generate a one-dimensional row vector, and repeatedly replicating the row vector to generate the third state map. The method of claim 14,
And the first attention map is a result of decoding a high level feature of the original image with a plurality of convolution layers for decoding. The method of claim 17,
Calculating a content loss of the retargeting image based on a classification score obtained by inputting the retargeting image to an image classifier;
Calculating a structure loss of the retargeting image by comparing shapes of pixels surrounding the corresponding images in the retargeting image and the original image; and
Learning convolution weights related to the retargeting image output such that the content loss and the structure loss are minimized
Further comprising, the image size adjustment method.

Images