KR20190069893A - Apparatus and method for retargeting images based on content-awareness - Google Patents

Abstract

An image size adjustment device operated by at least one processor, comprises: an encoder-decoder receiving an original image, extracting high-level features of the original image through a plurality of convolutional layers for encoding, decoding the high-level features by a plurality of convolutional layers for decoding, and outputting a first attention map including positional information for a meaningful region; and a shifter generating a second attention map in which the first attention map is scaled by a target aspect ratio, generating a shift map for shifting pixels of the original image to an output image based on the second attention map, and warping the original image with the shift map to output a retargeting image. Thus, the loss of main content and objects in the image is minimized.

Description

[0001] APPARATUS AND METHOD FOR RETARGETING IMAGES BASED ON CONTENT-AWARENESS [0002]

The present invention relates to a retouching method for adjusting image size.

So far, video retargeting methods have been studied in many ways, and can be classified into seam carving based method and warping based method. The seam-based retouching method changes the aspect ratio of an image by repeatedly removing or inserting seams in non-critical parts. The warping-based retouching method continuously transforms an input image into an image of a target size. The warping-based retagging method recursively calculates the optimal scaling factor for the local region and updates the warping image supported by the edge map and the saliency map.

As such, the prior art techniques use a low-level hand-crafted feature that is manually created, such as an edge map, to identify significant and semantic regions within the image, saliency map, and adjusts the size of the image based on the saliency map. However, in the low-level homogeneous feature, it does not accurately determine the meaningful region in the image, and thus, it is retargeted without remaining a semantically important part. For example, FIG. 1 shows a result of image retargeting using low-level features, in which a core part of an image is lost or distorted. This is a fundamental limitation of the technique of adjusting image size with low-level features.

A problem to be solved by the present invention is to extract high-level features of an input image through a neural network and to generate a attention map including positional information on a meaningful region using a high-level feature And an apparatus and method for adjusting the size of an input image using a shift map generated based on a map of the state.

Another object of the present invention is to provide an apparatus and method for calculating the content loss and the structure loss of a scaled output image and learning convolution weights related to output image generation so as to minimize the loss of the output image will be.

An image size adjustment apparatus operated by at least one processor according to one embodiment, the apparatus comprising: an input unit for receiving an original image, extracting high-level features of the original image through a plurality of convolution layers for encoding, An encoder-decoder for decoding a plurality of convolutional layers for decoding and outputting a first attention map including positional information on a meaningful region, and an encoder-decoder for scaling the first- And generating a shift map for shifting the pixels of the original image to the output image based on the map of the second state, warping the original image with the shift map, And an output shifter.

The shifter performs 1D Duplicate Convolution (1D Duplicate Convolution) of the map of the second week to generate a map of the third week in a uniform shape along the row and adds the map of the third week and the map of the second week in a weighted manner And the shift map can be generated by cumulative normalization of the final state map.

The shifter may generate a one-dimensional row vector by convoluting the second state map into a column vector, and replicate the row vector repeatedly to generate the third state map.

The column vectors may have learned weights to minimize the loss of the retargeting image.

The plurality of convolution layers for decoding may be anti-symmetric with the plurality of convolution layers for encoding.

The plurality of convolutional layers for decoding may have learned weights to minimize the loss of the retargeting image.

The image size adjustment apparatus may further include a learning device for calculating a loss of the retargeting image and for learning the convolution weights related to the retargeting image output in the encoder-decoder and the shifter so that the loss is minimized have.

The loss may include a content loss, and the learning apparatus may calculate the content loss based on the classification score acquired by inputting the retargeting image into the image classifier.

The loss may further include a structural loss, and the learning apparatus may calculate the structural loss by comparing shapes of pixels corresponding to each other in the retargeting image and the original image.

An image resizing apparatus operated by at least one processor in accordance with another embodiment, the apparatus comprising: an input means for receiving an original image, extracting high-level features of the original image through a plurality of convolutional layers for encoding, Decoder for outputting a first attention map including position information on a meaningful region by decoding the first target map with a plurality of convolutional layers for decoding, A shifter for generating a retargeting image in which the original image is scaled to the target aspect ratio on the basis of the map of the second state, and a loss of the retargeting image is calculated And weights of the plurality of convolutional layers for decoding are learned so that the loss is minimized And a humidifying device.

Wherein the loss includes a content loss and a structure loss, the learning apparatus calculates the content loss based on a classification score acquired by inputting the retargeting image into an image classifier, The structure loss can be calculated by comparing the shapes around the corresponding pixels.

The image classifier may comprise a plurality of convolution layers for encoding.

Wherein the learning apparatus inputs each of the original image and the retargeting image into a plurality of convolutional layers for encoding and then outputs the characteristic of the original image output from the lower convolution layer of the plurality of convolutional layers for encoding and The structural loss can be calculated by comparing features of the retargeting image.

There is provided a method of adjusting an image size of an image size adjusting apparatus operated by at least one processor according to another embodiment, including the steps of: obtaining a first attention map including a positional information of a meaningful region of an original image and a target aspect ratio Generating a map of a second state in which the map of the first state is scaled to the target aspect ratio, performing a one-dimensional (1D) Duplicate Convolution of the map of the second state, Generating a map of a third state, generating a shift map using a final state map weighted by adding the map of the third state and the map of the second state, and generating a shift map using the shift map, And outputting a retargeting image in which the size of the image is adjusted.

The step of generating the shift map may generate the shift map by cumulative normalization of the final state map.

The step of generating the map of the third state may generate a one-dimensional row vector by convoluting the map of the second state into a column vector, and replicating the row vector repeatedly to generate the map of the third state.

The map of the first note may be a result of decoding the high-level feature of the original image with a plurality of convolutional layers for decoding.

Wherein the image size adjustment method further comprises: calculating a content loss of the retargeting image based on a classification score obtained by inputting the retargeting image into the image classifier, calculating a content loss of the retargeting image, Calculating a structural loss of the retargeting image by comparing shapes of the retargeting images, and learning convolutional weights related to the retargeting image output so that the content loss and the structural loss are minimized.

According to the embodiment of the present invention, while preserving an area regarded as important in an image, relatively less important areas are enlarged or reduced to convert the size of the image, thereby minimizing loss of main contents and objects in the image. According to an embodiment of the present invention, the present invention can be applied to various systems requiring image size conversion.

FIG. 1 shows a result of image retargeting using a conventional low-stage feature.
FIG. 2 is a configuration diagram of an image size adjusting apparatus according to an embodiment.
FIG. 3 is a detailed configuration of an image size adjusting apparatus according to an embodiment.
4 is a diagram illustrating a method for generating a final attention map using a one-dimensional replica convolution according to an embodiment.
5 is a diagram illustrating performance of a learning method according to an embodiment.
6 is a flowchart of a method of adjusting an image size according to an exemplary embodiment of the present invention.
7 is a flowchart of a learning method according to an embodiment.
8 to 11 are comparative diagrams illustrating the performance of the present invention.
12 shows the result of evaluating the image classification accuracy according to the image size change.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, "" module, "and the like, which are described in the specification, refer to a unit for processing at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software have.

FIG. 2 is a configuration diagram of an image size adjusting apparatus according to an embodiment, FIG. 3 is a detailed configuration of an image size adjusting apparatus according to an embodiment, FIG. 4 is a diagram illustrating an image size adjusting apparatus according to an embodiment, Fig. 5 is a diagram illustrating a method for generating a final attention map. Fig.

Referring to FIGS. 2 and 3, the image size adjustment device 10 receives an original image and a target aspect ratio, resizes the original image to a target aspect ratio, and outputs a retargeting image. An image whose original image is scaled to the target aspect ratio is called a retargeting image. The original image is the input image of the image size adjustment device 10 and the retargeting image is the output image of the image size adjustment device 10. [

The image resizing apparatus 10 includes an encoder-decoder 100, which is operated by at least one processor and generates an attention map for the original image, and an attention- And a shifter 200 for converting the signal.

The learning apparatus 300 learns convolution weights related to output image generation so that the retouching image output from the shifter 200 is naturally converted while preserving a meaningful region. Learning apparatus 300 The learning apparatus 300 may be included in the image size adjustment apparatus 10, but will be described separately in order to separately explain image size adjustment and learning.

The encoder-decoder 100 receives the original image and outputs a state map including positional information on a semantic area using high-level features of the original image. The encoder-decoder 100 includes an encoder 110 for extracting high-level features of an original image using a neural network, a processor 110 for incrementally updating features output from the encoder 110, And a decoder 130 for generating an attention map by sampling.

The encoder 110 may be a neural network composed of a plurality of convolutional layers and pooling layers for pooling them. The neural network may be a Deep Convolutional Network and may use a pre-learned deep learning model (e.g., VGG16). In this case, it is assumed that the weights of the plurality of convolution layers constituting the encoder 110 are already fixed.

 Decoder 130 generates convolution maps from the features output from encoder 110 through upsampling, with convolution layers being configured inversely to the encoder. The decoder 130 has a symmetric structure with respect to the encoder 110, and some layers may be changed as needed. For example, the ReLU (Rectified Linear Unit) layers of the encoder 110 may be changed from the decoder 130 to Exponential Linear Unit (ELU) layers. On the other hand, the weights of the convolution layers constituting the decoder 130 are learned by the learning apparatus 300.

The shifter 200 receives, from the decoder 130, a state map generated from the high-level characteristics of the original image. Accordingly, the shifter 200 can generate a content-aware shift map using the attention map including the positional information on the meaningful region. This allows the shifter 200 to accurately determine a meaningful region, thereby allowing retargeting to increase or decrease a relatively less important region (e.g., background) while preserving an area that is considered important in the image .

The shifter 200 generates a shift map based on the attention map of the original image output from the decoder 130, warps the original image based on the shift map, and outputs the retagging image. The shift map S (x, y) is a map defining how much each pixel should be shifted from the input image I having the size of WxH to the output image having the size of W'xH. For the sake of explanation, the case where only the horizontal size of the image is changed is described as an example, but the vertical size is also performed in the same manner as the horizontal size change. The relationship between the input image I and the output image O through the shift map S (x, y) is expressed by Equation (1). In Equation 1, (x, y) is the spatial coordinate of the output image. When only the horizontal size of the image is changed, S is set within a range of from 0 to the horizontal change size (W-W ').

The caution map A _d output from the decoder 130 includes positional information for a meaningful region, and thus can be directly used for content-aware image retargeting. However, to maintain the overall shape of the image, pixels in similar rows must have a similar shift value, otherwise the overall shape of the image may be distorted. Therefore, the shifter 200 uses the 1D Duplicate Convolution (1D Duplicate Convolution) instead of directly using the attention map A _d of the original image output from the decoder 130 for retargeting, Use caution maps that are uniformly limited along the row (vertical).

To this end, the shifter 200 includes a shift map generating unit 210 and a warping unit 230.

First, the shift map generation unit 210 resizes the attention map A _d output from the decoder 130 to a retargeting image size. The shift map generating unit 210 may convert the attention map A _d to the retargeting image size based on the target aspect ratio. The attention map (A _r ) converted into the retargeting image size is expressed by Equation (2). In Equation (2), A _r is a caution map adjusted to the size of a retargeting image, A _d is a caution map output from the decoder 130, and R is an output image size, that is, the size of the retargeting image.

Next, the shift map generating unit 210 corrects the distortion (warping) by making one-dimensional replica convolution of the scaled caution map A _r to be uniform along the column (vertical).

Referring to FIG. 4, the shift map generator 210 converts the scaled map A _r (H x W) into an H-dimensional column vector (H x 1) to generate a one-dimensional row vector 1 x W). The weights of column vectors of the H dimension are learned by the learning apparatus 300. [ The shift map generating unit 210 repeatedly replicates a one-dimensional row vector H times and generates an attention map (A _1D ) (H x W) by one-dimensional replicated convolution. The one-dimensional replica convolution is expressed by Equation (3).

는 H차원의 열 벡터(H dimensional column vector)와의 컨볼루션을 의미한다.

는 1차원 행 벡터를 H번 반복시키는 연산을 의미한다.In Equation (3), A _r is a state map converted into the size of a retargeting image, and A _1D is a state map uniformly generated along a column (vertical) by a one-dimensional replica convolution.

Denotes a convolution with an H dimensional column vector.

Means an operation of repeating a one-dimensional row vector H times.

Since the attention map A _1D generated by the one-dimensional replica convolution is filled only with a uniform value along the column (vertical), the shift map generation unit 210 generates the shift map (A _1D ) and the state map converted into the size of the retargeting image (A _r ) are weighted to generate a final state map (A). The final state map (A) is expressed as Equation (4). In Equation (4),? Is a weighting variable.

이고, W는 입력 영상의 가로 크기이고, W'는 출력 영상의 가로 크기이다.The shift map generating unit 210 cumulatively normalizes the final state map A to generate a shift map S (x, y). The shift map S (x, y) can be calculated as shown in Equation (5). In Equation (5)

W is the width of the input image, and W 'is the width of the output image.

가 곱해 짐으로써, 시프트 지도 값의 범위가 0~

가 된다. 최종 주의 지도의 값이 낮은 부분에서는 시프트 지도의 증가량이 적기 때문에, 출력 영상에서 많은 영역을 차지하게 되며, 반대로 최종 주의 지도의 값이 높은 부분은 시프트 지도의 증가량이 커서 출력 영상에 적은 영역을 차지하게 된다.The shift map generating unit 210 performs the cumulative normalization so that the shift map monotonically increases. The input image is warped to the output image by the shift map. At this time, if the order of the pixels of the input image is changed, the output image is not output properly, so the order of the pixels should not be reversed. That is, the value of the shift map is cumulatively normalized so as to monotonically increase along the space axis (x, y direction). The value is switched from 0 to 1 by the cumulative normalization,

So that the range of the shift map value is 0 to < RTI ID = 0.0 >

. In the low value map of the final state map, the increase amount of the shift map is small, and therefore, the area occupied by the output image occupies a large area. On the other hand, .

The warping unit 230 generates a retargeting image (output image) by warping the original image (input image) using the shift map generated by the shift map generating unit 210. [ Retargeting is performed according to Equation (1).

Since the shift map has subpixel precision, linear interpolation is performed using four neighboring pixels. A differentiable loss can be applied to a linearly warped image. On the other hand, retargeting images are used to learn neural networks. To do this, the retargeting image can be zero padded to fit the input size of the neural network model.

The learning apparatus 300 calculates the loss of the output image output from the shifter 200 and learns the convolution weights related to the output image generation so as to minimize the loss of the output image. The convolution weights are weights used in the convolution operation in the encoder-decoder 100 and the shifter 200, and the weights of the plurality of convolution layers constituting the decoder 130, the H-dimensional column vectors of the shifter 200 Lt; / RTI & gt ;

The learning apparatus 300 utilizes the loss of the output image, which may include content loss and structure loss. Content loss contributes to preserving the important content of the input image. The structural loss (E _s ) contributes to making similar pixels in similar columns in the attention map.

Content loss is information indicating whether the main object in the image is well preserved in the output image, and is learning data for allowing the main object of the original image to be well preserved in the output image. If the size-converted output image from the original image is well classified by the image classifier, it can be assumed that the main part of the original image is well preserved in the output image. Accordingly, the learning apparatus 300 inputs the output image of the shifter 200 to the image classifier, and defines the content loss based on the class score output from the image classifier. The higher the classification score (i. E., The better the classification), the lower the outlet loss (i. E. The major part of the original image is well preserved in the output image). The learning apparatus 300 can use a previously learned deep learning model (for example, VGG16) as a video classifier. This may be the same as the model in which the encoder 110 outputs the feature of the original image.

와

는 정답 라벨과 분류기의 시그모이드(sigmoid) 출력이다. The learning apparatus 300 can calculate the content loss (E _c ) of the output image as shown in Equation (6). In Equation (6), C and N are the class number of the image classifier and the number of learning samples,

Wow

Is the sigmoid output of the correct answer label and sorter.

The structure loss is information for determining whether the size-converted output image is unnatural, and is learning data for learning to minimize the unnatural part in the size-converted output image. If the retargeting image and the shape of the surrounding pixels corresponding to each other in the original image are similar, it can be assumed that the size conversion is natural.

는 미리 학습된 딥 러닝 모델(예를 들면, VGG16)의 첫 번째 컨볼루션 레이어

의 출력이다. 첫 번째 컨볼루션 레이어

의 출력은 경계 지도(edge map)와 같은 낮은 단계의 특징을 표현한다.The learning apparatus 300 can calculate the structure loss (E _s ) of the output image as shown in Equation (7). The learning apparatus 300 can compute the structural loss (E _s ) of the output image by comparing characteristics of the low-level of the output image and the input image. In Equation (7)

(E.g., VGG 16) of the previously learned deep learning model

. The first convolution layer

The output of the second stage represents a low-level feature such as an edge map.

The learning apparatus 300 learns the convolution weights related to the output image generation such that the content loss (E _c ) and the structure loss (E _s ) of the output image are minimized.

5 is a diagram illustrating performance of a learning method according to an embodiment.

Referring to FIG. 5, (a) is the input image, (b) to (d) are the attention map (left side) of the input image and the retargeting image (right side) in which the input image is scaled. In this case, (b) is a retargeting image output from a model learned only by content loss, (c) is a retargeting image output from a model learned by content loss and structure loss, (d) , Which is a retargeting image acquired from a state map generated using the 1D duplicate convolution layer.

Comparing (b) and (c), it can be seen that the results obtained by learning more by using the structural loss are better than those obtained by learning only by content loss. Comparing (c) and (d), it can be seen that the best retargeting image is output when the attention map is generated using the 1D replica convolution layer.

That is, content loss contributes to preserving important contents of the input image. In the present invention, the structure loss and the 1D redundant convolution layer contribute to reducing the artifacts of the image by making pixels in the column have similar values in the image.

6 is a flowchart of a method of adjusting an image size according to an exemplary embodiment of the present invention.

Referring to FIG. 6, the image size adjuster 10 receives the original image and the target aspect ratio (S110).

The image size adjustment device 10 encodes the original image through a neural network composed of a plurality of convolution layers to extract high-level features (S120). The plurality of convolution layers used in encoding may be a pre-learned deep-run model.

The image size adjustment device 10 decodes the feature of the original image through a neural network composed of a plurality of convolution layers and outputs a state map A _d including positional information on a meaningful region in operation S 130. The plurality of convolutional layers used for decoding may have an antisymmetric structure with a plurality of convolutional layers used for encoding.

The image size adjustment device 10 generates a state map A _r that converts the attention map A _d to the retargeting image size based on the target aspect ratio at step S 140.

The image size adjustment device 10 generates a vertically aligned uniform attention map A _1D along the column (S150) by performing one-dimensional replica convolution of the converted map A _r .

The image size adjustment apparatus 10 generates a final state map A by weighting the state map A _1D generated by the one-dimensional replica convolution and the state map A _r converted into the size of the retargeting image (S160).

The image size adjustment device 10 generates a shift map using the final state map A (S170). The shift map is a map that defines how much each pixel should be shifted from the input image to the scaled output image. The image size adjustment device 10 can cumulatively normalize the final state map A to generate a shift map, as shown in Equation (5).

The image size adjustment device 10 warps the input original image using a shift map to output a retargeting image (S180).

In this way, the image size adjustment device 10 retargets through a shift map, which is a pixel unit mapping between the input image and the output image, and the state map obtained through the high-level feature is displayed on a content- As you use it, you can preserve important content while increasing or decreasing the low priority background.

7 is a flowchart of a learning method according to an embodiment.

Referring to FIG. 7, the learning apparatus 300 calculates the loss of the output image output from the image size adjustment apparatus 10, and learns the image size adjustment apparatus 10 so as to minimize the loss of the output image. Through learning by the learning apparatus 300, convolution weights related to output image generation in the image size adjustment device 10 are determined.

The learning apparatus 300 receives the original image input to the image size adjustment apparatus 10 and the retargeting image output from the image size adjustment apparatus 10 (S210).

The learning apparatus 300 calculates the content loss based on the classification score obtained by inputting the retargeting image into the image classifier (S220).

The learning apparatus 300 compares shapes of pixels corresponding to each other in the retargeting image and the original image to calculate a structural loss (S230). The learning apparatus 300 can calculate the feature (for example, a low-level feature such as a boundary map) output from the lower convolution layer in a neural network composed of a plurality of convolutional layers for extracting a feature of an image, ) Can be compared to calculate the structural loss.

The learning apparatus 300 learns the convolution weights in the image size adjusting apparatus 10 so that the content loss and the structural loss of the retargeting image are minimized (S240).

8 to 11 are comparative diagrams illustrating the performance of the present invention.

Referring to FIG. 8, (a) is an input image, and (b) is an image obtained by linear scaling a horizontal size of an image to a horizontal size of a retargeting image. (c) is a retargeting image according to the present invention. Even if the horizontal size of the image is reduced to the horizontal size of the retargeting image, it can be confirmed that the main content (automobile) of the image is almost maintained.

Referring to FIG. 9, the retargeting image is obtained by fixing the vertical length of the input image (leftmost) and continuously reducing the aspect ratio from 0.9 to 0.3.

(a) is an input image. (b) is a retargeting image by linear scaling. Since the horizontal size is linearly reduced from 0.9 to 0.3 without consideration of the content, it can be confirmed that the important content (person) of the image is not preserved . (c) are retargeting images according to the present invention. Even if the horizontal size of the image is reduced, it can be seen that the main content of the image is maintained almost intact.

Referring to Fig. 10, the vertical length of the image is fixed and the retargeting image is enlarged.

(a) is an input image. (b) is an image enlarged 1.5 times by linear interpolation, and (c) is an image 1.5 times enlarged by the present invention. (d) is an image enlarged 1.7 times by linear interpolation, and (e) is an image enlarged 1.7 times by the present invention.

Like the video reduction, the present invention can confirm that even if the video is enlarged, the main content of the video remains almost intact and the background increases.

Referring to FIG. 11, (a) is input images. (b) are retargeting images by linear interpolation. (c) are retargeting images by manual cropping. (d) are retargeting images by seam carving, which is a conventional retargeting method. (e) are retargeting images according to the present invention.

As described above, even when the vertical size of the image is reduced, it can be confirmed that the main content of the retargeting images according to the present invention is almost preserved.

12 shows the result of evaluating the image classification accuracy according to the image size change.

12, the horizontal axis represents the ratio of the size of the input image reduced, and the vertical axis represents the classification result ratio of the retargeting image to the classification result of the original image.

The present invention (our) shows that even if the size is reduced, the amount of reduction in the classification accuracy of the retargeting image is remarkably small as compared with other methods. That is, it can be seen that the main part of the image is well preserved even after retargeting.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (18)

An image size adjustment device operated by at least one processor,
Extracts a high-level feature of the original image through a plurality of convolutional layers for encoding, decodes the high-level feature into a plurality of convolutional layers for decoding, An encoder-decoder for outputting a first attention map including
Generates a shift map in which pixels of the original image are shifted to an output image on the basis of the map of the second state, A shifter for warping an original image with the shift map to output a retargeting image
And an image size adjusting device. The method of claim 1,
The shifter
1D Duplicate Convolution (1D Duplicate Convolution) of the map of the second week to generate a uniform map of the third state along the heat, weighting the map of the third state and the map of the second state, And generates the shift map by cumulative normalization of the final state map. 3. The method of claim 2,
The shifter
And generates a one-dimensional row vector by convoluting the map of the second note into a column vector, and replicates the row vector repeatedly to generate the map of the third note. 4. The method of claim 3,
Wherein the column vectors have weights learned to minimize the loss of the retargeting image. The method of claim 1,
Wherein the plurality of convolutional layers for decoding are symmetric with respect to the plurality of convolutional layers for encoding. The method of claim 1,
Wherein the plurality of convolutional layers for decoding have learned weights to minimize loss of the retargeting image. The method of claim 1,
A learning unit for calculating a loss of the retargeting image and learning convolution weights related to the output of the retargeting video in the encoder-decoder and the shifter such that the loss is minimized;
Further comprising an image size adjusting device. 8. The method of claim 7,
The loss includes content loss,
The learning device
And the content loss is calculated on the basis of a classification score obtained by inputting the retargeting image into the image classifier. 9. The method of claim 8,
Wherein the loss further comprises a structural loss,
The learning device
And compares the retargeting image and shapes of neighboring pixels in the original image to calculate the structural loss. An image size adjustment device operated by at least one processor,
Extracts a high-level feature of the original image through a plurality of convolutional layers for encoding, decodes the high-level feature into a plurality of convolutional layers for decoding, An encoder-decoder for outputting a first attention map,
A second aspect of the map in which the first aspect of the map is scaled to the target aspect ratio is generated and a retargeting image in which the original image is scaled to the target aspect ratio based on the second aspect of the map, Output shifter, and
A learning apparatus for calculating a loss of the retargeting image and learning weight values of a plurality of convolutional layers for decoding so as to minimize the loss,
And an image size adjusting device. 11. The method of claim 10,
The loss includes content loss and structural loss,
The learning device
Calculating the content loss based on a classification score obtained by inputting the retargeting image into the image classifier, and calculating the structural loss by comparing the retargeting image and the shapes around the pixels corresponding to each other in the original image, Image size adjustment device. 12. The method of claim 11,
Wherein the image classifier comprises a plurality of convolutional layers for encoding. The method of claim 12,
The learning device
The method comprising the steps of: inputting the original image and the retargeting image into a plurality of convolutional layers for encoding and outputting a feature of the original image output from a lower convolution layer of a plurality of convolutional layers for encoding, To calculate the structural loss. A method of adjusting an image size of an image size adjusting apparatus operated by at least one processor,
Receiving a first attention map and a target aspect ratio including positional information on a meaningful region of an original image,
Generating a second state map in which the first state map is scaled to the target aspect ratio,
1D Duplicate Convolution (1D Duplicate Convolution) of the map of the second state to generate a map of a third state of uniform shape along the line,
Generating a shift map using a final state map weighted by adding the map of the third state and the map of the second state, and
And outputting a retargeting image in which the size of the original image is adjusted using the shift map
And adjusting the size of the image. The method of claim 14,
The step of generating the shift map
Wherein the shift map is generated by cumulative normalization of the final state map. The method of claim 14,
The step of generating the map of the third state
Dimensional row vector by convoluting the map of the second note into a column vector and replicating the row vector repeatedly to generate the map of the third note. The method of claim 14,
Wherein the map of the first note is a result of decoding a high-level feature of the original image with a plurality of convolutional layers for decoding. The method of claim 17,
Calculating a content loss of the retargeting video based on a classification score obtained by inputting the retargeting video into a video classifier,
Calculating a structural loss of the retargeting image by comparing shapes of the retargeting image and surrounding pixels corresponding to each other in the original image, and
Learning convolution weights related to the retargeting video output such that the content loss and the structural loss are minimized
And adjusting the image size.

Images