KR20200057849A - Image processing apparatus and method for retargetting image - Google Patents

Abstract

Disclosed is a method for processing an image according to one embodiment of the present invention. The method comprises the steps of: obtaining a resizing image, wherein a size of the resizing image is changed from an original image according to a predetermined aspect ratio; inputting the resizing image into a pre-learned neural network model; and obtaining a retargeting image changed from the resizing image based on a shift map outputted from the neural network model. The method for processing the image may allow continuity between a plurality of temporally related images to be preserved.

Description

IMAGE PROCESSING APPARATUS AND METHOD FOR RETARGETTING IMAGE}

The present disclosure relates to the field of image processing. More specifically, the present disclosure relates to an image processing apparatus and method for retargeting an image so that distortion of an important portion of the image is minimized.

Devices having various resolutions and aspect ratios are being released according to the development of cameras and display devices. Under such an environment, a situation may arise in which the aspect ratio of the image captured by the camera does not match the aspect ratio of the display device. For example, when an image photographed by a camera has an aspect ratio of 4: 3, and a display device has an aspect ratio of 16: 9, the image is displayed in order to play the image on a full screen on the display device. It needs to be stretched, which causes distortion of the image.

Image retargeting technology for adjusting the aspect ratio of an image is used. The image retargeting technology refers to a technique of generating an image having a desired aspect ratio from an original image by changing positions of pixels in the original image, adding new pixels, or removing existing pixels. However, an efficient processing method for generating an image having a predetermined aspect ratio has not been proposed so far.

An image processing apparatus and method according to an embodiment has a technical problem of retargeting an image so that distortion of an important part of the image is minimized.

In addition, an image processing apparatus and method according to an embodiment has a technical problem to ensure that continuity between a plurality of time-related images is preserved.

An image processing method according to an embodiment,

Obtaining a resized image whose size is changed from the original image according to a predetermined aspect ratio; Inputting the resizing image into a pre-trained neural network model; And obtaining a retargeting image converted from the resizing image based on the shift map output from the neural network model.

In an exemplary embodiment, the neural network model includes a convolutional long short-term memory (LSTM) layer that processes a first sub-model receiving the resizing image and a feature map output from the first sub-model according to a hidden state. And a second sub-model that outputs the shift map by processing data output from the convolution LSTM layer.

In an exemplary embodiment, the convolution LSTM layer may process a resizing image corresponding to the original image in a previous time period and then process the resizing image corresponding to the original image using the updated hidden state.

In an exemplary embodiment, each of the first sub-model and the second sub-model may include at least one convolution layer that convokes input data using a predetermined filter kernel.

In an exemplary embodiment, a first location map and a second location map of pixels of the resizing image are input to the second sub-model, and the original image and the resizing are performed in the first row and the last row of the first location map. Values corresponding to ratios between vertical sizes between images are allocated, and values corresponding to ratios between horizontal sizes between the original image and the resizing image are assigned to first and last columns of the second location map. Can be assigned.

In an exemplary embodiment, values having a predetermined difference from the first row to the last row of the first location map are sequentially assigned to each row, from the first column to the last column of the second location map. Values having a predetermined difference may be sequentially assigned to each column.

In an exemplary embodiment, the step of acquiring the retargeting image may include converting the converted image based on a shift map for retargeting in the first direction from the resizing image, and a shift map for retargeting in the second direction. And acquiring the retargeting image by converting based on.

In an exemplary embodiment, the step of acquiring the retargeting image may include updating a shift map of the current time zone by combining a shift map of the previous time zone and the shift map of the current time zone; And obtaining a converted retargeting image from the resizing image based on the updated shift map of the current time zone.

In an exemplary embodiment, the neural network model may be trained such that first loss information corresponding to a result of applying a mask to a difference between a ground truth (GT) image and a training retargeting image is reduced.

In an exemplary embodiment, the training resizing image is transformed according to a shift map for retargeting in the first direction, and second loss information corresponding to a result of applying a first background mask to a difference between the obtained image and the GT image. , Reconstruct the training resizing image according to the shift map for retargeting in the first direction and the shift map for retargeting in the second direction, and remove the difference between the training retargeting image and the GT image obtained. The neural network model may be trained such that third loss information corresponding to a result of applying one foreground mask is reduced.

In an exemplary embodiment, the first foreground mask is obtained by padding a second foreground mask corresponding to a training original image, and the first background mask includes a second background mask corresponding to the second foreground mask. The third background mask corresponding to the third foreground mask obtained by changing the size of the second foreground mask may be obtained by combining each other.

In an exemplary embodiment, the GT image may include a foreground object obtained by applying the second foreground mask to the training original image and a background object obtained by applying the third background mask to the training resizing image. Can be obtained in combination.

In an exemplary embodiment, the neural network model is configured such that the fourth loss information derived based on the relationship between the shift map corresponding to the original training image in the previous time zone and the shift map corresponding to the original training image in the current time zone is reduced. Can be trained.

An image processing apparatus according to an embodiment,

Processor; And a memory for storing a pre-trained neural network model and at least one program, wherein the processor is configured to display a resized image whose size is changed from an original image according to a predetermined aspect ratio as the at least one program stored in the memory is executed. Acquiring, inputting the resizing image into the neural network model, and obtaining a retargeting image converted from the resizing image based on a shift map output from the neural network model.

In an exemplary embodiment, the image processing apparatus includes a display that outputs the retargeting image, and the processor can determine an aspect ratio of the resizing image based on the aspect ratio of the display.

In an exemplary embodiment, the processor converts an image converted based on a shift map for retargeting from the resizing image to a first direction based on a shift map for retargeting in the second direction, and Retargeting images can be obtained.

In an exemplary embodiment, the processor updates the shift map of the current time zone by combining the shift map of the previous time zone and the shift map of the current time zone, and based on the updated shift map of the current time zone, the Resizing images can be converted.

In an exemplary embodiment, the processor inputs a training resizing image corresponding to a training original image into the neural network model, wherein the neural network model is a result of applying a mask to the difference between the training retargeting image and the GT image. It can be trained to reduce the loss information corresponding to.

In an exemplary embodiment, the image processing device may further include a communication module that receives data of the pre-trained neural network model from an external device.

The apparatus and method for processing an image according to an embodiment may retarget the image to minimize distortion of an important part in the image.

In addition, the apparatus and method for processing an image according to an embodiment may ensure that continuity between a plurality of temporally related images is preserved.

However, the effects that the image processing apparatus and method according to an embodiment can achieve are not limited to those mentioned above, and other effects not mentioned are common in the art to which the present disclosure pertains from the following description. It will be clearly understood by those with knowledge.

A brief description of each drawing is provided to better understand the drawings cited herein.
1 is a view for explaining an environment to which an image processing apparatus according to an embodiment is applied.
2 is a flowchart illustrating an image processing method according to an embodiment.
3 is a view for explaining a retargeting process according to an embodiment.
4 is a diagram for explaining the structure of a neural network model according to an embodiment.
5 is a view for explaining the structure of a neural network model according to an embodiment in detail.
6 and 7 are exemplary views showing a location map input to the second sub-model.
8 is an exemplary diagram showing a mask used to generate a GT image corresponding to a training original image.
9 is a diagram for explaining a method of generating a GT image based on a training original image and a training resizing image.
10 and 11 are views for explaining a method of generating a foreground mask and a background mask used for training a neural network model.
12 is a diagram for explaining loss information used for training a neural network model.
13 is a block diagram showing the configuration of an image processing apparatus according to an embodiment.

Since the present disclosure can be modified in various ways and have various embodiments, specific embodiments will be illustrated in the drawings and described through detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that all modifications, equivalents, and substitutes included in the spirit and scope of the present disclosure are included.

In describing the embodiments, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter, detailed descriptions thereof will be omitted. In addition, the numbers (for example, first, second, etc.) used in the description of the embodiment are merely identification numbers for distinguishing one component from other components.

In addition, when one component is referred to as “connected” or “connected” with another component in the present specification, the one component may be directly connected to or directly connected to the other component, but is specifically opposed It should be understood that as long as there is no description to be made, it may be connected or connected through another element in the middle.

In addition, in this specification, two or more components represented by '~ unit (unit)', 'module', etc. are combined into one component, or two or more components are divided into more detailed functions. It may be differentiated into. In addition, each of the components to be described below may additionally perform some or all of the functions of other components in addition to the main functions in charge of them, and some of the main functions of each component are different. Needless to say, it can also be carried out by components.

Hereinafter, embodiments according to the technical spirit of the present disclosure will be described in detail.

1 is a view for explaining an environment to which the image processing apparatus 100 according to an embodiment is applied.

The image processing apparatus 100 according to an embodiment acquires an original image and acquires a retargeting image based on the original image. The image processing apparatus 100 may generate a retargeting image having a predetermined aspect ratio from an original image having an aspect ratio different from a predetermined aspect ratio. The image processing apparatus 100 may display the retargeting image through the display apparatus 200.

In one embodiment, the image processing apparatus 100 may be included in the display apparatus 200. Alternatively, in one embodiment, the image processing apparatus 100 may be implemented as a server, and in this case, the image processing apparatus 100 may transmit the retargeting image to the display apparatus 200 through a network. In one embodiment, the network may include a wired network and a wireless network, specifically, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and the like. It can include a variety of networks. In addition, the network may include a known World Wide Web (WWW). However, the network is not limited to the networks listed above, and may include at least part of a known wireless data network, a known telephone network, and a known wired / wireless television network.

When generating a retargeting image, the image processing apparatus 100 according to an embodiment may enable an important portion of the original image to maintain the aspect ratio within the original image as much as possible even in the retargeting image.

2 is a flowchart illustrating an image processing method according to an embodiment, and FIG. 3 is a diagram illustrating a retargeting process according to an embodiment.

In step S210, the image processing apparatus 100 obtains a resized image 20 whose size is changed from the original image 10 according to a predetermined aspect ratio.

The size of the resizing image 20 may be larger than the size of the original image 10. For example, the vertical size of the resizing image 20 is the same as the vertical size of the original image 10, but the horizontal size of the resizing image 20 may be larger than the horizontal size of the original image 10. .

The predetermined aspect ratio may be determined based on an aspect ratio of a display outputting a retargeting image, which will be described later. For example, the predetermined aspect ratio may be determined to be the same as the aspect ratio of the display outputting the retargeting image.

The image processing apparatus 100 may obtain a resized image 20 whose size is changed by scaling the original image 10. In one example, the image processing apparatus 100 may obtain a resized image 20 whose size is changed by interpolating pixels in the original image 10.

In step S220, the image processing apparatus 100 inputs the resizing image 20 into the pre-trained neural network model 300. The neural network model 300 may be, for example, a deep learning model composed of a plurality of layers. The structure and training method of the neural network model 300 will be described in detail with reference to FIG. 8 and below.

In step S230, the image processing apparatus 100 acquires a shift map 90 output from the neural network model 300. The shift map 90 includes information for changing the positions of pixels in the resizing image 20. For example, the shift map 90 may include information about where each pixel in the resizing image 20 should be moved. The shift map 90 may have the same size as the size of the resizing image 20.

In operation S240, the image processing apparatus 100 acquires the converted retargeting image from the resizing image 20 based on the shift map 90.

In the neural network model 300 according to an embodiment, the neural network model 300 re-targets a significant portion of the original image 10 even if the resizing image 20 is retargeted according to the output shift map 90. Can be trained to stay within.

4 is a diagram for explaining the structure of a neural network model 300 according to an embodiment.

Referring to FIG. 4, the neural network model 300 may include a first sub-model 310, a convolutional LSTM layer 320 and a second sub-model 330.

The first sub-model 310 receives and processes the resizing image 20. The output data of the first sub-model 310, for example, a feature map, is input into a convolutional long short-term memory (LSTM) layer. Then, data output from the convolution LSTM layer 320 is input to the second sub-model 330. In the second sub-model 330, a shift map 90 for retargeting the resizing image 20 is output.

In one example, the second sub-model 330 includes a foreground mask 92 for extracting a foreground object in the resizing image 20 and a first shift map for retargeting the resizing image 20 along the first direction ( 94) and the second shift map 96 for retargeting the resizing image 20 along the second direction may be output.

For example, the first shift map 94 is used to retarget pixels in the resizing image 20 in the vertical direction, and the second shift map 96 repixels pixels in the resizing image 20 in the horizontal direction. It can be used to target. In one embodiment, in order to change only the vertical size of the original image 10 in order to convert the aspect ratio of the original image 10, the second sub-model 330 is configured to output only the first shift map 94 Can be. Conversely, when only the horizontal size of the original image 10 is to be changed to convert the aspect ratio of the original image 10, the second sub-model 330 may be configured to output only the second shift map 96. .

The foreground mask 92 output from the second sub-model 330 is for evaluating whether the neural network model 300 is properly trained, and according to an embodiment, the second sub-model 330 has a foreground mask 92 It may be configured not to be output.

The convolution LSTM layer 320 receives the data output from the first sub-model 310 and processes it according to a hidden state, and transmits the processed result to the second sub-model 330. The hidden state in the convolution LSTM layer 320 is updated after processing the data output from the first sub-model 310, and the updated hidden state can be used for image processing in a next time period.

For example, the resizing image 20 corresponding to the original image 10 in time t is processed in the first sub-model 310, and data output from the first sub-model 310 is a convolution LSTM layer 320 ) According to the hidden state. Here, the hidden state used to process the resizing image 20 in the t time zone may be an updated hidden state after processing the resizing image 20 in the t-1 time period in the convolution LSTM layer 320. When the data output from the first sub-model 310 for the resizing image 20 in the t time zone is processed according to the hidden state in the convolution LSTM layer 320, the hidden state is updated, and the updated hidden state is t + It is used to process the resizing image 20 in one hour.

That is, in the neural network model 300 according to an embodiment, the hidden state of the convolution LSTM layer 320 used to process the resizing image 20 in the current time period is used to process the resizing image 20 in the next time period. Since it is used, in retargeting a video composed of several frames, continuity between frames can be maintained.

5 is a view for explaining the structure of the neural network model 300 according to an embodiment in detail.

Referring to FIG. 5, the first sub-model 310 may include a plurality of convolutional layers. Each convolutional layer convolutionally processes input data using a predetermined size and number of filter kernels, and then transmits the convolutional output data to the next layer. FIG. 5 shows that the first sub-model 310 includes five convolutional layers, but the number of convolutional layers may be variously changed within the scope of an autonomous ring by those skilled in the art. In addition, although not shown in FIG. 5, an activation layer and / or a pooling layer may be located before or after at least some of the convolutional layers in the first sub-model 310.

When an active layer and / or a pooling layer is located in front of any one convolutional layer, output data of the active layer and / or a pooling layer may be input to any one of the convolutional layers. In addition, when an activation layer and / or a pooling layer is located behind one of the convolutional layers, the output data of the one of the convolutional layers may be input as an activation layer and / or a pooling layer. have.

The active layer may impart a non-linear characteristic to the output result of the previous layer. The activation layer may use an activation function. The activation function may include a sigmoid function, a tanh function, and a rectified linear unit (ReLU) function, but is not limited thereto.

The second sub-model 330 may include a plurality of deconvolution layers. Each deconvolution layer convolutionally processes the input data using a predetermined size and number of filter kernels, and then delivers the convolutional output data to the next layer. FIG. 5 shows that the second sub-model 330 includes five deconvolution layers for each of the data finally output, but the number of deconvolution layers is variously changed within the scope of an autonomous ring by those skilled in the art. Can be.

As described above, when the foreground mask 92, the first shift map 94, and the second shift map 96 are output from the second sub-model 330, the foreground mask 92, the first shift map ( 94) and deconvolution layers for the output of each of the second shift map 96 may be arranged in parallel with each other. When only the first shift map 94 and the second shift map 96 are output from the second sub-model 330, deconvolution layers for outputting the foreground mask 92 (dconv4-1, dconv5-1) May not be included in the second sub-model 330. Similarly, when only the second shift map 96 is output from the second sub-model 330, the deconvolution layers for outputting the foreground mask 92 (dconv4-1, dconv5-1) and the first shift map The deconvolution layers dconv4-2 and dconv5-2 for the output of 94 may not be included in the second sub-model 330.

In addition, although not illustrated in FIG. 5, an activation layer and / or a pooling layer may be located before or after at least some of the deconvolution layers in the second sub-model 330.

In one embodiment, a location map indicating the positions of pixels in the resizing image 20 may be input to at least one deconvolution layer among the deconvolution layers included in the second sub-model 330.

6 and 7 are exemplary views showing a location map input to the second sub-model 330.

The first location maps 610 and 710 and the second location maps 630 and 730 may be input as at least one deconvolution layer among the deconvolution layers included in the second sub-model 330. The location maps 610 and 710 may indicate vertical positions of pixels in the resizing image 20, and the second location maps 630 and 730 may indicate horizontal positions of pixels in the resizing image 20.

In one embodiment, when the image processing apparatus 100 generates the first location maps 610 and 710 and the second location maps 630 and 730, the first location maps 610 and 710 and the second location map ( 630 and 730 may include size ratio information between the original image 10 and the resizing image 20.

In one example, the image processing apparatus 100 includes the original image 10 and the pixels included in the first row 611 and 711 and the last row 619 and 719 of the first location maps 610 and 710. A value corresponding to a ratio between vertical sizes of the resizing image 20 may be assigned. In addition, the image processing apparatus 100 includes pixels included in the first column 631 and 731 and the last column 639 and 739 of the second location maps 630 and 730, and the original image 10 and the resizing image. A value corresponding to the ratio between the horizontal sizes of 20 may be assigned.

6 shows a first location map 610 and a second location map 630 when the size and aspect ratio of the original image 10 and the resizing image 20 are the same.

Referring to FIG. 6, pixels included in the first row 611 of the first location map 610 include values corresponding to a ratio between vertical sizes of the original image 10 and the resizing image 20, that is, 1 is assigned, but the sign can be determined as-. And, the pixels included in the last row 619 of the first location map 610 are assigned a value corresponding to the ratio between the vertical sizes of the original image 10 and the resizing image 20, that is, 1, The sign can be determined as +. Values having a predetermined difference from the first row 611 of the first location map 610 to the last row 619 may be sequentially assigned to pixels of each row. For example, the pixels in the second row of the first location map 610 may be assigned values greater than a assigned to pixels in the first row 611 as much as a, and the first location map ( The pixels in the third row of 610 may be assigned values larger than a allocated to the pixels in the second row.

Pixels included in the first column 631 of the second location map 630 are assigned a value corresponding to a ratio between the horizontal sizes of the original image 10 and the resizing image 20, that is, 1 The sign can be determined by-. In addition, a value corresponding to the ratio between the horizontal sizes of the original image 10 and the resizing image 20 is assigned to the pixels included in the last column 639 of the second location map 630, that is, 1 The sign can be determined as +. Values having a predetermined difference from the first column 631 of the second location map 630 to the last column 639 may be sequentially assigned to pixels of each column. For example, the pixels in the second column of the second location map 630 may be assigned values greater than a assigned to pixels in the first column 631 as a, and the second location map 630 ), Pixels larger than a value assigned to pixels in the second column may be assigned to pixels in the third column.

7 shows the first position when the vertical size of the resizing image 20 and the original image 10 are the same, and the horizontal size of the resizing image 20 is 1.5 times larger than the horizontal size of the original image 10. The map 710 and the second location map 730 are shown.

Referring to FIG. 7, pixels included in the first row 711 of the first location map 710 include values corresponding to a ratio between vertical sizes of the original image 10 and the resizing image 20, that is, 1 is assigned, but the sign can be determined as-. In addition, a value corresponding to a ratio between vertical sizes of the original image 10 and the resizing image 20 is assigned to the pixels included in the last row 719 of the first location map 710, namely 1 The sign can be determined as +. Values having a predetermined difference from the first row 711 of the first location map 710 to the last row 719 may be sequentially assigned to pixels of each row. For example, the pixels of the second row of the first location map 710 may be assigned values greater than a assigned to pixels of the first row 711 by a, and the first location map ( Pixels in the third row of 710 may be assigned values larger than a that are allocated to the pixels in the second row.

Pixels included in the first column 731 of the second location map 730 are assigned a value corresponding to a ratio between horizontal sizes of the original image 10 and the resizing image 20, that is, 1.5. The sign can be determined by-. In addition, a value corresponding to the ratio between the horizontal sizes of the original image 10 and the resizing image 20 is assigned to the pixels included in the last column 739 of the second location map 730, that is, 1.5. The sign can be determined as +. Values having a predetermined difference from the first column 731 of the second location map 730 to the last column 739 may be sequentially assigned to pixels of each row. For example, the pixels in the second column of the second location map 730 may be assigned values greater than b, which are assigned to pixels in the first column 731, and a second location map 730. ), Pixels larger than b may be assigned to pixels in the third column.

The difference a of pixel values allocated along each column of the second location map 630 of FIG. 6 is greater than a difference b of the pixel values allocated along each column of the second location map 730 of FIG. 7. It can be small. Because, when the horizontal size of the second location map 630 of FIG. 6 and the second location map 730 of FIG. 7 are equal to each other, the first column 631 of each second location map 630, 730, This is because the pixel values assigned to the 731) and the last columns 639 and 739 are fixed and the pixel values are allocated to the columns at equal intervals.

6 and 7, the pixel values of the first row 611, 711 of the first location map 610, 710 have a sign of-, and the last row of the first location map 610, 710 ( Although the pixel values of 619 and 719 are described as having a sign of +, in one example, the pixel values of the first row 611 and 711 of the first location maps 610 and 710 have a sign of +, Pixel values of the last rows 619 and 719 of the 1 location map 610 and 710 may have a sign of-. Similarly, pixel values of the first column 631, 731 of the second location map 630, 730 have a sign of-, and pixel values of the last column 639, 739 of the second location map 630, 730 Although they are described as having a sign of +, in one example, the pixel values of the first column 631, 731 of the second position map 630, 730 have a sign of +, and the second position map 630, 730 ), The pixel values of the last columns 639 and 739 may have a sign of-.

In one embodiment, the image processing apparatus 100 corresponds to a resizing image 20 of a previous time zone in order to maintain continuity between retargeting images corresponding to original images 10 of various time zones. Retargeting the resizing image 20 of the current time zone by combining the shift map 90 output from the 300 and the shift map 90 output from the neural network model 300 corresponding to the resizing image 20 of the current time zone. You may.

For example, the image processing apparatus 100 may correspond to the shift map 90 output from the neural network model 300 corresponding to the resizing image 20 in the t-1 time zone, and the resizing image 20 in the t time zone. After combining the shift map 90 output from the neural network model 300, the resizing image 20 in time t may be retargeted. The update process of the shift map 90 for each time zone may be expressed by Equation 1 below.

[Equation 1]

Sb '(t) = (t / (t + 1)) * Sb' (t-1) + (1 / (t + 1)) * Sb (t)

Sf '(t) = (t / (t + 1)) * Sf' (t-1) + (1 / (t + 1)) * Sf (t)

In Equation 1, Sb (t) is the first shift map 94 output from the neural network model 300 corresponding to the resizing image 20 in time t, and Sb '(t) is updated from Sb (t). After the first shift map 94 and Sb '(t-1) are output from the neural network model 300 corresponding to the t-1 time zone resizing image 20, the updated first shift map 94 and Sf (t) is the second shift map 96 output from the neural network model 300 corresponding to the resizing image 20 in the t time zone, and Sf '(t) is the second shift map 96 updated from Sf (t). ), Sf '(t-1) means the updated second shift map 96 after being output from the neural network model 300 corresponding to the t-1 time zone resizing image 20. The first shift map 94 is a shift map for retargeting the resizing image 20 in the vertical direction, and the second shift map 96 is a shift map for retargeting the resizing image 20 in the horizontal direction. Can be.

Hereinafter, a method of training the neural network model 300 will be described with reference to FIGS. 8 to 12.

8 is an exemplary view showing a mask used to generate a GT image 930 corresponding to the training original image 910, and FIG. 9 is an original training image 910 and a training resizing image 920. A diagram for explaining a method of generating a GT image 930 based on the above.

In one embodiment, the mask may be a binary mask. Pixels included in the mask may have one of two values. The foreground mask is a mask for extracting a foreground object in an image, and may be used, for example, to extract only an important region in the image. The background mask is a mask for extracting a background object in an image, and can be used, for example, to extract only a non-critical region in the image.

The image processing apparatus 100 may generate a ground truth (GT) image 930 corresponding to the original training image 910 for training the neural network model 300. The GT image 930 may have the same aspect ratio and size as the training resizing image 920.

The image processing apparatus 100 may obtain the first foreground mask 32 and the second background mask 44 as shown in FIG. 8 to generate the GT image 930.

The image processing apparatus 100 may obtain a first foreground mask 32 corresponding to the training original image 910. The image processing apparatus 100 may receive an original image 910 for training and a first foreground mask 32 corresponding thereto from an external device or input from an administrator. The original training image 910 and the first foreground mask 32 may have the same size and the same aspect ratio. In one embodiment, the image processing apparatus 100 analyzes the original image 910 for training according to a pre-stored algorithm, identifies an important part in the original image 910 for training, and extracts the identified part. 1 A foreground mask 32 may be created.

The image processing apparatus 100 may obtain a second foreground mask 34 that stretches the first foreground mask 32 according to a predetermined aspect ratio. The image processing apparatus 100 may generate the second foreground mask 34 by interpolating the first foreground mask 32.

The image processing apparatus 100 may acquire the first background mask 42 corresponding to the first foreground mask 32 by inverting pixel values of the first foreground mask 32. The first background mask 42 can be used to generate the third background mask 46, as described below. Then, the image processing apparatus 100 may acquire the second background mask 44 corresponding to the second foreground mask 34 by inverting the pixel values of the second foreground mask 34. Here, inverting the pixel values of the foreground mask may mean changing one value assigned to pixels in the foreground mask to another of two values that can be assigned to the mask. For example, a pixel having a value of 1 in the first foreground mask 32 may have a value of 0 in the first background mask 42. In one embodiment, the second background mask 44 may be obtained by changing the size of the first background mask 42.

Referring to FIG. 9, the image processing apparatus 100 applies the first foreground mask 32 to the training original image 910 to extract the foreground object 915 from the training original image 910, and for training A second background mask 44 is applied to the resizing image 920 to extract the background object 927 from the training resizing image 920. Then, the image processing apparatus 100 generates a GT image 930 by combining the foreground object 915 extracted from the training original image 910 and the background object 927 extracted from the training resizing image 920. do. Here, combining means that one GT image 930 is generated by combining the object 915 extracted from the training original image 910 and the object 927 extracted from the training resizing image 920. Can be.

As the foreground object 925 included in the training resizing image 920 is changed in size ratio compared to the foreground object 915 included in the training original image 910, the GT image ( An empty area 935 exists in the 930. In other words, when the second background mask 44 is applied to the training resizing image 920, the foreground object 925 included in the training resizing image 920 remains as it is included in the training resizing image 920. It is not extracted, and has a pixel value of 0, for example, through the second background mask 44. Since the size of the foreground object 915 included in the training original image 910 is smaller than the size of the foreground object 925 included in the training resizing image 920, the foreground extracted from the training original image 910 When the object 915 is combined with the background object 927 extracted from the training resizing image 920, a portion having a pixel value of 0 becomes an empty space 935.

The image processing apparatus 100 inputs a training resizing image 920 into the neural network model 300 for training the neural network model 300. The neural network model 300 outputs a shift map corresponding to the training resizing image 920. Then, the neural network model 300 may train to reduce the difference between the training retargeting image and the GT image 930 converted from the training resizing image 920 based on the shift map. In one example, the neural network model 300 may be trained to reduce loss information corresponding to a result of applying a mask to a difference between the training retargeting image and the GT image 930. The mask for deriving the loss information may be generated based on the first foreground mask 32, the first background mask 42, and the second background mask 44 of FIG. 8, as described above. This will be described with reference to 11.

10 and 11 are diagrams for explaining a method of generating a foreground mask and a background mask used for training the neural network model 300.

Referring to FIG. 10, the image processing apparatus 100 may pad the first foreground mask 32 to generate a third foreground mask 36. The image processing apparatus 100 may pad the first foreground mask 32 such that the size and aspect ratio of the third foreground mask 36 are the same as the size and aspect ratio of the GT image 930. Pixels newly generated according to padding may be assigned the same values as the surrounding pixel values included in the first foreground mask 32. The third foreground mask 36 may be used to extract foreground objects from the GT image 930.

Referring to FIG. 11, the image processing apparatus 100 may generate a third background mask 46 by combining the first background mask 42 and the second background mask 44. The image processing apparatus 100 may generate one third background mask 46 by combining the first background mask 42 and the second background mask 44. The third background mask 46 may be used to extract a background object from the GT image 930.

A method of training the neural network model 300 using the GT image 930, the third foreground mask 36 and the third background mask 46 will be described with reference to FIG. 12.

12 is a diagram for describing loss information used for training the neural network model 300.

The image processing apparatus 100 inputs a training resizing image 920 corresponding to the training original image 910 into the neural network model 300, and in the neural network model 300, the first of the training resizing image 920. A first shift map 994 for retargeting in the direction (eg, portrait) and a second for retargeting in the second direction (eg, landscape) of the training resizing image 920 The shift map 996 is output.

The training resizing image 920 is retargeted based on the first shift map 994, and a third background mask 46 is applied to the difference between the retargeted image and the GT image 930. The first loss information 1210 corresponding to the result of applying the third background mask 46 may be extracted. The neural network model 300 may be trained such that the first loss information 1210 is reduced. In one example, the neural network model 300 may be trained such that the first loss information 1210 is minimized. The first loss information 1210 may be calculated by Equation 2 below.

[Equation 2]

In Equation 2, Lb is the first loss information 1210, (x, y) is a position of a pixel in the image, I is a training resizing image 920, Sb is a first shift map 994, GT is The GT image 930 and Mb represent the third background mask 46.

In addition, the training resizing image 920 is retargeted based on the first shift map 994 and the second shift map 996, and the training resizing image 920 is generated by retargeting the training retargeting image. The third foreground mask 36 is applied to the difference between the and the GT image 930. The second loss information 1230 corresponding to the result of applying the third foreground mask 36 may be extracted. The neural network model 300 may be trained to reduce the second loss information 1230. In one example, the neural network model 300 may be trained such that the second loss information 1230 is minimized. The second loss information 1230 may be calculated by Equation 3 below.

[Equation 3]

In Equation 3, Lf is the second loss information 1230, (x, y) is the position of the pixel in the image, I is the training resizing image 920, Sb is the first shift map 994, Sf is The second shift map 996, GT is a GT image 930, and Mf is a third foreground mask 36.

In one embodiment, the neural network model 300 is derived based on a relationship between a shift map corresponding to the training resizing image 920 of the previous time zone and a shift map corresponding to the training resizing image 920 of the current time zone. The third loss information can be trained to be reduced. In one example, the neural network model 300 may be trained to minimize third loss information.

The third loss information is calculated to maintain continuity between retargeted frames in a video composed of several frames. The third loss information may correspond to a flow consistency loss. The third loss information may be calculated according to Equation 4 below.

[Equation 4]

In Equation 4, warp denotes a warping function using a flownet, and Sb (t) is a first shift map output by the neural network model 300 after receiving a training resizing image 920 for t time zone ( 994), Sb (t + 1) is a first shift map 994 received by the resizing image 920 for training in the t + 1 time zone and output by the neural network model 300, and Sf (t) is training in the t time zone. The second shift map 996, Sf (t + 1) output by the neural network model 300 after receiving the resizing image 920 for use, and Sf (t + 1) receive the training resizing image 920 for the t + 1 time zone and receive the neural network model ( 300) represents the second shift map 996 output, and Mo is an occlusion mask, and is calculated by Equation 5 below.

[Equation 5]

In Equation 5, a denotes a predetermined weight, and I (t) is a training resizing image 920 corresponding to the training image 910 of time t, and I (t + 1) is time t + 1. A training resizing image 920 corresponding to the original training image 910 of FIG.

The neural network model 300 may be trained so that the final loss information combining the first loss information 1210, the second loss information 1230, and the third loss information is reduced. The final loss information may be determined according to Equation 6 below.

[Equation 6]

Final loss information = a · first loss information + b · second loss information + c · third loss information

In Equation 6, a, b, and c are predetermined weights applied to each loss information.

In the neural network model 300 according to an embodiment, the difference between the background of the image retargeting the training resizing image 920 in the vertical direction based on the first shift map 994 and the background of the GT image 930 And GT images 930 trained to be reduced or minimized, and retargeting the training resizing image 920 in the vertical and horizontal directions based on the first shift map 994 and the second shift map 996. ) Can be trained to reduce or minimize the difference between foregrounds. In particular, the third foreground mask 36 is applied to the difference between the retargeting image for training and the GT image 930, and the third foreground mask 36 is the same as the foreground object included in the training original image 910. Since the size of the object can be extracted, and the GT image 930 includes an object of the same size as the size of the foreground object included in the training original image 910, in the end, the neural network model 300 is used for training The foreground object included in the targeting image may be trained to be the same or similar to the foreground object included in the training original image 910.

13 is a block diagram showing the configuration of an image processing apparatus 100 according to an embodiment.

Referring to FIG. 13, the image processing apparatus 100 according to an embodiment may include a memory 110 and a processor 130.

The memory 110 may store at least one program and neural network model 300. The processor 130 may perform an image processing operation including retargeting of an image as at least one program stored in the memory 110 is executed.

Specifically, the processor 130 may obtain a resized image whose size is changed from the original image according to a predetermined aspect ratio, and input the resized image to the neural network model 300. The processor 130 may obtain a converted retargeting image from the resizing image based on the shift map output from the neural network model 300. A method of inputting a resizing image into the neural network model 300 and generating a retargeting image based on a shift map output from the neural network model 300 has been described in detail above, and thus description thereof is omitted here.

Although not shown in FIG. 13, the processor 130 may transmit the retargeting image to the display so that the retargeting image is reproduced by the display. In addition, in obtaining a resizing image from the original image, the processor 130 may determine an aspect ratio of the resizing image based on the aspect ratio of the display. For example, the processor 130 may determine the aspect ratio of the resizing image to be the same as the aspect ratio of the display.

The image processing apparatus 100 according to an embodiment may be included in a smart phone, a tablet PC, a desktop PC, a television, a smart watch, a smart glass, and a server.

Training of the neural network model 300 stored in the image processing apparatus 100 may be performed in various devices.

In one embodiment, training of the neural network model 300 is performed on an external device (eg, a server, etc.), and the trained neural network model 300 may be stored in the image processing apparatus 100.

In one embodiment, training of the neural network model 300 is performed on an external device (for example, a server), and data of the trained neural network model 300 is communicated through the network to the communication module 100 of the image processing apparatus 100 ( (Not shown).

Alternatively, in one example, while the neural network model 300 is stored in the image processing apparatus 100, training of the neural network model 300 may be performed in the image processing apparatus 100.

Alternatively, in one example, after training of the neural network model 300 is performed on an external device (for example, a server), information representing the internal weight of the neural network model 300 is image-processed from the external device 100 It can also be transmitted over the network. The image processing apparatus 100 may train the pre-stored neural network model 300 according to the received internal weight information.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the created program can be stored in a medium.

The medium may be a computer that continuously stores executable programs or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a combination of single or several hardware, and is not limited to a medium that is directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like. In addition, examples of other media include an application store for distributing applications, a site for distributing or distributing various software, and a recording medium or storage medium managed by a server.

As described above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments and has ordinary knowledge in the art within the scope of the technical idea of the present disclosure. Various modifications and changes are possible by the ruler.

100: image processing device
110: memory
130: processor

Claims (20)

Obtaining a resized image whose size is changed from the original image according to a predetermined aspect ratio;
Inputting the resizing image into a pre-trained neural network model; And
And obtaining a converted retargeting image from the resizing image based on the shift map output from the neural network model.
According to claim 1,
The neural network model,
The first sub-model receiving the resizing image, the convolutional long short-term memory (LSTM) layer for processing the feature map output from the first sub-model according to the hidden state, and the data output from the convolutional LSTM layer And a second sub-model for processing and outputting the shift map.
According to claim 2,
The convolution LSTM layer,
And processing the resizing image corresponding to the original image in the previous time period, and then processing the resizing image corresponding to the original image using the updated hidden state.
According to claim 2,
Each of the first sub-model and the second sub-model,
And at least one convolutional layer for convolutionally processing input data using a predetermined filter kernel.
According to claim 2,
The first location map and the second location map of pixels of the resizing image are input to the second sub-model,
Values corresponding to ratios between vertical sizes between the original image and the resizing image are allocated to the first row and the last row of the first location map,
The first and last columns of the second location map are assigned values corresponding to a ratio between horizontal sizes between the original image and the resizing image.
The method of claim 5,
Values having a predetermined difference from the first row to the last row of the first location map are sequentially assigned to each row,
An image processing method, characterized in that values having a predetermined difference from the first column to the last column of the second location map are sequentially assigned to each column.
According to claim 1,
Acquiring the retargeting image,
And converting the converted image based on the shift map for retargeting in the first direction from the resizing image, and obtaining the retargeting image by converting the converted image based on the shift map for retargeting in the second direction. Image processing method characterized in that.
According to claim 1,
Acquiring the retargeting image,
Updating the shift map of the current time zone by combining the shift map of the previous time zone and the shift map of the current time zone; And
And obtaining a converted retargeting image from the resizing image based on the updated shift map of the current time zone.
According to claim 1,
An image processing method characterized in that the neural network model is trained such that first loss information corresponding to a result of applying a mask to a difference between a ground truth (GT) image and a training retargeting image is reduced.
According to claim 1,
Second loss information corresponding to a result of applying a first background mask to a difference between an acquired image and a GT image by converting a training resizing image according to a shift map for retargeting in a first direction,
The difference between the training retargeting image and the GT image obtained by converting the training resizing image according to the shift map for retargeting in the first direction and the shift map for retargeting in the second direction is the first. The neural network model is trained so that the third loss information corresponding to the result of applying the foreground mask is reduced.
The method of claim 10,
The first foreground mask is obtained by padding a second foreground mask corresponding to the training original image,
The first background mask is obtained by combining a second background mask corresponding to the second foreground mask and a third background mask corresponding to a third foreground mask obtained by changing the size of the second foreground mask. Image processing method characterized in that.
The method of claim 11,
The GT video,
An image processing method comprising obtaining a foreground object obtained by applying the second foreground mask to the training original image and a background object obtained by applying the third background mask to the training resizing image. .
According to claim 1,
The neural network model is trained such that the fourth loss information derived based on the relationship between the shift map corresponding to the original training image in the previous time zone and the shift map corresponding to the original training image in the current time period is reduced. Image processing method.
A program stored in a medium to execute the image processing method of any one of claims 1 to 13 in combination with hardware.
Processor; And
A pre-trained neural network model and a memory storing at least one program,
The processor, as the at least one program stored in the memory is executed,
Obtain a resized image whose size is changed from the original image according to a predetermined aspect ratio,
Input the resizing image into the neural network model,
An image processing apparatus characterized by acquiring a retargeting image converted from the resizing image based on a shift map output from the neural network model.
The method of claim 15,
The image processing device,
It includes a display for outputting the retargeting image,
The processor,
And an aspect ratio of the resizing image is determined based on the aspect ratio of the display.
The method of claim 15,
The processor,
And converting the converted image based on the shift map for retargeting in the first direction from the resizing image, and obtaining the retargeting image by converting the transformed image based on the shift map for retargeting in the second direction. Image processing device.
The method of claim 15,
The processor,
An image characterized by combining the shift map of the previous time zone and the shift map of the current time zone, updating the shift map of the current time zone, and converting the resizing image based on the updated shift map of the current time zone. Processing unit.
The method of claim 15,
The processor,
The training resizing image corresponding to the original training image is input to the neural network model,
The neural network model, an image processing apparatus characterized in that the loss information corresponding to the result of applying the mask to the difference between the training retargeting image and the GT image is reduced.
The method of claim 15,
The image processing device,
And a communication module receiving data of the pre-trained neural network model from an external device.

Images