KR102229572B1 - Apparatus and method for image style transfer - Google Patents

Abstract

Disclosed are an image style conversion device and a method thereof. According to an embodiment of the present invention, an image style conversion model learning device comprises: a model learning unit learning an image style conversion model for converting a style of a content image based on the content image, a style image, and one or more style parameters; and a loss calculation unit calculating a conversion loss based on the content image, a style conversion image for the content image, and the style image. Therefore, distortion in the converted content image can be prevented.

Description

Apparatus and method for converting image style {APPARATUS AND METHOD FOR IMAGE STYLE TRANSFER}

The disclosed embodiments relate to a video style conversion technique.

With the development of video editing technology and effect insertion technology, individuals can use various devices such as PCs, tablets, and smartphones to edit the video they choose and apply the desired effect to create their own video. Became.

Recently, a technology that creates a new feeling image through transformation in which an individual's image is overlaid with another image style is attracting attention, but even when the intensity of the style conversion effect is adjusted to 0, the style is converted instead of the original image. There is a problem in that a biased image is output.

In addition, when applying various style effects at different intensities, the existing style conversion technology makes a part of the original image black, or when the intensity of the style conversion effect is finely adjusted, the style of the original image is refined by the degree of adjustment. It has a problem that it cannot be converted properly.

Republic of Korea Patent Publication No. 10-1882111 (registered on July 19, 2018)

The disclosed embodiments are for converting an image style into various effects and intensities.

An image style transformation model learning apparatus according to an embodiment is an image style transformation model that converts a style of the content image based on a content image, a style image, and one or more style parameters. A model learning unit for learning the content image, a style conversion image for the content image, and a loss calculator configured to calculate a conversion loss based on the style image, wherein each value of the one or more style parameters, Each time the video style conversion model is repeatedly trained a predetermined number of times, it is randomly selected from values within a preset range, and the model learning unit learns the video style conversion model based on the conversion loss.

When all values of the one or more style parameters are minimum values within the preset range, the model learning unit may train the image style transformation model so that the image style transformation model uses the content image instead of the style image.

The video style conversion model includes at least one encoder that generates a plurality of feature vectors from each of the content image and the style image, the at least one style parameter, and the content based on a plurality of feature vectors of the style image. One or more conversion units for converting at least a portion of a plurality of feature vectors of an image, and one or more generating the style-converted image based on the converted feature vector and an unconverted feature vector among a plurality of feature vectors of the content image. It may include a decoder.

Each value of the one or more style parameters may be independently selected, and the one or more conversion units may include at least some of a plurality of feature vectors of the content image based on a value of a style parameter corresponding to each of the one or more style parameters. Can be converted.

Each of the one or more converters may receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders.

The loss calculator calculates a first loss based on a content loss and a style loss, calculates a second loss based on the content image and the restored content image, and calculates a second loss based on the first loss and the second loss. The conversion loss can be calculated.

An image style conversion model execution apparatus according to an embodiment includes an input unit for acquiring a content image, a style image, and setting values for one or more style parameters, and a setting value for the content image, the style image, and the one or more style parameters. A style conversion image generator that generates a style conversion image for the content image using a pre-learned image style conversion model used as input data, and the setting values for the one or more style parameters are selected within a preset range. It's a value.

When all of the set values for the one or more style parameters are minimum values within the preset range, the style conversion image generator may control the image style conversion model to use the content image instead of the style image.

The video style conversion model includes at least one encoder that generates a plurality of feature vectors from each of the content image and the style image, the at least one style parameter, and the content based on a plurality of feature vectors of the style image. One or more conversion units for converting at least a portion of a plurality of feature vectors of an image, and one or more generating the style-converted image based on the converted feature vector and an unconverted feature vector among a plurality of feature vectors of the content image. It may include a decoder.

The setting values for the one or more style parameters may be independently selected, and the one or more conversion units may include a plurality of feature vectors of the content image based on a setting value for each corresponding style parameter among the one or more style parameters. At least some of them can be converted.

Each of the one or more converters may receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders.

According to an exemplary embodiment, a method for training a video style conversion model includes inputting input data including values of a content image, a style image, and one or more style parameters into an image style conversion model, based on the input data. Comprising a conversion loss based on the style conversion image generated by the conversion model, the content image, and the style image, and learning a learning parameter of the image style conversion model based on the conversion loss, the The step of inputting, the step of learning and the step of calculating are repeatedly performed until a preset condition is satisfied, and each value of the one or more style parameters is randomly selected from among values within a preset range each time the repetition is performed. .

The training may include training the image style transformation model so that the image style transformation model uses the content image instead of the style image when all values of each of the one or more style parameters are minimum values within the preset range. have.

The video style conversion model includes at least one encoder that generates a plurality of feature vectors from each of the content image and the style image, the at least one style parameter, and the content based on a plurality of feature vectors of the style image. Based on at least one conversion unit for converting at least a portion of a plurality of feature vectors of an image and an unconverted feature vector from among the converted feature vectors and a plurality of feature vectors of the content image, a style-converted image for the content image is It may include one or more decoders to generate.

Each value of the one or more style parameters may be independently selected, and the one or more conversion units may include at least some of a plurality of feature vectors of the content image based on a value of a style parameter corresponding to each of the one or more style parameters. Can be converted.

Each of the one or more converters may receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders.

The calculating may include calculating a first loss based on a content loss and a style loss, calculating a second loss based on the content image and the restored content image, and calculating a second loss based on the first loss and the second loss. Thus, the conversion loss can be calculated.

A method of executing an image style conversion model according to an embodiment includes the steps of obtaining a content image, a style image, and setting values for one or more style parameters, and determining a setting value for the content image, the style image, and the one or more style parameters. And generating a style converted image for the content image by using a pre-trained image style conversion model used as input data, and setting values for the one or more style parameters are selected within a preset range.

In the generating step, when all of the set values for the one or more style parameters are minimum values within the preset range, the image style conversion model may control to use the content image instead of the style image.

The video style conversion model includes at least one encoder that generates a plurality of feature vectors from each of the content image and the style image, the at least one style parameter, and the content based on a plurality of feature vectors of the style image. Based on at least one conversion unit for converting at least a portion of a plurality of feature vectors of an image and an unconverted feature vector from among the converted feature vectors and a plurality of feature vectors of the content image, a style-converted image for the content image is It may include one or more decoders to generate.

The setting values for the one or more style parameters may be independently selected, and the one or more conversion units may include a plurality of feature vectors of the content image based on a setting value for each corresponding style parameter among the one or more style parameters. At least some of them can be converted.

Each of the one or more converters may receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders.

According to the disclosed embodiments, by applying a randomly extracted style strength to train an image style transformation model, stochastic regression learning is performed without increasing computational complexity and memory usage when training an image style transformation model. You can do it.

In addition, according to the disclosed embodiments, when the style intensity applied when actually converting the style of the content image is 0, distortion in the converted content image is caused by using the content image instead of the style image in the image style conversion model. It can be prevented from occurring.

In addition, according to the disclosed embodiments, by inputting feature vectors obtained from different encoders to a plurality of transform units and independently applying style parameters to each other, style conversion effects may be applied to content images to different degrees.

1 is a block diagram illustrating an apparatus for training an image style transformation model according to an exemplary embodiment;
2 is a block diagram illustrating an image style conversion model according to an exemplary embodiment;
3 is a diagram for explaining in detail the structure of an image style conversion model and a loss calculation unit according to an embodiment;
4 is a block diagram illustrating an apparatus for executing an image style conversion model according to an exemplary embodiment;
5 is a diagram for explaining a method of selecting a style parameter according to an exemplary embodiment
6 is a diagram for describing an operation of an image style conversion model according to an exemplary embodiment;
7 is a flowchart illustrating a method of training an image style transformation model according to an exemplary embodiment;
8 is a flowchart illustrating a method of executing an image style conversion model according to an exemplary embodiment;
9 to 12 are diagrams showing a result of converting an image style according to an exemplary embodiment;
13 is a diagram showing a screen for providing a video style conversion service according to an embodiment
14 is a block diagram illustrating and describing a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, a specific embodiment will be described with reference to the drawings. The following detailed description is provided to aid in a comprehensive understanding of the methods, devices, and/or systems described herein. However, this is only an example, and the disclosed embodiments are not limited thereto.

In describing the embodiments, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the disclosed embodiments, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the disclosed embodiments and may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout the present specification. The terms used in the detailed description are only for describing the embodiments, and should not be limiting. Unless explicitly used otherwise, expressions in the singular form include the meaning of the plural form. In the present description, expressions such as "comprising" or "feature" are intended to refer to certain features, numbers, steps, actions, elements, some or combination thereof, and one or more It should not be construed to exclude the presence or possibility of other features, numbers, steps, actions, elements, any part or combination thereof.

In the following embodiments, machine learning or deep learning is an element in which the model is trained for newly input data by learning a specific model using previously stored data. This is a technical methodology for outputting the result data reflecting the data. To this end, the above-described model may include an artificial neural network structure such as a convolutional neural network (CNN) structure and a multi-layer perceptron structure.

In addition, image style transfer refers to an operation of changing a brightness, saturation, contrast value, etc. of an image by reflecting a certain part of the style of another image on one image. Through this, it is possible to create a style-transformed image that reflects the style of the artist's work desired in the image of the real world.

Hereinafter, an image to which the style is changed among images input when the image style is converted is referred to as a content image. Meanwhile, an image that provides a style reflected in a content image among images input during image style conversion is referred to as a style image. Also, a content image in which the style of the style image is reflected is referred to as a style conversion image. In this case, a variable value that determines the intensity of the style of the style image is reflected in the content image is referred to as a style parameter. Specifically, when there are a plurality of style parameters, each style parameter may be related to a different style element (eg, brightness, saturation, contrast, etc.).

In addition, learning in the image style conversion model 200 refers to a process of minimizing the loss calculated using input data, and the training parameter learned in this process minimizes the loss as much as possible. It refers to parameters that are learned in the direction, that is, weight and bias.

In addition, the content information of the content image refers to information that should not be changed during style conversion among the information of the content image, and the content information of the style image refers to information that should not be reflected in the content image when the style is changed among the information of the style image. .

In addition, the style information of the content image refers to information that must be changed during style conversion among the information of the content image, and the style information of the style image refers to information that must be reflected in the content image when the style is changed among the information of the style image.

1 is a block diagram illustrating an apparatus 100 for training an image style transformation model according to an exemplary embodiment. Referring to FIG. 1, an apparatus 100 for training an image style conversion model according to an embodiment includes a model learning unit 110 and a loss calculation unit 120.

The model learning unit 110 trains an image style conversion model 200 that converts a style of a content image based on a content image, a style image, and one or more style parameters.

In the following embodiment, the image style conversion model 200 optimizes weights and biases in the process of being trained by the model learning unit 110 to correspond to the values of the style parameters input to the input content image. The style converted image is output by reflecting the style of the style image by the intensity. In this regard, it will be described in detail below with reference to FIGS. 2 and 3.

According to an embodiment, the model learning unit 110, when all values of one or more style parameters are minimum values within a preset range, the image style conversion model so that the image style conversion model 200 uses the content image instead of the style image. (200) can be learned.

For example, for one or more style parameters having a value in the range of 0 to 1, if all style parameters are 0, the image style conversion model 200 is extracted from the content image regardless of the input style image. The information can be used in calculations for converting the style of the content image. In this regard, it will be described in detail below with reference to FIG. 6.

At this time, each value of one or more style parameters is randomly selected from among values within a preset range whenever the image style conversion model 200 is repeatedly trained for a preset number of times. In this regard, it will be described in detail below with reference to FIG. 5.

According to an embodiment, the preset number may be a batch size, and the batch size means the number of input data included in one batch.

According to an embodiment, the model learning unit 110 may use an error backpropagation algorithm in iteratively learning the image style transformation model 200. The error backpropagation algorithm is a method of learning the image style transformation model 200 by propagating the transformation loss calculated by the loss calculator 120 in a direction opposite to the direction in which the calculation using the input data is performed when converting the original image style. Refers to.

According to an embodiment, each value of one or more style parameters may be independently selected.

Specifically, when the image style conversion model 200 is repeatedly trained a preset number of times, the model learning unit 110 separately separates values of one or more style parameters among values within a preset range before performing the next training. Random sampling can be performed. At this time, the model learning unit 110 randomly selects a domain of a preset function f having a value within a preset range as a range, so that the value of the function f corresponding to the specified domain value is the value of the style parameter. It can be judged as.

The loss calculator 120 calculates a conversion loss based on a content image, a style converted image for the content image, and a style image.

Specifically, the conversion loss refers to an error of the style converted image when comparing the converted style converted image based on one or more style parameter values with a ground truth. In this case, the correct answer may mean content information that should not be changed among information in the existing content image and style information that should be reflected in a style conversion image among the existing style images.

According to an embodiment, the loss calculator 120 calculates a first loss based on a content loss and a style loss, calculates a second loss based on the content image and the restored content image, and calculates the first loss and the second loss. 2 The conversion loss can be calculated based on the loss.

According to an embodiment, the loss calculator 120 may calculate the conversion loss by Equation 1 below.

[Equation 1]

는 변환 손실,

는 제1 손실,

는 제2 손실,

는 제2 손실 가중치를 나타낸다.At this time,

Is the conversion loss,

Is the first loss,

Is the second loss,

Represents the second loss weight.

Specifically, the loss calculation unit 120 may calculate the first loss by Equation 2 below.

[Equation 2]

는 컨텐츠 손실,

는 컨텐츠 손실 가중치,

는 스타일 손실,

는 스타일 손실 가중치,

는 스타일 변환 영상,

는 컨텐츠 영상,

는 스타일 영상,

는 스타일 파라미터를 나타낸다.At this time,

Loss of content,

Is the content loss weight,

Loss of style,

Is the style loss weight,

Is the style conversion video,

Is the content video,

The style video,

Represents the style parameter.

In more detail, the loss calculator 120 may calculate the content loss by Equation 3 below, and calculate the style loss by Equation 4 and Equation 5 below.

[Equation 3]

는 L2 norm,

는 스타일 변환 영상의 컨텐츠 정보와 대응되는 특징 벡터,

는 컨텐츠 영상의 컨텐츠 정보와 대응되는 특징 벡터를 나타낸다.At this time,

Is L2 norm,

Is a feature vector corresponding to the content information of the style converted video,

Denotes a feature vector corresponding to the content information of the content image.

[Equation 4]

[Equation 5]

는 스타일 변환 영상의 스타일 정보와 대응되는 특징 벡터,

는 스타일 파라미터가

인 경우 정답 중의 스타일 정보와 대응되는 특징 벡터,

는 스타일 영상의 스타일 정보와 대응되는 특징 벡터,

는 컨텐츠 영상의 스타일 정보와 대응되는 특징 벡터를 나타낸다.At this time,

Is a feature vector corresponding to the style information of the style conversion video,

Is the style parameter

In the case of, a feature vector corresponding to the style information in the correct answer,

Is a feature vector corresponding to the style information of the style image,

Denotes a feature vector corresponding to the style information of the content image.

In addition, specifically, the loss calculation unit 120 may calculate the second loss by Equation 6 below.

[Equation 6]

은 L1 norm,

은 복원된 컨텐츠 영상을 나타낸다. 복원된 컨텐츠 영상이라 함은, 컨텐츠 영상이 영상 스타일 변환 모델(200)에 의해 부호화 된 후 스타일 영상의 정보가 반영되지 않은 채 다시 복호화 되어 출력된 영상을 의미한다.At this time,

Is L1 norm,

Represents the reconstructed content image. The reconstructed content image refers to an image that is encoded by the image style conversion model 200 and then decoded again without reflecting information on the style image, and then output.

Thereafter, the model learning unit 110 trains the image style transformation model 200 based on the transformation loss.

Meanwhile, according to an embodiment, the model learning unit 110 and the loss calculation unit 120 may be an artificial neural network structure including one or more convolutional neural networks (CNNs).

2 is a block diagram illustrating an image style conversion model 200 according to an exemplary embodiment. Referring to FIG. 2, the image style conversion model 200 according to an embodiment may include one or more encoders 210, one or more transform units 220, and one or more decoders 230.

The encoder 210 may generate a plurality of feature vectors from each of the content image and the style image.

Specifically, the feature vector may include RGB value information corresponding to each of a plurality of pixels constituting the content image and the style image, and position value information at which each pixel is located in the content image and the style image.

According to an embodiment, the encoder 210 may receive a content image and a style image, respectively, and extract a feature vector including the RGB value information of the pixel and the position value information of the pixel from each of the content image and the style image. have.

The converter 220 may convert at least some of the plurality of feature vectors of the content image based on one or more style parameters and the plurality of feature vectors of the style image.

According to an embodiment, the conversion unit 220 converts information corresponding to style information of a content image among a plurality of pieces of information included in a plurality of feature vectors of a content image, and a plurality of pieces of information included in a plurality of feature vectors of the style image. It can be changed to information corresponding to the style information of the style image.

According to an embodiment, the converter 220 may convert at least some of a plurality of feature vectors of a content image based on values of a style parameter corresponding to each of one or more style parameters. In this regard, it will be described in detail below with reference to FIG. 3.

According to an embodiment, the converter 220 may receive a feature vector of a content image and a feature vector of a style image generated by a corresponding encoder among one or more encoders, respectively. In this regard, it will be described in detail below with reference to FIG. 3.

The decoder 230 may generate a style transformed image based on the transformed feature vector and the unconverted feature vector among a plurality of feature vectors of the content image.

According to an embodiment, the decoder 230 may generate a style-converted image by decoding both a feature vector converted through style conversion and an unconverted feature vector among feature vectors of a content image.

3 is a diagram 300 for explaining in detail the structures of the image style conversion model 200 and the loss calculation unit 120 according to an exemplary embodiment. Referring to FIG. 3, a video style conversion model according to an embodiment may include at least one encoder 210, at least one conversion unit 220, at least one decoder 230, and a loss calculation unit 120.

At least one encoder 210 receives a content image and a style image. For example, each encoder (E _0, E ₁ , E ₂ , E ₃ ) having different sizes encodes the input content image, and thus feature vectors f _c.0 , f _{c. 1,} f extract _{c.2, c.3} and f, by encoding an input style image extracting each feature having a different size _s.0 vector f, f _{s.1, s.2} f, f _s.3 do.

Each of the transform units (T ₀ , T ₁ , T ₂ , T _{3 ) constituting one or more transform units 220 is a feature vector f c.0} , f _c.1 , f _c.2 , f _c extracted from the content image. _0.3 of each of the feature vectors corresponding to their _s.0 f, f _{s.1, s.2} f, f _s.3 extracted from the feature vector and style image extracted by each encoder that their size and the corresponding The feature vector extracted by the encoder is input. Thereafter, each conversion unit changes at least some of information in the corresponding feature vector extracted from the content image based on the value of each corresponding style parameter and information in the feature vector extracted from the style image.

_{Each decoder (D 1} , D ₂ , D ₃ , D ₄ ) constituting one or more decoders 230 is disposed between each of the above-described transform units (T ₀ , T ₁ , T ₂ , T _{3 ), and} A feature vector that has been transformed by a transform unit corresponding to the size and a feature vector that has a size corresponding to the one extracted from the content image are input. Thereafter, each decoder decodes the received feature vectors to generate a style-converted image and a reconstructed content image.

) 및 제2 손실(

)에 기초하여 변환 손실(

)을 산출하고, 산출된 변환 손실을 각 디코더(D₁, D₂, D₃, D₄)에 되먹임(feedback)하여, 영상 스타일 변환 모델(200)이 변환 손실을 최소화하는 방향으로 학습될 수 있도록 한다.The loss calculation unit 120 is the first loss (

) And the second loss (

) Based on the conversion loss (

), and the calculated conversion loss _{is fed back to each decoder (D 1} , D ₂ , D ₃ , D ₄ ), so that the video style conversion model 200 can be trained in a direction that minimizes the conversion loss. To be there.

4 is a block diagram illustrating an apparatus 400 for executing an image style conversion model according to an exemplary embodiment. Referring to FIG. 4, an apparatus 400 for executing an image style conversion model according to an embodiment includes an input unit 410 and a style conversion image generation unit 420.

The input unit 410 acquires a content image, a style image, and setting values for one or more style parameters.

In this case, the setting value for one or more style parameters is a value selected within a preset range. For example, a user of the image style conversion model execution apparatus 400 may input setting values for one or more style parameters within a preset range through the input unit 410.

According to an embodiment, setting values for one or more style parameters may be independently selected. For example, when a first style parameter and a second style parameter exist, the user of the video style conversion model execution device 400 may use the input unit 410 to set the first style parameter and the second style parameter within a preset range. By separately entering the two parameters can be selected independently.

The style conversion image generation unit 420 generates a style conversion image for a content image using a pre-learned image style conversion model 200 that uses a content image, a style image, and setting values for one or more style parameters as input data. do.

In this case, the video style conversion model 200 may include one or more encoders 210, one or more conversion units 220, and one or more decoders 230, as described with reference to FIG. 2, and the encoder 210 , Since each of the transform unit 220 and the decoder 230 performs the same or similar function as in the embodiment described above with reference to FIG. 2, a more detailed description thereof will be omitted.

According to an embodiment, the style conversion image generation unit 420 controls the image style conversion model 200 to use a content image instead of a style image when all set values for one or more style parameters are minimum values within a preset range. can do. This embodiment is the same as or similar to that described in the learning process for the image style conversion model 200 of the model learning unit 110 described above with reference to FIG. 1, and thus a detailed description thereof will be omitted.

5 is a diagram 500 for describing a method of selecting a style parameter according to an exemplary embodiment.

Referring to FIG. 5, a style parameter (control parameter) is randomly selected within a preset range (eg, 0 or more and 1 or less). When a style parameter is selected, a style strength corresponding thereto is determined, and the image style conversion model 200 converts the style of the input content image based on the input style image according to the determined style strength.

For example, when the style parameter is 1, the style intensity has a maximum value within the controllable range of the image style conversion model 200, and when the style parameter is 0, the style intensity is the image It has a minimum value within the range of the style intensity that can be set in the style conversion model 200.

6 is a diagram 600 for explaining an operation of an image style conversion model according to an exemplary embodiment.

6, if at least one of the values of one or more style parameters is not a minimum value within a preset range, the video style conversion model 200 learned by the model learning unit 110 is the video style conversion model ( 200), the style of the content image is converted based on the content image and the style image 610.

However, if the values of one or more style parameters are all minimum values within a preset range, the image style conversion model 200 redundantly uses the content image 620 instead of the input style image.

In this case, since the information included in the content image is replaced with the same information included in the same content image, as a result, the information of the existing content image is not changed, so that the image style conversion model 200 has all values of one or more style parameters. When it has a minimum value within a preset range, a reconstructed content image is generated as a style conversion image.

7 is a flowchart illustrating a method of training an image style conversion model according to an exemplary embodiment. The method illustrated in FIG. 7 may be performed, for example, by the apparatus 100 for learning the image style conversion model described above.

Referring to FIG. 7, the apparatus 100 for learning an image style conversion model first inputs input data including values of a content image, a style image, and one or more style parameters into the image style conversion model 200 (710). .

Thereafter, the video style conversion model learning apparatus 100 calculates a conversion loss based on the style conversion image, the content image, and the style image generated by the video style conversion model 200 based on the input data (720).

According to an embodiment, the image style conversion model training apparatus 100 calculates a first loss based on a content loss and a style loss, calculates a second loss based on the content image and the restored content image, and The conversion loss can be calculated based on the first loss and the second loss.

Thereafter, the video style conversion model training apparatus 100 learns a learning parameter of the video style conversion model 200 based on the conversion loss (730 ).

According to an embodiment, the video style conversion model training apparatus 100 may use the video style conversion model 200 to set the content video instead of the style video when the values of each of the one or more style parameters in the input data are all minimum values within a preset range. The image style conversion model 200 may be trained to use. This embodiment is the same as or similar to that described in the learning process for the image style conversion model 200 of the model learning unit 110 described above with reference to FIG. 1, and thus a detailed description thereof will be omitted.

Thereafter, the image style conversion model training apparatus 100 determines whether a preset learning end condition is satisfied (740), and if not, repeats steps 710 to 730 until the condition is satisfied. .

At this time, each value of one or more style parameters is randomly selected from among values within a preset range for each repetition.

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, performed together, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

8 is a flowchart illustrating a method of executing an image style conversion model according to an exemplary embodiment. The method illustrated in FIG. 8 may be performed, for example, by the image style conversion model execution apparatus 400 described above.

Referring to FIG. 8, the apparatus 400 for executing an image style conversion model first obtains a content image, a style image, and setting values for one or more style parameters (810).

At this time, the setting values for one or more style parameters are values selected within a preset range.

Thereafter, the image style conversion model execution apparatus 400 uses a pre-learned image style conversion model 200 that uses a content image, a style image, and a setting value for one or more style parameters as input data to determine the style for the content image. A transformed image is generated (820).

According to an embodiment, the image style conversion model execution apparatus 400 uses a content image instead of a style image when the setting values for one or more style parameters are all minimum values within a preset range. Can be controlled to do. This embodiment is the same as or similar to that described in the learning process for the image style conversion model 200 of the model learning unit 110 described above with reference to FIG. 1, and thus a detailed description thereof will be omitted.

In the illustrated flowchart, the method is described by dividing the method into a plurality of steps, but at least some of the steps are performed in a different order, combined with other steps, performed together, omitted, divided into detailed steps, or not shown. One or more steps may be added and performed.

9 to 12 are diagrams illustrating a result of converting an image style according to an exemplary embodiment. The style conversion images shown in FIGS. 9 to 12 may be generated, for example, by the image style conversion model 200 described above.

)이 0과 1 사이에서 변화됨에 따라 상이한 스타일 변환 영상을 생성한다.9 to 12, when a content image and a style image are input to the image style conversion model 200 to generate a style conversion image for the content image, the image style conversion model 200 includes each conversion unit (T Setting values of one or more style parameters corresponding to ₀ , T ₁ , T ₂ , T _{3, one-to-one (}

) Is changed between 0 and 1, creating a different style transformed image.

)을 나타낸 것이다. 이때, 하나의 스타일 파라미터의 설정 값이 상단에 표시된 특정 값을 갖는 경우, 나머지 스타일 파라미터의 설정 값은 모두 0.0이라고 가정한다.Specifically, the values from 0.0 to 1.0 displayed at the top of FIGS. 9 to 12 are set values of one or more style parameters corresponding to _{each conversion unit (T 0} , T ₁ , T ₂ , T _{3) one-to-one (}

). In this case, when the setting value of one style parameter has a specific value displayed at the top, it is assumed that all of the setting values of the other style parameters are 0.0.

는 스타일 변환 영상의 색상 변환과 관련된 값으로써, 영상 스타일 변환 모델(200)은

의 값이 1.0에 가까워질수록 스타일 영상의 색상과 유사한 색상의 스타일 변환 영상을 생성한다.In FIGS. 9 to 12, for example, a setting value of a style parameter corresponding to the _{conversion unit T 0}

Is a value related to color conversion of the style converted image, and the image style conversion model 200 is

As the value of is closer to 1.0, a style conversion image with a color similar to the color of the style image is generated.

는 스타일 변환 영상의 명암 변환과 관련된 값으로써, 영상 스타일 변환 모델(200)은

의 값이 1.0에 가까워질수록 스타일 영상의 명암과 유사한 명암의 스타일 변환 영상을 생성한다.Also, the setting value of the style parameter corresponding to the conversion unit T ₃

Is a value related to the contrast conversion of the style converted image, and the image style conversion model 200 is

As the value of is closer to 1.0, a style conversion image with a contrast similar to that of the style image is generated.

13 is a diagram illustrating a screen for providing a video style conversion service according to an exemplary embodiment.

Referring to FIG. 13, a content image for performing style conversion is input into an image input box located on the left side of the screen, and a style image is input into an image input box located on the right side of the screen.

In addition, the setting values of one or more style parameters are independently adjusted according to the movement of the control unit on the bar displayed at the top of the center of the screen between 0 and 1.

In addition, when a content image and a style image are respectively input into the image input box and the adjustment unit on the bar displayed at the top of the screen is adjusted, as a result, style conversion is performed and the style conversion image is displayed at the bottom of the center of the screen.

14 is a block diagram illustrating and describing a computing environment 10 including a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12. In an embodiment, the computing device 12 may be the image style conversion model training device 100. In addition, the computing device 12 may be an image style conversion model execution device 400.

The computing device 12 includes at least one processor 14, a computer-readable storage medium 16 and a communication bus 18. The processor 14 may cause the computing device 12 to operate in accordance with the aforementioned exemplary embodiments. For example, the processor 14 may execute one or more programs stored in the computer-readable storage medium 16. The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions are configured to cause the computing device 12 to perform operations according to an exemplary embodiment when executed by the processor 14 Can be.

The computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer-readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, the computer-readable storage medium 16 includes memory (volatile memory such as random access memory, nonvolatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, other types of storage media that can be accessed by computing device 12 and store desired information, or a suitable combination thereof.

The communication bus 18 interconnects the various other components of the computing device 12, including the processor 14 and computer readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide interfaces for one or more input/output devices 24. The input/output interface 22 and the network communication interface 26 are connected to the communication bus 18. The input/output device 24 may be connected to other components of the computing device 12 through the input/output interface 22. The exemplary input/output device 24 includes a pointing device (mouse or track pad, etc.), a keyboard, a touch input device (touch pad or touch screen, etc.), a voice or sound input device, various types of sensor devices, and/or photographing devices. Input devices, and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included in the computing device 12 as a component constituting the computing device 12, and may be connected to the computing device 102 as a separate device distinct from the computing device 12. May be.

Meanwhile, an embodiment of the present invention may include a program for performing the methods described in the present specification on a computer, and a computer-readable recording medium including the program. The computer-readable recording medium may include a program command, a local data file, a local data structure, or the like alone or in combination. The media may be specially designed and configured for the present invention, or may be commonly used in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Includes hardware devices. Examples of the program may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the exemplary embodiments of the present invention have been described in detail above, those of ordinary skill in the art to which the present invention pertains will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. . Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should not be determined by the claims to be described later, but also by the claims and equivalents.

10: computing environment
12: computing device
14: processor
16: computer readable storage medium
18: communication bus
20: program
22: input/output interface
24: input/output device
26: network communication interface
100: image style conversion model learning device
110: Model Learning Department
120: loss calculation unit
200: image style conversion model
210: encoder
220: conversion unit
230: decoder
400: image style conversion model execution device
410: input
420: style conversion image generation unit

Claims (22)

A model learning unit that trains an image style conversion model for converting a style of the content image based on a content image, a style image, and one or more style parameters; And
A loss calculator configured to calculate a conversion loss based on the content image, a style conversion image for the content image, and the style image,
Each value of the one or more style parameters is randomly selected from values within a preset range each time the image style conversion model is repeatedly trained a preset number of times,
The model learning unit trains the image style transformation model based on the transformation loss,
The video style conversion model,
At least one encoder for generating a plurality of feature vectors from each of the content image and the style image;
At least one conversion unit for converting at least a portion of a plurality of feature vectors of the content image based on the one or more style parameters and a plurality of feature vectors of the style image; And
And one or more decoders configured to generate the style transformed image based on the transformed feature vector and a feature vector that has not been transformed from among a plurality of feature vectors of the content image.
The method according to claim 1,
The model learning unit, when all values of the one or more style parameters are minimum values within the preset range, trains the image style conversion model so that the image style conversion model uses the content image instead of the style image. Transformation model learning device.
delete The method according to claim 1,
The values of each of the one or more style parameters are independently selected,
The at least one conversion unit converts at least a part of a plurality of feature vectors of the content image based on values of a style parameter corresponding to each of the at least one style parameter.
The method according to claim 1,
The one or more transforming units, respectively, receiving a feature vector of the content image generated by a corresponding encoder among the one or more encoders and a feature vector of the style image, the image style transformation model training apparatus.
The method according to claim 1,
The loss calculator calculates a first loss based on a content loss and a style loss, calculates a second loss based on the content image and the restored content image, and calculates a second loss based on the first loss and the second loss. A video style conversion model training device that calculates the conversion loss.
An input unit for obtaining a content image, a style image, and setting values for one or more style parameters; And
A style conversion image generator for generating a style conversion image for the content image by using a pre-learned image style conversion model that uses the content image, the style image, and setting values for the one or more style parameters as input data, ,
The setting value for the one or more style parameters is a value selected within a preset range,
The video style conversion model,
At least one encoder for generating a plurality of feature vectors from each of the content image and the style image;
At least one conversion unit for converting at least a portion of a plurality of feature vectors of the content image based on the one or more style parameters and a plurality of feature vectors of the style image; And
And one or more decoders configured to generate the style transformed image based on the transformed feature vector and a feature vector that has not been transformed from among a plurality of feature vectors of the content image.
The method of claim 7,
The style conversion image generation unit, when all of the set values for the one or more style parameters are minimum values within the preset range, controls the image style conversion model to use the content image instead of the style image. Running device.
delete The method of claim 7,
Setting values for the one or more style parameters are independently selected,
The at least one conversion unit converts at least a part of a plurality of feature vectors of the content image based on a setting value for each corresponding style parameter among the at least one style parameter.
The method of claim 7,
The one or more transforming units, respectively, receiving a feature vector of the content image generated by a corresponding encoder among the one or more encoders and a feature vector of the style image, a video style conversion model execution apparatus.
Inputting input data including values of each of a content image, a style image, and one or more style parameters into an image style conversion model;
Calculating a conversion loss based on the style converted image generated by the image style conversion model based on the input data, the content image, and the style image; And
And learning a learning parameter of the image style transformation model based on the transformation loss,
The inputting, learning, and calculating steps are repeatedly performed until a preset condition is satisfied,
Each value of the one or more style parameters is randomly selected from among values within a preset range each time the repetition is performed,
The video style conversion model,
At least one encoder for generating a plurality of feature vectors from each of the content image and the style image;
At least one conversion unit for converting at least a portion of a plurality of feature vectors of the content image based on the one or more style parameters and a plurality of feature vectors of the style image; And
And at least one decoder for generating a style transformed image for the content image based on the transformed feature vector and an unconverted feature vector among a plurality of feature vectors of the content image.
The method of claim 12,
The training may include training the image style transformation model so that the image style transformation model uses the content image instead of the style image when all values of each of the one or more style parameters are minimum values within the preset range, How to train the image style transformation model.
delete The method of claim 12,
The values of each of the one or more style parameters are independently selected,
The at least one conversion unit converts at least a part of a plurality of feature vectors of the content image based on values of a style parameter corresponding to each of the at least one style parameter.
The method of claim 12,
The one or more transforming units each receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders, respectively.
The method of claim 12,
The calculating may include calculating a first loss based on a content loss and a style loss, calculating a second loss based on the content image and the restored content image, and calculating a second loss based on the first loss and the second loss. The method for training a video style conversion model to calculate the conversion loss.
Obtaining a content image, a style image, and setting values for one or more style parameters; And
Generating a style conversion image for the content image using a pre-learned image style conversion model using the content image, the style image, and a setting value for the one or more style parameters as input data,
The setting value for the one or more style parameters is a value selected within a preset range,
The video style conversion model,
At least one encoder for generating a plurality of feature vectors from each of the content image and the style image;
At least one conversion unit for converting at least a portion of a plurality of feature vectors of the content image based on the one or more style parameters and a plurality of feature vectors of the style image; And
And at least one decoder for generating a style transformed image for the content image based on the transformed feature vector and an unconverted feature vector among a plurality of feature vectors of the content image.
The method of claim 18,
In the generating step, when all the setting values of the one or more style parameters are minimum values within the preset range, the video style conversion model is controlled to use the content video instead of the style video, executing a video style conversion model. Way.
delete The method of claim 18,
Setting values for the one or more style parameters are independently selected,
The at least one conversion unit converts at least a part of a plurality of feature vectors of the content image based on a setting value for each corresponding style parameter among the at least one style parameter.
The method of claim 18,
The one or more transforming units respectively receive a feature vector of the content image and a feature vector of the style image generated by a corresponding encoder among the one or more encoders.

Images