TWI723547B - Style transfer method and computer program product thereof - Google Patents

Abstract

A style transfer method incudes: obtaining a first convolution neural network (CNN) model, wherein the first convolution neural network model is trained to extract features from a content image, and includes a first specific number of feature filters and a second specific number of feature filters; reducing the number of the first specific number of feature filters to one-eighth; reducing the number of the second specific number of feature filters to one-eighth; inputting the content image to the first convolution neural network model; generating a plurality of first feature images from the content image through performing a convolution calculation by the first convolution neural network model.

Description

Style transfer method and computer program product

The present invention relates to a method of style transfer, in particular to a method of quick style transfer applied to a web interface and a computer program product thereof.

Style transfer (style transfer) uses artificial intelligence (AI) visual algorithms to extract the content of an original image (content image) and extract the style of a style image (style image), and combine the above two It is a technology synthesized by the author.

Web browsing is currently the most widely used medium in society. In order to make the style transfer application more visible to more people, how to make a huge style transfer algorithm can quickly produce the final synthesis on the web page The result is the first priority. Now that various hardware AI accelerations have not yet considered web applications, the above tasks are even more difficult.

The method of style transfer according to an embodiment of the present invention, applied to a web interface, includes: obtaining a first convolution neural network (CNN) model, wherein the first convolution neural network The model is trained to extract features from a content image, and includes a plurality of convolutional layers including a first specific number of feature filters and a second specific number of features Filter; reduce the number of the first specific number of feature filters to one eighth of the original; reduce the number of the second specific number of feature filters to one eighth of the original; enter the original image To the first convolutional neural network model; perform a convolution operation through the first convolutional neural network model to generate a complex first feature map of the original image. Among them, each feature filter is used to extract different features of the original image.

The method for style transfer as described above further includes: obtaining a second convolutional neural network model; inputting the plural first feature maps and a style map of the original image to the second convolutional neural network model ; Through the convolution operation, extract the spatial feature map associated with the style map from the plural first feature maps, and extract the non-spatial feature map associated with the style map from the style map; through A loss function operation to merge the spatial feature map associated with the original image and the non-spatial feature map associated with the style map to obtain a composite image after style transfer; and display the composite image on the In the web interface.

Such as the above-mentioned style transfer method, wherein the first convolutional neural network model performs convolution operation on the original image with the first specific number and the second specific number of feature filters to extract and generate The plural first feature maps of the original image.

The method for style transfer as described above, wherein the spatial feature map includes a shape feature map and a boundary feature map; the non-spatial feature map includes a color feature map and a texture feature map.

The method for style transfer as described above, wherein the first specific number is 32, and the second specific number is 64.

The method for style transfer as described above, wherein the second convolutional neural network model is VGG19.

The method for style transfer as described above, wherein the loss function calculation includes: calculating a first feature gap between the spatial feature map associated with the original image and the composite image; calculating the non-spatial feature map associated with the style map A second feature gap between the feature map and the composite image; the first feature gap and the second feature gap are added together to obtain the loss function; a gradient descent method is used to minimize the loss function to Obtain the composite picture.

A computer program product according to an embodiment of the present invention is applied to a web interface for style transfer between an original image and a style image; the computer program product loads the program through the computer to execute: a first call Command, a filter setting command, a first input command, and a first feature extraction command. The first call instruction causes a processor of the computer to obtain a first convolutional neural network model from the storage of the computer; wherein, the first convolutional neural network model is trained to extract the original image Features and includes a plurality of convolutional layers, and the plurality of convolutional layers includes a first specific number of feature filters and a second specific number of feature filters. The filter setting instruction causes the processor to reduce the number of the first specific number of feature filters to one eighth of the original, and reduce the number of the second specific number of feature filters to one eighth of the original One. The first input instruction causes the processor to input the original image to the first convolutional neural network model. The first feature extraction instruction causes the processor to perform a convolution operation through the first convolutional neural network module to generate a complex first feature map of the original image. Among them, each feature filter is used to extract different features of the original image.

The computer program product described above further includes: a second call command, a second input command, a second feature extraction command, a synthesis command, and a display command. The second call instruction causes the processor to obtain a second convolutional neural network model from the computer's storage. The second input instruction enables the processor to input the plural first feature maps and the style map of the original image to the second convolutional neural network model. The second feature extraction instruction causes the processor to extract the spatial feature maps associated with the original image from the plurality of first feature maps through the convolution operation, and the extraction from the style map is associated with the style The non-spatial feature map of the graph. The synthesis instruction causes the processor to combine the spatial feature map associated with the original image with the non-spatial feature map associated with the style map through a loss function operation to obtain a composite image after style transfer . The display instruction causes the processor to display the composite image in the web page interface.

Such as the above-mentioned computer program product, wherein the first convolutional neural network model performs convolution operation on the original image with the first specific number and the second specific number of feature filters to extract and generate the original The plural first feature maps of the graph.

The computer program product described above, wherein the spatial feature map includes a shape feature map and a boundary feature map; the non-spatial feature map includes a color feature map and a texture feature map.

The computer program product described above, wherein the first specific number is 32, and the second specific number is 64.

Such as the computer program product mentioned above, wherein the second convolutional neural network model is VGG 19.

The computer program product described above, wherein the loss function calculation includes: calculating a first feature gap between the spatial feature map associated with the original image and the composite image; calculating the non-spatial feature map associated with the style map And a second feature gap of the composite image; sum the first feature gap and the second feature gap to obtain the loss function; use a gradient descent method to minimize the loss function to obtain the Composite image.

The style transfer method of the present invention is applied to a web interface. The web interface uses, for example, a web graphics database Web Graphics Library (WebGL) to display interactive web pages in any compatible web browser. 3D or 2D graphics. In this embodiment, the user can upload a content image and a style image to the web interface, or the user can upload the original image and choose the style provided by the web page. It also uses the style transfer method of the present invention to extract content from the original image through convolution, and extract the style from the style image, and synthesize, and finally output a synthesized image in the web interface. And to achieve the effect of style transfer.

Figure 1 is a schematic diagram of the convolution operation of the disclosed embodiment. As shown in Figure 1, the input image 100 (for example, the original image and represented by a 7*7 matrix) is convolved with the feature filter 102 (for example, represented by a 3*3 matrix), and A feature map 104 is obtained (it becomes a 5*5 matrix after convolution operation). If the feature in the input image 100 is more similar to the feature of the feature filter 102, the corresponding convolution value in the feature map 104 obtained by the convolution operation will be larger.

For example, as shown in Figure 1, some of the features 110 in the input image 100 (represented by a 3*3 matrix, for example) are convolved with the feature filter 102, because some of the features 110 are related to the feature filter 102. It is exactly the same. After the convolution operation, the corresponding convolution value "4" can be obtained in the feature map 104. Furthermore, some of the features 112 in the input image 100 are completely different from the feature filter 102. After a convolution operation, the corresponding convolution value "0" can be obtained in the feature map 104. Through the aforementioned convolution operation, the feature filter 102 can extract the features in the input image 100 that are closest to the feature filter 102 itself, and obtain the corresponding feature map 104. The values in the matrix of the input image 100, the feature filter 102, and the feature map 104 in Fig. 1 are only examples, and not as a limitation of the present invention.

The style transfer method of the present invention uses a trained first convolution neural network (CNN) model to extract the features in the original image. Generally speaking, the first convolutional neural network model is an algorithm. This algorithm, for example, taking Python syntax as an example, includes multiple function functions of convolution operations, such as conv1 = _conv_layer (image, 32, 9, 1), where _conv_layer is the function function of the above convolution operations, which can be based on Its application requirements include setting different input images (image), the number of feature filters (filter number, for example, 32), the feature filter size (filter size, for example, 9), and the convolution operation stride (stride, for example, 1).

FIG. 2 is a schematic diagram of a first convolutional neural network model 200 according to an embodiment of the disclosure. As shown in Figure 2, the trained first convolutional neural network 200 includes a complex convolution layer (eg convolution layer 210, 212, 214, 216), and a complex residual block (residual block) (For example, residual blocks 218, 220, 222, 224, 226, 228). Among them, each convolutional layer and each residual block of the first convolutional neural network 200 can respectively correspond to different convolution operation function functions in its algorithm.

For example, the convolution layer 210 may correspond to the convolution function conv1 = _conv_layer (image, 32, 9, 1) in the algorithm, and the residual block 218 may correspond to the convolution function conv3 = _conv_layer ( conv2, 128, 3, 2). Therefore, by setting the above-mentioned convolution function function (for example, pre-setting the number or types of feature filters), the trained first convolutional neural network model 200 can have a plurality of preset feature filters for Extract the different features of the input original image.

For example, when the original image is input to the convolutional layer 210, the original image and 32 different feature filters in the convolutional layer 210 (such as the feature filter 102 in the first image) will perform convolution operations respectively. It is used to capture 32 different features (such as shape, border, color, texture, etc.) of the original image, and generate 32 different feature images (such as the first image feature map 104). Similarly, the 32 different feature images generated by the convolutional layer 210 will be sequentially input to the convolutional layer 212, and each feature image will be captured again by 64 different feature filters in the convolutional layer 212. 64 different feature images of one feature image.

To put it simply, the original image is input to the convolutional layer 210 of the first convolutional neural network model 200, and passes through the convolutional layer 212, residual blocks 218~228, convolutional layer 214, and convolutional layer 216 in sequence on the way. Product operation, and output the complex first feature maps of the original image from the convolution layer 216. In this embodiment, each feature filter in the complex convolutional layers (convolutional layers 210~216) performs a convolution operation with the input original image, and the complex residual blocks (residual blocks) Each feature filter in 218~228) performs two convolution operations with the input original image.

In this embodiment, the first convolutional neural network model 200 can be, for example, the following algorithm (using Python syntax programming as an example): def net(image): conv1 = _conv_layer (image, 32, 9, 1) conv2 = _ conv_layer (conv1, 64, 3, 2) conv3 = _conv_layer (conv2, 128, 3, 2) resid1 = residual_block (conv3, 3) resid2 = residual_block (resid1, 3) resid3 = residual_block (resid2, 3) resid4 = residual_block (resid3, 3) resid5 = residual_block (resid4, 3) conv_t1 = conv_transpose_layer (resid5, 64, 3, 2) conv_t2 = conv_transpose_layer (conv_t1, 32, 3, 2) …

The conv1 = _conv_layer (image, 32, 9, 1) in the algorithm is to input the original image into the convolutional layer 210, and the convolutional layer 210 has 32 feature filters.

The conv2 = _ conv_layer (conv1, 64, 3, 2) in the algorithm is to input the output feature map calculated by the convolution layer 210 into the convolution layer 212, and the convolution layer 212 has 64 feature filters.

The conv3 = _conv_layer (conv2, 128, 3, 2) in the algorithm is to input the output feature map calculated by the convolutional layer 212 into the residual block 218, and the residual block 218 has 128 feature filters .

The residual1 = residual_block (conv3, 3) in the algorithm is to input the output feature map calculated by the residual block 218 into the residual block 220, and the residual block 220 also has 128 feature filters .

Resid2 = residual_block (resid1, 3) in the algorithm is to input the feature map calculated by the residual block 220 into the residual block 222, and the residual block 222 also has 128 feature filters.

Resid3 = residual_block (resid2, 3) in the algorithm is to input the feature map calculated by the residual block 222 into the residual block 224, and the residual block 224 also has 128 feature filters.

Resid4 = residual_block (resid3, 3) in the algorithm is to input the feature map calculated by the residual block 224 into the residual block 226, and the residual block 226 also has 128 feature filters.

The residual5 = residual_block (resid4, 3) in the algorithm is to input the feature map calculated by the residual block 226 into the residual block 228, and the residual block 228 also has 128 feature filters.

The conv_t1 = conv_transpose_layer (resid5, 64, 3, 2) in the algorithm is to input the feature map calculated by the residual block 228 to the convolutional layer 214, and the convolutional layer 214 has 64 feature filters.

The conv_t2 = conv_transpose_layer (conv_t1, 32, 3, 2) in the algorithm is to input the feature map calculated by the convolution layer 214 to the convolution layer 216, and the convolution layer 216 has 32 feature filters.

Figures 3A and 3B are flowcharts of the style transfer method according to an embodiment of the disclosure. As shown in FIG. 3A, in step S300, the style transfer method of the present invention first obtains a first convolutional neural network (for example, the first convolutional neural network 200 in FIG. 2). Next, the style transfer method of the present invention executes step S302 to reduce the number of a first specific number of feature filters to one-eighth of the original number. And in step S304, the number of a second specific number of feature filters is reduced to one-eighth of the original number.

FIG. 4 is a schematic diagram of a first convolutional neural network model 400 according to an embodiment of the disclosure. In this embodiment, the style transfer method of the present invention reduces the 32 feature filters in the convolutional layers 210 and 216 in the first convolutional neural network model 200 to 4 feature filters, and generates the first Convolutional layers 410 and 416 in the convolutional neural network model 400. The style transfer method of the present invention reduces the 64 feature filters in the convolutional layers 212 and 214 in the first convolutional neural network model 200 to 8 feature filters, and accordingly generates the first convolutional neural network model Convolutional layers 412, 414 in 400.

Furthermore, the style transfer method of the present invention reduces the 128 feature filters in the residual blocks 218, 220, 222, 224, 226, and 228 in the first convolutional neural network model 200 to 16. The feature filter generates the residual blocks 418, 420, 422, 424, 426, 428 in the first convolutional neural network model 400 correspondingly. Finally, the style transfer method of the present invention reduces the file size of the first convolutional neural network model 200 (tensorflow file) from the original 6580KB to the file size of the first convolutional neural network model 400 of 148KB.

In step S306, the style transfer method of the present invention inputs the original image to the first convolutional neural network model 400. Next, in step S308, the style transfer method of the present invention performs a convolution operation through the first convolutional neural network model 400 to generate a complex first feature map of the original image. For example, the 32 feature filters in the convolutional layer 410 in the first convolutional neural network model 400 are used to extract 32 different features of the original image, and the 64 feature filters in the convolutional layer 412, It is used to extract 64 different features of the original image.

After obtaining the plural first feature maps of the original image, the style transfer method of the present invention executes step S310 to obtain a second convolutional neural network model. In this embodiment, the second convolutional neural network model is also an algorithm, for example, Visual Geometry Group 19 (VGG19), which also has multiple convolutional layers. Through the principle of image recognition technology, the input is input to the second Convolutional neural network model images or pictures are classified.

In step S312, the style transfer method of the present invention inputs the plural first feature maps of the original image and the style map to the second convolutional neural network model, and in step S314, the second convolution Through the convolution operation, the neural network model extracts spatial feature maps (such as shape feature maps, boundary feature maps) related to the original image from the plural first feature maps, and extracts related features from the style map Non-spatial feature maps (such as color feature maps, texture feature maps) linked to the style map. The principle of the second convolutional neural network model for performing convolution operations is the same as that of the first convolutional neural network model 200 in Figure 2 and the first convolutional neural network model 400 in Figure 4, so it will not be repeated here. .

Next, in step S316, the style transfer method of the present invention uses a loss function operation to merge the spatial feature map associated with the original image and the non-spatial feature map associated with the style map to obtain the classic A composite picture after the style transfer. Finally, in step S318, the style transfer method of the present invention displays the composite image in the web page interface.

In step S316, the loss function calculation performed by the style transfer method of the present invention includes: calculating a first feature gap between the spatial feature map associated with the original image and the composite image; calculating the difference between the spatial feature map associated with the original image and the composite image; A second feature gap between the non-spatial feature map of the graph and the composite image; the first feature gap and the second feature gap are summed to obtain the loss function; the loss function is minimized by a gradient descent method化 to obtain the composite image. The gradient descent method is a first-order optimization algorithm, and the local minimum of the loss function is found through the gradient descent method to minimize the loss function.

In this embodiment, when the input resolution of the original image or the style image is 256*256, when the first convolutional neural network model 200 of the second image is used in conjunction with the second convolutional neural network When the road model performs style transfer, that is, step S300, step S306, step S308, step S310, step S312, step S314, step S316, and step S318 in Figures 3A and 3B are executed, but step S302 and step S304 are not executed, then The processing time for performing style transfer on the web interface is 1710 microseconds per image. In other words, after the user uploads the original image and the style map to the web interface, until the style transfer method of the present invention displays the final composite image on the web interface, the cost of the style transfer process is The time is 1710 microseconds.

In another embodiment, when the input resolution of the original image or the style image is 256*256, when the first convolutional neural network model 400 of the fourth image is used in conjunction with the second convolutional neural network When the network model executes style transfer, it will execute all the steps in Figures 3A and 3B (including step S302 and step S304), and the processing time for performing style transfer on the web interface is only time-consuming for each image 130 microseconds. In other words, the style transfer method of the present invention reduces the number of feature filters in the first convolutional neural network model 200 to become the first convolutional neural network model 400 to reduce the style transfer. The total execution time.

In another embodiment, when the input resolution of the original image or the style image is 480*480, when the first convolutional neural network model 400 of the fourth image is used in conjunction with the second convolutional neural network When the network model executes style transfer, it will execute all the steps in Figures 3A and 3B (including step S302 and step S304), and the processing time for performing style transfer on the web interface is only time-consuming for each image 270 microseconds, which is far more than using the first convolutional neural network model 200 with the second convolutional neural network model, and the resolution of the input image is 256*256. Each image takes 1710 microseconds. Even faster.

Although the number of feature filters is reduced, the number of feature points extracted from the original image or the style map is also reduced, so that the effect of the composite image after the style transfer is deteriorated. The style transfer method of the present invention sacrifices a little synthesis effect, so that the user can effectively reduce the processing time of style transfer under the premise of accepting the visual effect of the synthesized image, so as to improve the execution of style transfer on the web interface User experience.

The present invention further discloses a computer program product applied to a web interface for style transfer between an original image and a style image; the computer program product is loaded into the program by the computer to execute: a first call command, A filter setting instruction, a first input instruction, a first feature extraction instruction, a second call instruction, a second input instruction, a second feature extraction instruction, a synthesis instruction, and a display instruction. The first call instruction causes a processor of the computer to execute step S300 in FIG. 3A. The filter setting instruction causes the processor to execute step S302 and step S304 in FIG. 3A. The first input instruction causes the processor to execute step S306 in FIG. 3A. The first feature extraction instruction causes the processor to execute step S308 in FIG. 3A.

According to the computer program product disclosed in the present invention, the second call instruction causes the processor to execute step S310 in FIG. 3B. The second input instruction causes the processor to execute step S312 in FIG. 3B. The second feature extraction instruction causes the processor to execute step S314 in FIG. 3B. The synthesis instruction causes the processor to execute step S316 in FIG. 3B. Finally, the display instruction causes the processor to execute step S318 in FIG. 3B.

When the computer program product disclosed in the present invention executes the synthetic instruction, the processor executes a loss function operation. The loss function operation includes: calculating a spatial feature map associated with the original image and a first of the composite image. Feature gap; calculate a second feature gap between the non-spatial feature map and the composite image associated with the style map; add the first feature gap and the second feature gap to obtain the loss function; The gradient descent method minimizes the loss function to obtain the composite image.

The style transfer method and computer program product disclosed in the present invention enable the style transfer function to be implemented on low-level hardware devices through a web interface, so that users can also use low-level hardware devices. Experience the charm of style transfer smoothly.

Although the embodiments of the present invention are as described above, we should understand that what is presented above is only an example, not a limitation. According to this embodiment, many changes of the above exemplary embodiment can be implemented without violating the spirit and scope of the invention. Therefore, the breadth and scope of the present invention should not be limited by the embodiments described above. More precisely, the scope of the present invention should be defined by the following patented scope and its equivalents.

100 ~ input image 102 ~ Feature Filter 104 ~ Feature Map 110, 112 ~ some features 200, 400 ~ the first convolutional neural network model 210, 212, 214, 216 ~ Convolutional layer 410, 412, 414, 416 ~ Convolutional layer 218, 220, 222 ~ residual block 224, 226, 228 ~ residual block 418, 420, 422 ~ residual block 424, 426, 428 ~ residual block S300, S302, S304, S306, S308 ~ step S310, S312, S314, S316, S318 ~ step

Figure 1 is a schematic diagram of the convolution operation of the disclosed embodiment. FIG. 2 is a schematic diagram of a first convolutional neural network model 200 according to an embodiment of the disclosure. FIG. 3A is a flowchart of a style transfer method according to an embodiment of the disclosure. FIG. 3B is a flowchart of the style transfer method according to the embodiment of the disclosure. FIG. 4 is a schematic diagram of a first convolutional neural network model 400 according to an embodiment of the disclosure.

S300, S302, S304 ~ steps S306, S308 ~ step

Claims (12)

A style transfer method, applied to a web interface, includes: obtaining a first convolution neural network (CNN) model, wherein the first convolution neural network model is Training is used to extract features from a content image, and includes complex convolutional layers, the complex convolutional layers including a first specific number of feature filters and a second specific number of feature filters ; Reduce the number of the first specific number of feature filters to one eighth of the original; reduce the number of the second specific number of feature filters to one eighth of the original; input the original image to the A first convolutional neural network model; a convolution operation is performed through the first convolutional neural network model to generate a complex first feature map of the original image; wherein, each feature filter is used to extract the original image Different features of the graph, the first specific number is 32, and the second specific number is 64. The method described in item 1 of the scope of patent application further includes: obtaining a second convolutional neural network model; inputting the plural first feature maps and a style image of the original image into the first image Two convolutional neural network models; through the convolution operation, the complex number first feature maps are extracted from the first feature maps associated with the The spatial feature map of the original image, and extract the non-spatial feature map associated with the style map from the style map; through a loss function operation, the spatial feature map associated with the original image is associated with the style The non-spatial feature maps of the map are merged to obtain a composite image after style transfer; the composite image is displayed in the web page interface. According to the method described in claim 1, wherein the first convolutional neural network model performs a convolution operation on the original image with the first specific number and the second specific number of feature filters for Extract and generate the plural first feature maps of the original image. The method described in item 2 of the scope of patent application, wherein the spatial feature map includes a shape feature map and a boundary feature map; the non-spatial feature map includes a color feature map and a texture feature map. The method described in item 2 of the scope of patent application, wherein the second convolutional neural network model is VGG19. The method described in item 2 of the scope of the patent application, wherein the loss function calculation includes: calculating a first feature gap between the spatial feature map associated with the original image and the composite image; and calculating the spatial feature map associated with the style map A second feature gap between the non-spatial feature map and the composite image; sum the first feature gap and the second feature gap to obtain the loss Function; Through a gradient descent method, the loss function is minimized to obtain the composite image. A computer program product applied to a web interface for style transfer between an original image and a style image; the computer program product loads the program through the computer to execute: a first call command to make the computer's A processor obtains a first convolutional neural network model from the storage of the computer; wherein the first convolutional neural network model is trained to extract features in the original image, and includes a complex convolutional layer, The plural convolutional layers include a first specific number of feature filters and a second specific number of feature filters; a filter setting instruction causes the processor to reduce the number of the first specific number of feature filters to One-eighth of the original, and the second specific number of feature filters is reduced to one-eighth; a first input instruction causes the processor to input the original image to the first convolution Neural network model; a first feature extraction instruction that causes the processor to perform a convolution operation through the first convolutional neural network module to generate a complex first feature map of the original image; wherein, each feature The filter is used to capture different features of the original image, the first specific number is 32, and the second specific number is 64. For example, the computer program product described in item 7 of the scope of patent application further includes: a second call instruction to make the processor obtain a first call from the computer's memory Two convolutional neural network models; a second input command to enable the processor to input the plural first feature maps of the original image and the style map to the second convolutional neural network model; a second feature The extraction instruction causes the processor to extract the spatial feature maps associated with the original image from the plural first feature maps through the convolution operation, and extract the spatial feature maps associated with the style map from the style map Non-spatial feature map; a synthesis instruction that causes the processor to perform a loss function operation to merge the spatial feature map associated with the original image and the non-spatial feature map associated with the style map to obtain the classic style A composite image after the transfer; a display instruction causes the processor to display the composite image in the web page interface. For example, the computer program product described in item 7 of the scope of patent application, wherein the first convolutional neural network model performs a convolution operation on the original image with the first specific number and the second specific number of feature filters, Used to extract and generate the plural first feature maps of the original image. The computer program product described in item 8 of the scope of patent application, wherein the spatial feature map includes a shape feature map and a boundary feature map; the non-spatial feature map includes a color feature map and a texture feature map. The computer program product described in item 8 of the scope of patent application, wherein the second convolutional neural network model is VGG19. The computer program product described in item 8 of the scope of patent application, wherein the loss function calculation includes: calculating a spatial feature map associated with the original image and a first feature of the composite image Calculate a second feature gap between the non-spatial feature map and the composite image associated with the style map; add the first feature gap and the second feature gap to obtain the loss function; The gradient descent method minimizes the loss function to obtain the composite image.

Images