CN115641253A - Material nerve style migration method for improving content aesthetic quality - Google Patents
Material nerve style migration method for improving content aesthetic quality Download PDFInfo
- Publication number
- CN115641253A CN115641253A CN202211182280.3A CN202211182280A CN115641253A CN 115641253 A CN115641253 A CN 115641253A CN 202211182280 A CN202211182280 A CN 202211182280A CN 115641253 A CN115641253 A CN 115641253A
- Authority
- CN
- China
- Prior art keywords
- style
- pictures
- data set
- image
- picture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 239000000463 material Substances 0.000 title claims abstract description 121
- 238000013508 migration Methods 0.000 title claims abstract description 45
- 230000005012 migration Effects 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 title claims abstract description 32
- 210000005036 nerve Anatomy 0.000 title claims description 10
- 230000001537 neural effect Effects 0.000 claims abstract description 20
- 238000012216 screening Methods 0.000 claims abstract description 9
- 230000006870 function Effects 0.000 claims description 8
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 2
- 230000004048 modification Effects 0.000 claims description 2
- 230000002194 synthesizing effect Effects 0.000 claims description 2
- 238000012549 training Methods 0.000 claims description 2
- 230000000717 retained effect Effects 0.000 abstract 1
- 239000004575 stone Substances 0.000 description 11
- 230000000694 effects Effects 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 7
- 239000002184 metal Substances 0.000 description 7
- 229910052751 metal Inorganic materials 0.000 description 7
- 239000002023 wood Substances 0.000 description 6
- 230000003993 interaction Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 239000007769 metal material Substances 0.000 description 3
- 239000002994 raw material Substances 0.000 description 3
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241000581017 Oliva Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses a material neural style migration method for improving content aesthetic quality, which comprises the following steps of 1, screening out a beautiful picture with a material part occupying a larger total area of the picture from a style and material data set; and add to the pictureA selected number of times tag; step 2, using the style material picture with higher aesthetic index and more selected times to the original picture I org Neural style migration to generate multiple migrated pictures I gen (ii) a Step 3, from the original drawing I org The region RegionTarget of the material to be transferred is segmented; step 4, a plurality of migrated images I gen The content corresponding to the region of the RegionTarget is synthesized with the background part of the original image to form a plurality of synthesized images; and 5, selecting the most beautiful one of the plurality of synthetic images, and adding 1 to the selected times label value of the style and material picture corresponding to the selected times label value. Thereby enabling the generation of the most elegant picture with the associated experience retained.
Description
Technical Field
The invention belongs to the field of image processing, in particular to a neural style migration algorithm based on style conversion of images, which is mainly applied to the field of virtual reality/augmented reality.
Background
The material neural style migration algorithm is used for converting a certain original material in an original image into a specified another material, for example, converting the content of a stone material in an original image A into the content of a specified metal material, wherein the content style of the metal material is derived from an image B in a data set, converting a house made of the stone material into a house made of the metal material, and keeping the rest unchanged to obtain a synthesized image C. The algorithm can be widely applied to the image synthesis direction in the field of virtual reality and has good application value.
At present, the effect of the result is judged by depending on the similarity of styles of the generated picture and the designated material part, but in addition to the characteristic of similarity, the form of the generated picture can be varied, and in the varied results, how to select the most beautiful picture, namely, how to select the most beautiful image result of the synthesized metal house, the style of which is metallic, but the metal house can also have a plurality of effects, and how to further select the most beautiful image result, the problem is that no related research scheme exists at present.
Disclosure of Invention
The technical problem to be solved by the invention is how to select the most beautiful image from a plurality of image effects after style migration, wherein the image B of a raw material is beautiful, and if the image C generated after the migration is beautiful, the related experience is fed back to the image in the field of the raw material, so that the checked knowledge is learned in the next task; therefore, the invention provides a material neural style migration method for improving the content aesthetic quality, which can generate the most beautiful picture on the basis of realizing the style migration of the generated image on the basis of the classical material neural style migration algorithm, thereby realizing the virtual reality effect more beautifully and increasing the comfort level of human-computer interaction; and secondly, the related experience is reserved and learned in the next task, so that the computing resources can be reduced, and a better engineering effect is obtained.
The method is based on the bases of deep learning, aesthetic indexes and the like, is oriented to the aesthetic problem of the generated picture, can generate the objective and beautiful picture on the basis of realizing style migration of the generated picture, can realize the virtual reality effect more beautifully, and increases the comfort level of human-computer interaction.
The invention provides a material neural style migration method for improving content aesthetic quality. The details are as follows
Step 1, preprocessing a style material data set Dataset1, screening out style material pictures with the material part occupying the total area ratio of the pictures larger than a threshold value T1, and forming a data set Dataset3;
further screening style material pictures with aesthetic index scores larger than a threshold value T2 from the data set Dataset3 to form a data set Dataset4;
adding a selected time label Pic4_ selected to all style material pictures in the Dataset4;
the main content of the process is that the larger the ratio of the material part in the sample of the Dataset3 to the total area of the picture is, the better the higher the aesthetic index score of the part is, the better the process is.
Step 2, selecting the style material picture with higher aesthetic index and the picture with more selected times Pic4 selected to be used for original drawing I org The overall nervous style is transferred to generate a plurality of transferred pictures I gen ；
Step 3, to the original drawing I org The material is finely divided to obtain an original drawing I org Defining a middle to-be-converted material part RegionTarget and an image I org Matrix I with completely consistent size and all numerical values of 0 mask And is combined with mask All the parts corresponding to the positions of the regions of the material to be converted, namely the regions of the material to be converted, are set to be 1.
Step 4, synthesizing the images and outputting a picture I out Expressed as follows:
I out ＝I gen I mask +I org (1-I mask )
wherein, I gen Are synthesized images, each synthesized image I out And a post-migration picture I gen Corresponding;
step 5, calculating all the synthetic images I out Obtaining the synthetic image I with the maximum NIMA value out That is, the finally obtained image is obtained, and 1 is added to the Pic4_ selected tag value of the style and material picture corresponding to the finally obtained image.
Further, style material pictures with the material part accounting for the total area ratio of the pictures larger than a threshold value T1 are screened out in the step 1, and a data set Dataset3 is formed; the method specifically comprises the following steps:
step 1.1, acquiring a primary material area tag data set SegDateset2;
randomly extracting all image-level label picture samples in FMD and EFMD to obtain a data set SegDateset1;
using a weakly supervised approach, learning the pixel-level semantic affinity PSA from the class activation map CAM of the multi-label CNN network, a primary material region label dataset SegDateset2 is obtained from the image-level label dataset SegDateset 1.
Step 1.2, training a harmonic dense connection network HarDNet named HarDNet1 by adopting a primary material area label data set SegDateset2, and finely adjusting the network HarDNet1 by using another group of pixel level annotation data set SegDateset3 to obtain a trained harmonic dense connection network HarDNet _ seg;
the data set SegDateset3 was randomly drawn from all pixel level label picture samples in FMD and EFMD.
Step 1.3, segmenting pixel level semantics of all images in a data set Dataset1 by adopting a trained HarDNet1 harmonic dense connection network; and labeled as Dataset2;
step 1.4, counting the total number of pixels of various semantic contents in the data set Dataset2 image, calculating the proportion of the pixels to the whole image, sequencing from high to low, and outputting the area ratio Ph2 (i) of the semantic classification with the highest sequence to each image in the data set Dataset2, wherein i is a number and the total number itotal of the images from 1 to Dataset 2.
Defining a threshold value T1 (0-1), deleting pictures Ph2 (i) < = T1 in the Dataset2, outputting the rest pictures as Dataset3, and obtaining a pixel-level semantic data set Dataset3 with the area of a single material occupying the whole picture and exceeding T1.
Further, in step 2, the style material picture with higher aesthetic index and the picture with more selected times Pic4_ selected are selected to be the original picture I org Moving the whole nerve style to generate multiple moved pictures I gen (ii) a The method comprises the following specific steps:
step 2.1, sorting each type of pictures in the data set Dataset4 from high to low according to the aesthetic NIMA index, wherein the style material category to be migrated has K pictures in total, and adopting the first N1 style pictures to perform original drawing I org Overall nerve style migration to obtain N1 migrated output pictures I gen (n 1); wherein the value range of N1 is 1-N1;
step 2.2, in the style and material category to be transferred; when the Pic4_ selected values of all pictures in the class are 0, let N3=0; entering the step 2.4;
otherwise, sorting the pictures from high to low according to the Pic4_ selected values of the pictures in the style material category to be migrated, selecting the style material pictures with the Pic4_ selected values not being 0, and taking the first N2 style material pictures;
step 2.3, removing the repeated pictures in the first N1 style material pictures in the step 2.1 from the first N2 style material pictures to obtain N3 style material pictures,
step 2.4, adopting the N3 style material pictures to carry out comparison on the original drawing I org Migration of overall neural style; respectively obtaining N3 transferred pictures I gen (n 3); wherein the value range of N3 is 1-N3;
step 2.5, finally obtaining N1+ N3 post-migration I gen And (5) collecting pictures.
Further, in step 2, a classical neural style migration method is adopted for material migration, and the modification loss function is as follows:
wherein, alpha and beta are both weight coefficients,
the index of the loss of content is represented,
the table style loss index, NIMA (k, clas) represents the NIMA value of the k-th style material picture in the clas class;
further, step 3 adopts the harmonic dense connection network HarDNet _ seg trained in step 1 to correct the original image I org The material is finely divided.
Has the advantages that: by the method, firstly, the generated image can be generated into the most beautiful picture on the basis of realizing the style migration on the basis of the classical material nerve style migration algorithm, so that the virtual reality effect can be more beautifully realized, and the comfort level of human-computer interaction is increased; and secondly, related experiences are reserved and learned in the next task, so that the computing resources can be reduced, and a better engineering effect is obtained.
Drawings
FIG. 1 is an overall flow diagram of the present invention;
DETAILED DESCRIPTION OF EMBODIMENT (S) OF INVENTION
The invention relates to a neural style migration method of a material for improving content aesthetic quality, which comprises the following steps:
step 1, preprocessing a style and material data set Dataset1, screening out pictures with the material part occupying the total area ratio of the pictures larger than a threshold value T1, and forming a data set Dataset3; further screening pictures with aesthetic index scores larger than a threshold value T2 from the data set Dataset3 to form a data set Dataset4; adding a selected frequency label Pic4_ selected to all style material pictures in the data set Dataset4, and setting an initial value of the selected frequency label Pic4_ selected of each style material picture as 0;
the method comprises the following specific steps:
step 1.1, screening out pictures with material portions occupying more than a threshold value T1 in the total area ratio of the pictures to form a data set Dataset3;
generally speaking, in an existing data set, a large number of semantic samples are marked at an image level, for example, a large area of a stone is marked in a graph, the whole graph is marked with a semantic of the stone manually, while a small number of semantic samples at a pixel level are marked in a graph, for example, a large area of the stone is marked in a graph, each pixel of a part of stone materials is marked with the stone, each pixel of a part of other materials is marked with other semantic, and this step needs refined pixel-level semantic identification.
Step 1.1.1, acquiring a primary material area tag data set SegDateset2;
the image samples in the Flickr Material Database (FMD) and the Extended FMD (EFMD) include image-level label image samples and pixel-level label image samples, in this embodiment, the data set SegDateset1 is obtained by randomly extracting 90% of all image-level label image samples in the FMD and the EFMD and then summarizing, and the model setting in the process of this step of the scheme is consistent with the scheme proposed by Ahn and Kwak.
The approach proposed by Ahn and Kwak (Ahn, J.; kwak, S.Learing pixel-level sensing with image-level super Vision for the super Vision of the procedure of the IEEE Conference Vision and Pattern recognition. IEEE,2018, pp.4981-4990.) is used herein to obtain from a Class Activation Map (CAM) of a multi-tag CNN network (Zhou, B.; khosla, A.; lapedriza, A.A.; oliva, A.toralba, A.left semantic facility for the semantic registration, A.A. Torllba, A.Learing devices for the high semantic registration of the image data set of the IEEE tag of the IEEE trade, the high semantic registration of the primary tag of the IEEE trade data set 2921, the low semantic data set of the IEEE tag of the View of the IEEE trade data set 2921, the high semantic data set of the primary tag of the high semantic registration of the image of the IEEE trade.
Step 1.1.2, a harmonic dense connection network (HarDNet) named HarDNet1 was trained using a primary material area tag dataset SegDateset2 (Chao, P.; kao, C.Y.; ruan, Y.S.; huang, C.H.; lin, Y.L.Hardnet: all memory transmission network in Proceedings of the IEEE International Conference on Computer Vision.IEEE,2019, pp.3552-3561), and the network HarDNet1 was trimmed using another set of pixel level annotation datasets SegDateteset 3.
In this embodiment, the data set SegDateset3 is obtained by randomly extracting 90% of FMD number and all pixel level label picture samples in the EFMD and summarizing the samples. The network structure of HarDNet1 is consistent with that in the paper of Chao, P, and the fine tuning process is consistent. Obtaining a trained harmonic dense connection network (HarDNet _ seg) for the original image I in the later step org And refining and dividing the material part.
Step 1.1.3, segmenting pixel level semantics of all images in a data set Dataset1 by adopting a trained HarDNet1 harmonic dense connection network; and labeled as Dataset2;
step 1.1.4, counting the total number of pixels of various semantic contents in the data set Dataset2 images, calculating the proportion of the pixels to the whole image, sequencing from high to low, and outputting the area ratio Ph2 (i) of the semantic classification with the highest sequence to each image in the data set Dataset2, wherein i is a number and the total number itotal of the images from 1 to Dataset 2.
Defining a threshold value T1 (0-1), deleting pictures Ph2 (i) < = T1 in the Dataset2, outputting the rest pictures as Dataset3, and obtaining a pixel-level semantic data set Dataset3 with the area of a single material occupying the whole picture and exceeding T1.
Step 1.2, further screening out pictures with the aesthetic NIMA index score larger than a threshold value T2 from the data set Dataset3, and sequencing each type of pictures in the data set Dataset3 from high to low according to the aesthetic NIMA index to form a data set Dataset4;
the NIMA index is derived from a paper of "Neural Image Assessment", wherein google research team proposes a deep CNN capable of predicting the distribution of human opinion on Image evaluation from a direct look and feel (technical point of view) and an attraction (aesthetic point of view). In the approach taken by google, the NIMA model does not simply score images for high or low scores, nor does it regress for average scores, but rather makes a score distribution for any image-in the range of 1 to 10, NIMA will assign the score probability of this image to the 10 scores, with the quality ranked from high to low with score.
Step 1.2.1, calculating NIMA for all images in Dataset3 by using a method in the paper of Neural Image Association to obtain an NIMA index result NIMA (j) of a picture j, wherein the value of the j is the total number jtotal of the pictures from 1 to Dataset3.
Step 1.2.2, defining a threshold value T2 (0-1), deleting all NIMA (j) < = T2 pictures in the Dataset3, and in the rest pictures, sorting each type of style material from high to low according to the aesthetic NIMA index to form a data set Dataset4, wherein the value NIMA of the aesthetic index is added in each picture in the data set Dataset4, and the selected times Pic4_ selected;
in this embodiment, T2 is set to 3, so that pictures with unclear, poor saturation, poor aesthetic sense, and the like are basically deleted.
The pictures in Dataset4 are denoted Pic4 (K, clas), K is the number of this picture in the clas class, the value range of K is 1 to K, K denotes the total number of pictures in each clas class, and Dataset4 has NIMA (K, clas) value, which is used in later retrieval, is the number of the class, e.g. 10 material classes in the Dataset, if the stone is the 1 st material class, then the clas is 1. And adopting Pic4_ selected (k, clas) to represent the number of times the k-th picture in the clas class is selected.
Thus, a preprocessed and screened data set is obtained, the area of all main body materials in the data set accounts for a large image ratio, the image aesthetic index is high, the images in the data set are separately tabulated according to the material classes, sorted according to the image aesthetic index, the value of the aesthetic index is kept, and a selected time variable Pic4_ selected (k, clas) is set for each picture Pic4 (k, clas), and all the Pic4_ selected (k, clas) variables are 0 at the beginning.
Step 2, selecting the style material picture with higher aesthetic index and the picture with more selected times Pic4 selected to be used for drawing the original picture I org The overall nervous style is transferred to generate a plurality of transferred pictures I gen ；
Material transformations were performed using the traditional neural style migration method of Gatys (Gatys, L.A.; ecker, A.S.; bethge, M.Image style transfer using a connected neural network, in Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. IEEE,2016, pp.2414-2423.) which uses a pre-trained VGG19 network to extract content and style features. According to the definition of the loss function of the classical traditional neural style migration method, the loss function is a new content loss function and a classical style loss function, wherein the feature distance is particularly minimized to be divided by the image aesthetic index (content loss index)
NIMA (k, clas)) and its Gram matrix (style loss index)
) To optimize the migrated images.
When the material image B is retrieved and converted in the data set, the content loss index of the classical traditional nerve style migration method is removed
Index of style loss
And also calculates the aesthetic index NIMA of the lattice material picture in Dataset4Since the larger the NIMA, the better and the smaller the loss function, the better, the reciprocal is obtained, which results in a new loss function:
since the target class is known at the time of the task, the clas is a known number, k of the search target is the number of the graph in the clas class, k is also known, and NIMA (k, clas) is known. The calculation of the classical neural style migration method is not affected as a whole. In the present embodiment, both α and β are set to 0.5.
The specific steps of the original image migration are as follows:
step 2.1, in the data set Dataset4, the style material category to be migrated has K pictures in total, and the original drawing I is subjected to the first N1 style pictures org Overall nerve style migration to obtain N1 migrated output pictures I gen (n 1); wherein the value range of N1 is 1-N1;
step 2.2, in the style and material category to be transferred; when the Pic4_ selected values of all pictures in the class are 0, let N3=0; entering the step 2.4;
otherwise, sorting the pictures from high to low according to the Pic4_ selected values of the pictures, selecting the pictures with the Pic4_ selected values not being 0, and taking the first N2 pictures;
step 2.3, removing the repeated pictures in the first N1 pictures in the step 2.1 from the first N2 pictures to obtain N3 pictures;
step 2.4, adopting the N3 pictures to carry out comparison on the original image I org Migration of overall neural style; respectively obtaining N3 transferred pictures I gen (n 3); wherein the value range of N3 is 1-N3;
step 2.5, obtaining N1+ N3 post-migration I gen Collecting pictures;
step 3, using the trained harmonic dense connection network (HarDNet _ seg) to finely divide the material of the original drawing to obtain an original drawing I org Defining a middle to-be-converted material part RegionTarget and an image I org The sizes are completely consistentMatrix I with values of all 0 mask And is combined with mask All the parts corresponding to the positions of the regions of the material to be converted, namely the regions of the material to be converted, are set to be 1.
Step 4, transferring the image I gen And matrix I mask Multiplying to obtain a migrated image I gen The content corresponding to the region of the RegionTarget, namely the material region of the image after the migration;
the original image I org And matrix I mask Multiplying to obtain an original image I org The background area of (1);
the material area of the image after the migration and the original image I org Is synthesized to the final output (I) out ) Wherein it is defined as follows:
I out ＝I gen I mask +I org (1-I mask )
here, I gen Is a synthesized image, I mask E {0,1} is the region mask obtained by HarDNet, I org Is the image of the content with the original object. Therefore, due to the migrated image I gen N1+ N3, so a group of pictures with corresponding quantity can be obtained;
step 5, evaluating and optimizing
In this step, the invention is to solve the problem that the generated image C is beautiful; and secondly, if the generated image C is beautiful, the related experience is fed back to the image in the field of raw materials, so that the experience knowledge can be learned in the next task.
The specific operation comprises the following steps:
step 5.1, for this group I out Calculating an Image by using a method in a paper of Neural Image Assessment to obtain an NIMA index, and selecting a picture with the maximum NIMA value as the final output of the whole algorithm from high to low;
and 5.2, increasing 1 for the Pic4_ selected (k, clas) variable of the original image of the selected picture in the Dataset4, increasing the probability that the style image is selected again in the step 2 in the next task, and realizing the learning of the empirical knowledge.
Therefore, through the implementation of the method, firstly, the generated image can be generated into the most beautiful picture on the basis of realizing the style migration on the basis of the classical material nerve style migration algorithm, so that the virtual reality effect can be realized more beautifully, and the comfort level of human-computer interaction is increased; and secondly, related experience is reserved and learned in the next task, so that computing resources can be reduced, and a better engineering effect is obtained.
Suppose original drawing I org The content of (1) comprises a bowl made of wood and a spoon made of stone, wherein the bowl made of wood is a material part to be replaced by styles. The task of material style migration turns into target style material metal to the bowl of wood material, and all the other remain unchanged, and the ladle of stone material is unchangeable promptly. Step 2, selecting a plurality of metal style material pictures with high NIMA index and more selected times in the metal class; to the original drawing I org The overall style of all the contents is transferred, namely, the bowl made of wood and the spoon made of stone are all converted into metal, and a plurality of pictures I are obtained gen . Step 3, the original drawing I is processed org The position RegionTarget of the part of the bowl made of wood is divided and forms a mask matrix I mask . Step 4 is to pass the mask matrix I mask Handle I gen Part of middle metal bowl is used for replacing original drawing I org The wood bowl part in the table obtains a plurality of outputs I out . Step 5 is outputting I from a plurality of outputs out Selecting the most beautiful sheet I out And adding 1 to the selected times of the corresponding style and material picture to increase the probability of selecting the picture next time.
Claims (5)
1. A neural style migration method for materials with improved content aesthetic quality is characterized by comprising the following steps:
step 1, preprocessing a style material data set Dataset1, screening out style material pictures with the material part occupying the total area ratio of the pictures larger than a threshold value T1, and forming a data set Dataset3;
further screening style material pictures with aesthetic index scores larger than a threshold value T2 from the data set Dataset3 to form a data set Dataset4;
adding a selected time label Pic4_ selected to all style material pictures in the Dataset4;
step 2, selecting the style material pictures with higher aesthetic index and the style material pictures with more selected times Pic4_ selected, and drawing the original picture I org The overall nervous style is transferred to generate a plurality of transferred pictures I gen ；
Step 3, for the original drawing I org The material is finely divided to obtain an original drawing I org Defining a middle to-be-converted material part RegionTarget and an image I org Matrix I with completely consistent size and all numerical values of 0 mask And is combined with mask All parts corresponding to the positions of the regioontargets of the materials to be converted are set to be 1;
step 4, synthesizing the images and outputting a picture I out Expressed as follows:
I out ＝I gen I mask +I org (1-I mask )
wherein, I gen Are synthesized images, each synthesized image I out And a post-migration picture I gen Corresponding;
step 5, calculating all the synthetic images I out Obtaining the synthetic image I with the maximum NIMA value out That is, the finally obtained image is obtained, and 1 is added to the Pic4_ selected tag value of the style and material picture corresponding to the finally obtained image.
2. The neural style migration method of materials for improving content aesthetic quality according to claim 1, characterized in that style material pictures with the material part occupying the total area ratio of the pictures larger than a threshold value T1 are screened in step 1 to form a data set Dataset3;
the method specifically comprises the following steps:
step 1.1, acquiring a primary material area tag data set SegDateset2;
randomly extracting all image-level label picture samples in FMD and EFMD to obtain a data set SegDateset1;
learning a pixel-level semantic affinity PSA from a class activation map CAM of a multi-label CNN network by using a weakly supervised scheme, and obtaining a primary material region label data set SegDateset2 from an image-level label data set SegDateset1;
step 1.2, training a harmonic dense connection network HarDNet named HarDNet1 by adopting a primary material area label data set SegDateset2, and finely adjusting the network HarDNet1 by using another group of pixel level annotation data set SegDateset3 to obtain a trained harmonic dense connection network HarDNet _ seg;
the data set SegDateset3 is obtained by randomly extracting all pixel level label picture samples in FMD and EFMD;
step 1.1.3, segmenting pixel level semantics of all images in a data set Dataset1 by adopting a trained HarDNet1 harmonic dense connection network; and labeled as Dataset2;
step 1.1.4, counting the total number of pixels of various semantic contents in the data set Dataset2 image, calculating the proportion of the pixels to the whole image, sequencing from high to low, and outputting the area ratio Ph2 (i) of the semantic classification with the highest sequencing to each image in the data set Dataset2, wherein i is a number and the total number itotal of the images from 1 to Dataset2;
defining a threshold value T1 (0-1), deleting pictures Ph2 (i) < = T1 in the Dataset2, outputting the rest pictures as Dataset3, and obtaining a pixel-level semantic data set Dataset3 with the area of a single material occupying the whole picture and exceeding T1.
3. The method as claimed in claim 1, wherein the step 2 of selecting the material with higher aesthetic index and selecting the selected pictures with higher Pic4 selected times to obtain the original image I org Moving the whole nerve style to generate multiple moved pictures I gen (ii) a The method comprises the following specific steps:
step 2.1, sorting each type of pictures in the Dataset Dataset4 from high to low according to the aesthetic NIMA index, wherein the style material category to be migrated has K pictures in total, and adopting the first N1 style pictures to perform original drawing I org Overall nerve style migration to obtain N1 migrated output pictures I gen (n 1); wherein the value range of N1 is 1-N1;
step 2.2, in the style and material category to be transferred; when the Pic4_ selected values of all pictures in the class are 0, let N3=0; entering the step 2.4;
otherwise, sorting the pictures from high to low according to the Pic4_ selected values of the pictures in the style material category to be migrated, selecting the style material pictures with the Pic4_ selected values not being 0, and taking the first N2 style material pictures;
step 2.3, removing the repeated pictures in the first N1 style material pictures in the step 2.1 from the first N2 style material pictures to obtain N3 style material pictures;
step 2.4, adopting the N3 style material pictures to carry out comparison on the original drawing I org Migration of overall neural style; respectively obtaining N3 transferred pictures I gen (n 3); wherein the value range of N3 is 1-N3;
step 2.5, finally obtaining N1+ N3 post-migration I gen And (5) collecting pictures.
4. The neural-style migration method of materials for improving content aesthetic quality according to claim 1, wherein a classical neural-style migration method is adopted for material migration in the step 2, and the modification loss function is as follows:
wherein, alpha and beta are both weight coefficients,
the index of the loss of content is represented,
the table style loss index, NIMA (k, clas), represents the NIMA value of the k-th style material picture in the clas class.
5. The content-oriented material nervous style migration method with improved aesthetic quality, according to claim 2, characterized in that in step 3, the HarDNet _ seg pair of the original drawing I trained in step 1 is adopted org The material of (2) is finely divided.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211182280.3A CN115641253B (en) | 2022-09-27 | 2022-09-27 | Material nerve style migration method for improving aesthetic quality of content |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211182280.3A CN115641253B (en) | 2022-09-27 | 2022-09-27 | Material nerve style migration method for improving aesthetic quality of content |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115641253A true CN115641253A (en) | 2023-01-24 |
| CN115641253B CN115641253B (en) | 2024-02-20 |
Family
ID=84941974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211182280.3A Active CN115641253B (en) | 2022-09-27 | 2022-09-27 | Material nerve style migration method for improving aesthetic quality of content |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115641253B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110458750A (en) * | 2019-05-31 | 2019-11-15 | 北京理工大学 | A kind of unsupervised image Style Transfer method based on paired-associate learning |
| CN111242841A (en) * | 2020-01-15 | 2020-06-05 | 杭州电子科技大学 | Image background style migration method based on semantic segmentation and deep learning |
| CN111950655A (en) * | 2020-08-25 | 2020-11-17 | 福州大学 | Image aesthetic quality evaluation method based on multi-domain knowledge driving |
| WO2022090483A1 (en) * | 2020-11-02 | 2022-05-05 | Tomtom Global Content B.V. | Neural network models for semantic image segmentation |
| CN114581356A (en) * | 2022-05-09 | 2022-06-03 | 南京理工大学 | Image enhancement model generalization method based on style migration data augmentation |
-
2022
- 2022-09-27 CN CN202211182280.3A patent/CN115641253B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110458750A (en) * | 2019-05-31 | 2019-11-15 | 北京理工大学 | A kind of unsupervised image Style Transfer method based on paired-associate learning |
| CN111242841A (en) * | 2020-01-15 | 2020-06-05 | 杭州电子科技大学 | Image background style migration method based on semantic segmentation and deep learning |
| CN111950655A (en) * | 2020-08-25 | 2020-11-17 | 福州大学 | Image aesthetic quality evaluation method based on multi-domain knowledge driving |
| WO2022090483A1 (en) * | 2020-11-02 | 2022-05-05 | Tomtom Global Content B.V. | Neural network models for semantic image segmentation |
| CN114581356A (en) * | 2022-05-09 | 2022-06-03 | 南京理工大学 | Image enhancement model generalization method based on style migration data augmentation |
Non-Patent Citations (2)
| Title |
|---|
| HOSSEIN TALEBI: "NIMA: Neural Image Assessment", 《ARXIV》, pages 1 - 15 * |
| 孙冬: "局部图像风格迁移绘制算法", 《安徽大学学报（自然科学版）》, pages 72 - 78 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115641253B (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107977671B (en) | Tongue picture classification method based on multitask convolutional neural network | |
| CN107256246B (en) | printed fabric image retrieval method based on convolutional neural network | |
| CN110837836B (en) | Semi-supervised semantic segmentation method based on maximized confidence | |
| CN107622104B (en) | Character image identification and marking method and system | |
| WO2017113232A1 (en) | Product classification method and apparatus based on deep learning | |
| CN108734719A (en) | Background automatic division method before a kind of lepidopterous insects image based on full convolutional neural networks | |
| CN108121781B (en) | Related feedback image retrieval method based on efficient sample selection and parameter optimization | |
| CN108629783B (en) | Image segmentation method, system and medium based on image feature density peak search | |
| CN109740686A (en) | A kind of deep learning image multiple labeling classification method based on pool area and Fusion Features | |
| CN108846444A (en) | The multistage depth migration learning method excavated towards multi-source data | |
| CN105469080B (en) | A kind of facial expression recognizing method | |
| CN102314614A (en) | Image semantics classification method based on class-shared multiple kernel learning (MKL) | |
| CN110866896A (en) | Image saliency target detection method based on k-means and level set super-pixel segmentation | |
| CN109272011A (en) | Multitask depth representing learning method towards image of clothing classification | |
| Akhand et al. | Convolutional Neural Network based Handwritten Bengali and Bengali-English Mixed Numeral Recognition. | |
| CN112507800A (en) | Pedestrian multi-attribute cooperative identification method based on channel attention mechanism and light convolutional neural network | |
| Islam et al. | A CNN based approach for garments texture design classification | |
| Qi et al. | Personalized sketch-based image retrieval by convolutional neural network and deep transfer learning | |
| Zhao et al. | Semi-supervised learning-based live fish identification in aquaculture using modified deep convolutional generative adversarial networks | |
| CN111783543B (en) | Facial activity unit detection method based on multitask learning | |
| Cogswell et al. | Combining the best of graphical models and convnets for semantic segmentation | |
| CN110991554B (en) | Improved PCA (principal component analysis) -based deep network image classification method | |
| CN115457332A (en) | Image multi-label classification method based on graph convolution neural network and class activation mapping | |
| CN112990340B (en) | Self-learning migration method based on feature sharing | |
| Al-Hmouz et al. | Enhanced numeral recognition for handwritten multi-language numerals using fuzzy set-based decision mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |