WO2020220807A1 - 图像生成方法及装置、电子设备及存储介质 - Google Patents

图像生成方法及装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2020220807A1
WO2020220807A1 PCT/CN2020/076470 CN2020076470W WO2020220807A1 WO 2020220807 A1 WO2020220807 A1 WO 2020220807A1 CN 2020076470 W CN2020076470 W CN 2020076470W WO 2020220807 A1 WO2020220807 A1 WO 2020220807A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
style
feature
target
content
Prior art date
Application number
PCT/CN2020/076470
Other languages
English (en)
French (fr)
Inventor
沈志强
黄明杨
石建萍
松永英树
Original Assignee
商汤集团有限公司
本田技研工业株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 商汤集团有限公司, 本田技研工业株式会社 filed Critical 商汤集团有限公司
Priority to JP2021564168A priority Critical patent/JP7394147B2/ja
Priority to CN202080032107.8A priority patent/CN113841179A/zh
Publication of WO2020220807A1 publication Critical patent/WO2020220807A1/zh
Priority to US17/452,388 priority patent/US11900648B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects

Definitions

  • the present disclosure relates to the field of computer technology, and in particular to an image generation method and device, electronic equipment, and storage medium.
  • the overall conversion of image styles can be achieved through deep learning and other methods.
  • the brightness, contrast, lighting, color, artistic features, or artwork of the image can be converted to obtain images with different styles.
  • the transformation of style can only affect the overall image, and the style of the target object in the image can only be transformed with the overall style of the image, which cannot reflect the style of the target object, resulting in monotonous style of the transformed image and the overall image Inconsistent with the area where the target object is located.
  • the embodiments of the present disclosure propose an image generation method and device, electronic equipment, and storage medium.
  • embodiments of the present disclosure provide an image generation method, including:
  • a third image is generated according to the content feature and the target style feature, so that the third image has a content corresponding to the content feature and a style corresponding to the target style feature.
  • the target style feature determined by the full-image style feature and the object style feature of the second image and the content feature of the first image can be used to obtain content corresponding to the content feature and consistent with the target style.
  • the third image with the feature corresponding to the style can not only change the overall image style of the image when the image is styled, but also appropriately convert the style of the local image block where the object in the image is located, so that the overall image of the image Coordinate with the area where the object is located, improve the fidelity of the style transfer image, and improve the detection accuracy of the object in the image.
  • the generated third image has content corresponding to the content feature of the first image and a style corresponding to the target style feature, which expands the richness of image samples with content corresponding to the content feature of the first image, especially for Difficult image samples or a few image samples greatly reduce the cost of manual collection and annotation.
  • an image generation device including:
  • the first extraction module is used to extract content features of the first image
  • the second extraction module is used to extract the full image style features of the second image and the object style features of the partial image blocks in the second image including the object respectively, wherein the second image and the first image Different styles
  • a determining module configured to determine a target style feature at least according to the full image style feature and the object style feature
  • the first generating module is configured to generate a third image according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature.
  • an electronic device including:
  • a memory for storing processor executable instructions
  • the processor is configured to execute the above-mentioned image generation method in the embodiment of the present disclosure.
  • an embodiment of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions, when executed by a processor, implement the image generation method described in the embodiment of the present disclosure.
  • Fig. 1 shows a first flowchart of an image generation method according to an embodiment of the present disclosure
  • Figure 2 shows a second flowchart of an image generation method according to an embodiment of the present disclosure
  • Fig. 3 shows a schematic diagram of style feature fusion according to an embodiment of the present disclosure
  • Fig. 4 shows a third flowchart of an image generation method according to an embodiment of the present disclosure
  • Fig. 5 shows a fourth flowchart of an image generation method according to an embodiment of the present disclosure
  • FIGS. 6A-6C show schematic diagrams of application of an image generation method according to an embodiment of the present disclosure
  • FIG. 7 shows a block diagram of an image generating device according to an embodiment of the present disclosure
  • Fig. 8 shows a second block diagram of an image generating device according to an embodiment of the present disclosure
  • FIG. 9 shows a first structural diagram of an electronic device according to an embodiment of the present disclosure.
  • Fig. 10 shows a second structural diagram of an electronic device according to an embodiment of the present disclosure.
  • Fig. 1 shows a first flowchart of an image generation method according to an embodiment of the present disclosure. As shown in Fig. 1, the method includes:
  • step S11 the content feature of the first image is extracted
  • step S12 the full image style features of the second image and the object style features of the partial image blocks in the second image including the object are extracted respectively, wherein the styles of the second image and the first image are different ;
  • a target style feature is determined based on at least the overall image style feature and the object style feature
  • step S14 a third image is generated according to the content feature and the target style feature, so that the third image has content corresponding to the content feature and a style corresponding to the target style feature.
  • the target style feature determined by the full-image style feature and the object style feature of the second image and the content feature of the first image can be used to obtain content corresponding to the content feature and the target style feature.
  • the third image corresponding to the style can not only change the overall image style of the image when performing the style conversion of the image, but also appropriately convert the style of the local image block where the object in the image is located.
  • the area where the object is located is coordinated to improve the fidelity of the image of style transfer, and the detection accuracy of the object in the image can be improved.
  • the generated third image has content corresponding to the content feature of the first image and a style corresponding to the target style feature, which expands the richness of image samples with content corresponding to the content feature of the first image, especially for Difficult image samples or a few image samples greatly reduce the cost of manual collection and annotation.
  • the method may be executed by a terminal device, which may be User Equipment (UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, personal digital processing (Personal Digital Processing) Digital Assistant (PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc.
  • UE User Equipment
  • PDA Personal Digital Processing
  • the method can be implemented by a processor calling computer-readable instructions stored in a memory.
  • the method is executed by a server.
  • both the first image and the second image may be images that include one or more objects, the objects may be objects of different categories, and the first image and/or the second image
  • the image includes at least one of the following objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, buildings, and obstacles.
  • the image styles of the first image and the second image are different, and the image styles may include brightness, contrast, lighting, color, artistic features, or artwork in the image.
  • the first image and the second image may be taken in different environments, for example, images taken in the daytime, night, rain, fog, etc., for example, the first image is taken at a certain place during the day The second image is an image taken at another location at night.
  • the content feature of the first image may be extracted.
  • the content feature can be used to represent the content included in the first image.
  • the content feature can be at least one of the category, shape, and location of the object in the first image, or The content feature may also include the background of the first image and so on.
  • the content feature may include the content feature of the entire image of the first image.
  • the content feature of the first image can be extracted by a feature extractor.
  • the feature extractor may be implemented by a convolutional neural network or the like, and the embodiment of the present disclosure does not limit the implementation of the feature extractor.
  • the content feature may include a feature map or feature vector of the first image obtained by the feature extractor.
  • the content feature of the first image may include the object content feature of the partial image block including the object in the first image, and multiple partial image blocks can be intercepted in the first image, and each partial image block
  • the image block may include an object, and the object content feature of each partial image block is extracted separately.
  • the feature extractor may be used to extract the object content feature of each partial image block of the first image.
  • a feature extractor may be used to extract the full image style feature of the second image and the object style feature of the partial image block including the object in the second image.
  • the feature extractor may be implemented by a neural network such as a convolutional neural network, and the embodiment of the present disclosure does not limit the implementation manner of the feature extractor.
  • the feature extractor may perform extraction processing on the second image and the partial image blocks of the second image respectively to obtain the full image style feature and the object style feature of each object.
  • the full-image style feature and the object style feature may be feature vectors with a length of 8 bits.
  • the embodiment of the present disclosure does not impose any restriction on the representation mode of the full-picture style feature and the object style feature.
  • extracting the object style features of the partial image blocks that include the object in the second image includes: A plurality of partial image blocks are intercepted in the image, and each partial image block includes an object; and the object style feature of each partial image block is extracted respectively.
  • partial image blocks each including an object in the second image may be intercepted in the second image, and the object style feature of each partial image block can be extracted separately by using a feature extractor.
  • extracting the object style features of the partial image blocks including the object in the second image further includes: fusing multiple partial images of the same category of objects The object style feature extracted by the block. For example, residual connection and other processing can be performed on the object style features of multiple objects of the same category, so as to merge the multiple object style features of the same category of objects.
  • the full image style feature and the object style feature of the first image can be extracted in the same way.
  • the full-picture style feature and object style feature of the first image can express the image taken during the day.
  • Style for example, the brightness, contrast, lighting, and color of the image are all the styles of images taken during the day
  • the full image style feature of the first image represents the style information of the first image
  • the object style feature of the first image represents the first image Information about the style of the area where the object in (ie, the partial image block in the first image) is located.
  • the full image style feature and object style feature of the second image can express the style of the image taken at night, the full image style feature of the second image represents the style information of the second image, and the object style feature of the second image represents the style of the second image
  • the style information of the area where the object is located ie, the partial image block in the second image.
  • the target style feature is determined at least according to the overall image style feature and the object style feature of the second image.
  • the style of the second image and the first image are different, the target style feature can be determined according to the full image style feature and the object style feature of the second image, and the style of the first image is changed to a style corresponding to the target style feature and
  • An image with the content of the first image, that is, the third image, the third image may include the first image full image after the style transformation (ie, having content corresponding to the content feature of the first image full image and corresponding to the target style feature A style image), and a partial image block after the style transformation (that is, an image having a content corresponding to the target content feature of the partial image block of the first image and a style corresponding to the target style feature).
  • step S13 may include: fusing the full-image style feature to the object style feature to obtain the target style feature.
  • the full image style feature and the object style feature of the second image are both feature vectors, and residual connection processing can be performed on the full image style feature and the object style feature to integrate the full image style feature into the object style feature. Get the target style characteristics.
  • the target style feature can be obtained by fusing the style feature of the whole image with the style feature of the object.
  • the third image has a style corresponding to the style feature of the whole image.
  • make the object in the third image have the style corresponding to the object's style feature and the style coordination corresponding to the style feature of the whole picture, so as to improve the fidelity of the style transfer image.
  • the method of fusing the full image style feature to the object style feature has better coordination between the style of the object in the third image and the style of the full image.
  • FIG. 2 shows a second flowchart of an image generation method according to an embodiment of the present disclosure. As shown in FIG. 2, the method further includes:
  • step S15 the background style feature of the background image block in the second image is extracted, where the background image block is another image block in the second image except the partial image block.
  • the background style feature of the background image block in the second image can be extracted by the feature extractor.
  • step S13 in the foregoing embodiment may include: determining the target style feature according to the full-image style feature, the object style feature, and the background style feature.
  • the target style feature can be determined according to the full-image style feature, the object style feature, and the background style feature, so that the third image has a style corresponding to the full-image style feature, and the background of the third image has a corresponding background style feature And make the object in the third image have a style corresponding to the object’s style characteristics.
  • the determining the target style feature according to the full-image style feature, the object style feature, and the background style feature includes: fusing the full-image style feature into all The background style feature; the background style feature that has been integrated with the full-image style feature is merged into the object style feature to obtain the target style feature.
  • the target style feature may be the fusion of the full image style feature of the second image with the background style feature of the second image, and then the fusion of the background style feature with the full image style feature of the second image into the second image.
  • the object style characteristics of the image, and thus the target style characteristics are obtained.
  • the target style feature may also be the target style feature obtained by fusing the full image style feature of the second image with the object style feature of the second image.
  • FIG. 3 shows a schematic diagram of style feature fusion according to an embodiment of the present disclosure.
  • the full image style feature, background style feature, and object style feature of the second image are all feature vectors.
  • the full image style feature and background style feature can be processed by residual connection to combine the full image style feature Fuse to the background style features, and perform residual connection processing on the background style features and object style features that have been integrated with the full-image style features, so as to merge the background style features that have been integrated with the full-image style features into the object style features, and get Target style characteristics.
  • the target style feature can be obtained by fusing the full image style feature to the background style feature and then the object style feature.
  • the third image has the same characteristics as the full image.
  • the style corresponding to the style feature, and make the background in the third image have a style corresponding to the background style feature and the style coordination corresponding to the style feature of the whole picture, so that the object in the third image has a style corresponding to the object style feature and is consistent with
  • the style corresponding to the style feature of the whole picture and the style corresponding to the background style feature are coordinated to improve the fidelity of the image of the style transfer.
  • the full image style feature is fused to the background style feature and then the object style feature is combined to obtain the style and background of the object in the third image.
  • the coordination between the style and the style of the whole picture is better.
  • a first image having content corresponding to the content feature of the first image and a style corresponding to the target style feature may be generated according to the content feature of the first image and the target style feature.
  • Three images For example, an image generator may be used to generate the third image according to the content feature of the first image and the target style feature.
  • the content feature of the first image may include the content feature of the full image of the first image and the object content feature of each partial image block of the first image.
  • the content feature of the full image of the first image and the target style feature may be residually connected, and/or the object content feature of each partial image block of the first image Perform residual connection with the target style feature.
  • the residual connection may be performed through methods such as Adaptive Instance Normalization (AdaIN), to obtain the content feature of the full image of the first image and the feature information of the target style feature, and/or To obtain feature information that has the object content feature of each partial image block of the first image and has the target style feature.
  • AdaIN Adaptive Instance Normalization
  • the feature information (the feature information may include: the content feature of the full image of the first image and the target style feature, and/or each of the first image The object content feature of the partial image block and the target style feature) are decoded to generate content corresponding to the content feature of the full image of the first image and an image of the style corresponding to the target style feature, and/or generate The content corresponding to the object content feature of each partial image block of the first image and the partial image block of the style corresponding to the target style feature.
  • the content corresponding to the content feature of the full image of the first image and the style image corresponding to the target style feature may be combined, and/or each partial image block of the first image
  • the content corresponding to the target content feature and the partial image block of the style corresponding to the target style feature are input to the image generator to obtain the content corresponding to the content feature of the full image of the first image and the style corresponding to the target style feature Image, and/or, obtain the content corresponding to the object content feature of each partial image block of the first image and the partial image block of the style corresponding to the target style feature.
  • the above-mentioned image and partial image block can be called the third image.
  • step S14 may include:
  • Step S141 Generate an image based on the content feature and the target style feature by the image generator, and determine the authenticity of the generated image by the image discriminator;
  • Step S142 training the image generator based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator;
  • Step S143 The image generator after training generates the third image.
  • the content feature of the generated image should be consistent with the content feature of the first image
  • the content feature of the partial image block in the generated image should be consistent with the content feature of the partial image block of the first image
  • the generated image The style should be consistent with the target style characteristics.
  • the generated image is an image generated by an image generator. There may be deviations between the content of the generated image and the first image or partial image blocks of the first image, and there may also be deviations between the style of the generated image and the target style features.
  • the network loss can be determined according to the deviation, and the image generator and feature extractor can be trained based on the network loss.
  • the generated image can be input to the image discriminator, and the authenticity of the generated image can be discriminated by the image discriminator; and the confrontation between the image generated by the image discriminator and the image generated by the image discriminator , Training image generator.
  • the first comprehensive network loss may be determined according to the difference between the generated image and the first image.
  • the cross-entropy loss may be determined according to the difference between the corresponding pixel points between the generated image and the first image, and the cross-entropy loss may be regarded as the first comprehensive network loss.
  • the second integrated network loss can be determined according to the difference between the generated partial image block and the partial image block of the first image.
  • the cross entropy loss may be determined according to the difference between the corresponding pixel points between the generated partial image block and the partial image block of the first image, and the cross entropy loss can be regarded as the second comprehensive network loss.
  • both the first integrated network loss and the second integrated network loss can be expressed by the following formula (1):
  • L r represents the loss of the first comprehensive network or the loss of the second comprehensive network
  • k represents the pixel point of the first image or the pixel point of the partial image block of the first image
  • It can also represent the content information, style information, etc. of the generated image or the generated partial image block
  • k can also represent the content information, style information, etc. of the first image or the partial image block of the first image.
  • the generated image or the generated partial image block may be input to the image discriminator for discrimination processing to obtain the discrimination result; wherein the discrimination result may include the first discrimination result corresponding to the generated image , And/or, corresponding to the second discrimination result of the generated partial image block, the first discrimination result and the second discrimination result may be used to respectively represent the authenticity of the generated image and the generated partial image block.
  • the first discrimination result and the second discrimination result may be in the form of probability.
  • the first discrimination result indicates that the probability that the generated image is a real image is 80%.
  • the training may be conducted against at least one of the first comprehensive network loss and the second comprehensive network loss, and at least one of the first discrimination result and the second discrimination result.
  • the feature extractor, the image generator, and the image discriminator that is, adjust the network parameters of the feature extractor, the image generator, and the image discriminator, until the first training condition and the second training condition reach a balanced state; wherein,
  • the first training condition is for example: the loss of the first integrated network and/or the loss of the second integrated network of the feature extractor and the image generator is minimized or less than a set first threshold;
  • the second training condition is for example: image discrimination The probability that the first discriminating result and/or the second discriminating result output by the detector is the real image is maximized or greater than the set second threshold.
  • the positions of the two images can be exchanged, and the feature extractor, image generator, and image discriminator
  • the first image is image A
  • the second image is image B
  • the first image is image B
  • the second image is image A.
  • the integrated network loss can be expressed by the following formula (2):
  • k represents the pixel of the first image and In the case of representing the pixels of the generated image, the first comprehensive network loss determined;
  • k represents the pixel of the first image and In the case of representing the pixels of the generated image, the first comprehensive network loss determined;
  • k represents the content information of the first image and In the case of representing the content information of the generated image, the determined first comprehensive network loss;
  • k represents the content information of the first image and In the case of representing the content information of the generated image, the determined first comprehensive network loss;
  • k represents the content information of the first image and In the case of representing the content information of the generated image, the determined first comprehensive network loss;
  • k represents the style information of the first image and In the case of representing the style information of the generated image, the first comprehensive network loss determined;
  • k represents the style information of the first image and In the case of representing the style information of the generated image, the first comprehensive network loss determined;
  • k represents the style information of the first image and in the case of representing the style information of the generated image, the
  • the integrated network loss determined according to the above formula (2) can be used against the training feature extractor, the image generator, and the image discriminator. Until the first training condition and the second training condition reach a balanced state, the training can be terminated, and the trained image generator and feature extractor can be obtained.
  • the third image can be generated by an image generator that has been trained.
  • the image generator can perform decoding processing such as up-sampling to determine the content features of the first image (for example, the content features of the full image of the first image or the object content features of each partial image block of the first image) and the target
  • the style feature is decoded to generate a third image, and the generated third image may have content corresponding to the content feature of the first image and a style corresponding to the target style feature.
  • the first image includes annotations (for example, category annotations, etc.) for each object in the image
  • the third image may have Annotate each object in the image.
  • the feature extractor and the image generator can be trained through two consistent images, which can improve training efficiency and improve training effects.
  • two identical images indicate that the aforementioned first image and second image are images with the same content.
  • the second image may be an image obtained by performing a scaling process on the first image, that is, the second image and the first image
  • the content is the same and the style is the same, but the resolution is inconsistent with the first image.
  • the feature extractor and the image generator can be trained through the first image and the second image, which can improve the universality of the feature extractor and the image generator for resolution, and improve the robustness of the feature extractor and the image generator.
  • the feature extractor and the image generator can be used to generate an image consistent with the first image or the partial image block of the first image to train the feature extractor and the image generator, which can improve Training efficiency, improve training effect.
  • the target style feature determined by the full image style feature and the object style feature of the second image and the content feature of the first image can be used to obtain a third image having content corresponding to the content feature and style corresponding to the target style feature ,
  • the image style is converted, not only the full image style of the image can be converted, but also the style of the partial image block where the object in the image is located can be appropriately converted, so that the full image of the image and the area where the object is located are coordinated.
  • the generated third image has content corresponding to the content feature of the first image and a style corresponding to the target style feature, and can also have the same annotation as the first image, which expands the content feature corresponding to the first image.
  • the richness of the image samples of the content especially for difficult image samples (such as images collected in a difficult weather environment, such as extreme weather conditions) or a few image samples (such as collection in a less-collected environment) Images, such as images collected at night), greatly reducing the cost of manual collection and labeling.
  • the method further includes:
  • step S16 the content feature of the source image is extracted; the content feature of the source image and the target style feature are input into the image generator after the training is completed to obtain a first target image, wherein the first target The image has content corresponding to the content feature of the source image and a style corresponding to the target style feature.
  • the content feature of the source image can be extracted by the feature extractor, and the style of the source image of any style can be converted into a specific style by the image generator.
  • the content feature of an image corresponds to the content and has the first target image with the target style feature.
  • FIG. 5 shows a flowchart 4 of an image generation method according to an embodiment of the present disclosure. As shown in FIG. 5, the method further includes:
  • step S17 the content feature of the source image is extracted; the content feature of the source image, the target style feature and random noise are input into the image generator after the training is completed to obtain a second target image, wherein:
  • the second target image has content corresponding to the content feature of the source image, and the second target image has a style specific to the target style, or the second target image has content that corresponds to the source image Features and content corresponding to the random noise, and the second target image has a style that specifically corresponds to the target style, or the second target image has content that corresponds to the content feature of the source image, and
  • the second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content feature of the source image and the random noise, and the The second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content feature of the source image, and the second target image has The style corresponding to the random noise, or the second target image has content corresponding
  • the random noise may include random content noise and/or random style features extracted from an image with random content or style, or random content noise and/or random content noise and/or random content of an image whose RGB value of each pixel is randomly generated.
  • random style features the random content noise and/or random style features of the image can be extracted by a feature extractor, and the image style corresponding to the random style feature is random.
  • the source image is an image taken at a certain place during the day, and the random noise may be a random style feature extracted from an artificially randomly generated image.
  • the image generator may generate the second target image according to at least one of the content characteristics of the source image, random noise (random noise may include random style characteristics and/or random content characteristics), and target style characteristics.
  • random noise may include random style characteristics and/or random content characteristics
  • target style characteristics may include random style characteristics and/or random content characteristics.
  • the content feature of the source image and the random content feature of the random noise can be fused to obtain the content feature corresponding to the source image and the random noise.
  • the target style feature and the random style feature of the random noise can also be fused to obtain the target style feature
  • the image generator can generate content corresponding to the source image and a second target image of the style corresponding to the target style feature, or generate content corresponding to the source image and random noise and match the target
  • the second target image of the style corresponding to the specific style or generate the content corresponding to the source image and the second target image of the style corresponding to the target style feature and random noise, or generate the content corresponding to the source image and random noise and the target
  • the second target image of the style corresponding to the style feature and random noise, or the second target image of the style corresponding to the content corresponding to the source image and the random noise is generated, or the content corresponding to the source image and the random noise and the random noise is generated
  • the second target image of the corresponding style As a result, it is realized that more than one style of images can be realized based on the neural network, and the diversity of the content and/or style of the generated images is improved.
  • a feature extractor and an image generator can perform style conversion processing on an image of any style, and the entire converted image can be coordinated with the area where the target object is located.
  • FIG. 6A-6C show schematic diagrams of the application of the image generation method according to an embodiment of the present disclosure.
  • the style of the first image is different from that of the second image.
  • the first image is an image taken at a certain place during the day.
  • the second image is an image taken at another location at night.
  • the full image of the first image and the partial image blocks containing the object in the first image can be style-transformed separately to obtain a third image.
  • the third image has the same content as the first image and is consistent with the style characteristics and object styles of the entire image.
  • the target style feature determined by the feature corresponds to the style. For example, as shown in FIG.
  • the first image is an image of a certain street taken during the day (such as the image shown in the upper left image in FIG. 6B), and it can be determined that the partial image block (such as The partial image block shown in the lower left image in FIG. 6B), and the first image is style-transformed by the target style feature determined by the full-image style feature and the object style feature to obtain the third image of the night style of the street.
  • the feature extractor can be used to extract the full image style feature of the first image, the object style feature of the partial image block of the first image, the full image style feature of the second image, and the partial image of the second image.
  • the object style feature of the style feature of the image block; further, the background style feature of the background image block in the first image and the background style feature of the background image block in the second image can also be extracted.
  • the content feature of the first image can be obtained by the feature extractor.
  • the first image can be down-sampled to obtain the feature map of the first image, and the The content feature of the feature map, the extracted content feature may include the content feature of the full image of the first image and the object content feature of the partial image block including the object in the first image.
  • the full-image style features of the second image, the background style features of the background image blocks in the second image, and the object style features of the partial image blocks in the second image including the object can be extracted by the feature extractor.
  • the second image may be down-sampled, and the above-mentioned full-image style features, background style features, and object style features can be extracted from the second image after the down-sampling process. Then, the full image style feature, the object style feature, and the background style feature are merged to determine the target style feature.
  • the full-picture style feature is fused to the background style feature; the background style feature that has been fused with the full-picture style feature is fused to the object style feature to obtain the target style feature.
  • the third image may be generated according to the content feature of the full image of the first image and/or the object content feature of the partial image block including the object in the first image, and the aforementioned target style feature.
  • the third image may be generated by a trained image generator.
  • the content feature of the full image of the first image and the target style feature may be subjected to residual connection processing to obtain the content feature of the full image of the first image and feature information having the target style feature
  • the object content feature of the partial image block including the object in the first image and the target style feature may be subjected to residual connection processing to obtain the object content feature of each partial image block of the first image and having the Feature information of the target style feature
  • further up-sampling processing can be performed on the obtained feature information to obtain an image with content corresponding to the content feature of the full image of the first image and a style corresponding to the target style feature
  • the full image style feature of the first image can also be fused to the object style feature of the first image to obtain the fusion style feature of the first image; or the full image style feature of the first image can be fused to The background style feature of the first image is then merged with the background style feature that has been fused with the full-image style feature to the object style feature of the first image to obtain the fused style feature of the first image.
  • the specific implementation manner of the fusion style feature of the first image can refer to the implementation manner of the target style feature in the foregoing embodiment, which will not be repeated here.
  • the content feature of the full image of the first image and the fusion style feature of the first image may be subjected to residual connection processing, to obtain the content feature of the full image of the first image and the fusion style feature of the first image Feature information, and/or, the object content feature of the partial image block containing the object in the first image and the fusion style feature of the first image can be subjected to residual connection processing to obtain each partial image with the first image
  • the object content feature of the block and the feature information of the fusion style feature of the first image; the obtained feature information can be further up-sampled to obtain the content corresponding to the content feature of the full image of the first image and the The image of the style corresponding to the fusion style feature of the first image, and/or obtain the content corresponding to the object content feature of each partial image block of the first image and the style corresponding to the fusion style feature of the first image Partial image block.
  • the generated image should be exactly the same as the first image, and the generated partial image block should be exactly the same as the partial image block
  • Fig. 7 shows a block diagram 1 of an image generation device according to an embodiment of the present disclosure. As shown in Fig. 7, the device includes:
  • the first extraction module 11 is used to extract content features of the first image
  • the second extraction module 12 is configured to extract the full image style features of the second image and the object style features of the partial image blocks in the second image including the object respectively, wherein the second image and the first image Different styles;
  • the determining module 13 is configured to determine a target style feature at least according to the full image style feature and the object style feature;
  • the first generating module 14 is configured to generate a third image according to the content feature and the target style feature, so that the third image has a content corresponding to the content feature and a style corresponding to the target style feature.
  • the determining module 13 is configured to merge the full-image style feature with the object style feature to obtain the target style feature.
  • Fig. 8 shows a second block diagram of an image generating device according to an embodiment of the present disclosure. As shown in Fig. 8, the device further includes:
  • the third extraction module 15 is configured to extract background style features of background image blocks in the second image, where the background image blocks are other image blocks in the second image except the partial image blocks;
  • the determining module 13 is configured to determine the target style feature according to the overall image style feature, the object style feature, and the background style feature.
  • the determining module 13 is configured to merge the full-image style features into the background style features; and merge the background style features that have been integrated with the full-image style features into the The target style feature, and the target style feature is obtained.
  • the first generating module 14 is configured to generate an image according to the content feature and the target style feature by an image generator, and determine the authenticity of the generated image by an image discriminator; Based on the confrontation between the discrimination result of the image discriminator and the image generated by the image generator, the image generator is trained; the image generator after training is completed to generate the third image.
  • the device further includes: a second generation module 16;
  • the first extraction module 11 is used to extract content features of the source image
  • the second generating module 16 is configured to input the content feature of the source image and the target style feature into the image generator after the training is completed to obtain a first target image, wherein the first target image has The content corresponds to the content feature of the source image and the style corresponds to the target style feature.
  • the first image and/or the second image includes at least one of the following objects: motor vehicles, non-motor vehicles, people, traffic signs, traffic lights, trees, animals, Buildings, obstacles.
  • the second extraction module 12 is used to intercept multiple partial image blocks in the second image when multiple objects are included in the second image, each The partial image block includes an object; the object style feature of each partial image block is extracted respectively.
  • the second extraction module 12 is configured to merge the object style features extracted from the multiple partial image blocks of the same category of objects in a situation where the multiple objects belong to multiple categories .
  • the device further includes: a third generation module 17;
  • the first extraction module 11 is used to extract content features of the source image
  • the third generating module 17 is configured to input the content feature of the source image, the target style feature, and random noise into the image generator after training to obtain a second target image, wherein: the second The target image has content corresponding to the content feature of the source image, and the second target image has a style that specifically corresponds to the target style, or the second target image has content features that correspond to the source image and The content corresponding to the random noise, and the second target image has a style corresponding to the target style, or the second target image has content corresponding to the content feature of the source image, and the The second target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content feature of the source image and the random noise, and the second The target image has a style corresponding to the target style feature and the random noise, or the second target image has content corresponding to the content feature of the source image, and the second target image has a style corresponding to the The style corresponding to random noise, or the second target image has content corresponding to the content
  • the embodiments of the present disclosure also provide image generation devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any method provided in the present disclosure.
  • image generation devices electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any method provided in the present disclosure.
  • the writing order of the steps does not mean a strict execution order but constitutes any limitation on the implementation process.
  • the specific execution order of each step should be based on its function and possibility.
  • the inner logic is determined.
  • the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
  • the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments.
  • brevity, here No longer refer to the description of the above method embodiments.
  • the embodiment of the present disclosure also proposes a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing methods of the embodiments of the present disclosure are implemented.
  • the computer-readable storage medium may be a non-volatile computer-readable storage medium.
  • An embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured as the method described in the embodiment of the present disclosure.
  • the electronic device can be provided as a terminal, a server or other forms of equipment.
  • Fig. 9 is a schematic structural diagram showing an electronic device according to an exemplary embodiment.
  • the electronic device 800 may be a terminal such as a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a personal digital assistant.
  • the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and Communication component 816.
  • the processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations.
  • the processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method.
  • the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components.
  • the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
  • the memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc.
  • the memory 804 can be implemented by any type of volatile or non-volatile storage devices or their combination, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (Read Only Memory) , ROM), magnetic memory, flash memory, magnetic disk or optical disk.
  • SRAM static random access memory
  • EEPROM Electrically erasable programmable read-only memory
  • EPROM Erasable Programmable Read-Only Memory
  • PROM Programmable Read-Only Memory
  • Read Only Memory Read Only Memory
  • the power supply component 806 provides power for various components of the electronic device 800.
  • the power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.
  • the multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user.
  • the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (Touch Panel, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.
  • the touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation.
  • the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
  • the audio component 810 is configured to output and/or input audio signals.
  • the audio component 810 includes a microphone (Microphone, MIC).
  • the microphone is configured to receive an external audio signal.
  • the received audio signal may be further stored in the memory 804 or transmitted via the communication component 816.
  • the audio component 810 further includes a speaker for outputting audio signals.
  • the I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module.
  • the peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
  • the sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation.
  • the sensor component 814 can detect the on/off status of the electronic device 800 and the relative positioning of the components.
  • the component is the display and the keypad of the electronic device 800.
  • the sensor component 814 can also detect the electronic device 800 or the electronic device 800.
  • the position of the component changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800.
  • the sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact.
  • the sensor component 814 may also include a light sensor, such as a Complementary Metal-Oxide Semiconductor (CMOS) or an image sensor or a Charge Coupled Device (CCD) image sensor for use in imaging applications.
  • CMOS Complementary Metal-Oxide Semiconductor
  • CCD Charge Coupled Device
  • the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
  • the communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices.
  • the electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof.
  • the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication.
  • the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (Infrared Data Association, IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BlueTooth, BT) technology and other technologies to realise.
  • RFID radio frequency identification
  • IrDA infrared data association
  • UWB ultra-wideband
  • Bluetooth Bluetooth
  • the electronic device 800 may be implemented by one or more application specific integrated circuits (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), Programmable logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), controller, microcontroller, microprocessor or other electronic components are implemented to implement the above methods.
  • ASIC application specific integrated circuits
  • DSP digital signal processor
  • DSPD digital signal processing device
  • PLD Programmable logic device
  • Field-Programmable Gate Array Field-Programmable Gate Array
  • controller microcontroller, microprocessor or other electronic components are implemented to implement the above methods.
  • a non-volatile computer-readable storage medium such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the foregoing method.
  • Fig. 10 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment.
  • the electronic device 1900 may be provided as a server.
  • the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs.
  • the application program stored in the memory 1932 may include one or more modules each corresponding to a set of instructions.
  • the processing component 1922 is configured to execute instructions to perform the above-described methods.
  • the electronic device 1900 may further include: a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958.
  • the electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.
  • the embodiment of the present disclosure also provides a non-volatile computer-readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the foregoing method.
  • a non-volatile computer-readable storage medium such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the foregoing method.
  • the embodiments of the present disclosure may be systems, methods and/or computer program products.
  • the computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure.
  • the computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Computer-readable storage media include: portable computer disks, hard drives, random access memory (Random Access Memory, RAM), read-only memory (Read Only Memory, ROM), erasable Type programmable read-only memory (EPROM or flash memory), static random access memory (Static Random Access Memory, SRAM), portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), digital multi-function disk ( DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards on which instructions are stored or raised structures in grooves, and any suitable combination of the above.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • EPROM or flash memory erasable Type programmable read-only memory
  • SRAM static random access memory
  • portable compact disc read-only memory Compact Disc Read-Only Memory
  • DVD digital multi-function disk
  • memory sticks floppy disks
  • mechanical encoding devices such as punch cards on which instructions are stored or raised structures in grooves, and any suitable combination of the above.
  • the computer-readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • the network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .
  • the computer program instructions used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or in one or more programming languages.
  • Source code or object code written in any combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages.
  • Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server carried out.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet connection).
  • LAN local area network
  • WAN wide area network
  • an electronic circuit such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions.
  • the computer-readable program instructions are executed to realize various aspects of the present disclosure.
  • These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner, so that the computer-readable medium storing instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more functions for implementing the specified logical function.
  • Executable instructions may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Editing Of Facsimile Originals (AREA)

Abstract

本公开实施例公开了一种图像生成方法及装置、电子设备及存储介质，所述方法包括：提取第一图像的内容特征；分别提取第二图像的全图风格特征和第二图像中包括有对象的局部图像块的对象风格特征；至少根据全图风格特征和对象风格特征，确定目标风格特征；根据内容特征和目标风格特征生成第三图像。

Description

图像生成方法及装置、电子设备及存储介质
相关申请的交叉引用
本申请基于申请号为201910352792.1、申请日为2019年4月29日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
技术领域
本公开涉及计算机技术领域，尤其涉及一种图像生成方法及装置、电子设备及存储介质。
背景技术
在相关技术中，可通过深度学习等方式实现图像风格的整体转换，例如，可将图像的中的明暗、对比度、光照、色彩、艺术特色或美工等进行转化，获得风格不同的图像。但风格的转换仅可作用于整体图像，而图像中的目标对象的风格仅可随着图像的整体风格一同转化，无法体现出目标对象的风格，造成转化后的图像风格单调，以及图像的整体与目标对象所在的区域不协调等问题。
发明内容
本公开实施例提出了一种图像生成方法及装置、电子设备及存储介质。
第一方面，本公开实施例提供了一种图像生成方法，包括：
提取第一图像的内容特征；
分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
根据本公开的实施例的图像生成方法，可利用由第二图像的全图风格特征和对象风格特征确定的目标风格特征以及第一图像的内容特征，获得具有与内容特征对应内容且与目标风格特征对应风格的第三图像，可在对图像进行风格转换时，不仅使图像的全图风格发生转换，也可使图像中的对象所在局部图像块的风格进行适当地转换，使图像的全图和对象所在的区域协调，提高风格迁移的图像的逼真性，且可提高对图像中的对象的检测精度。进一步地，生成的第三图像具有与第一图像的内容特征对应的内容且与目标风格特征对应风格，扩充了具有与第一图像的内容特征对应的内容的图像样本的丰富性，特别是对于困难图像样本或少数图像样本，极大降低人工采集和标注成本。
第二方面，本公开实施例提供了一种图像生成装置，包括：
第一提取模块，用于提取第一图像的内容特征；
第二提取模块，用于分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
确定模块，用于至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
第一生成模块，用于根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
第三方面，本公开实施例提供了一种电子设备，包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器被配置为：执行本公开实施例上述图像生成方法。
第四方面，本公开实施例提供了一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现本公开实施例上述图像生成方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开。
根据下面参考附图对示例性实施例的详细说明，本公开实施例的其它特征及方面将变得清楚。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开实施例的技术方案。
图1示出根据本公开实施例的图像生成方法的流程图一；
图2示出根据本公开实施例的图像生成方法的流程图二；
图3示出根据本公开实施例的风格特征融合示意图；
图4示出根据本公开实施例的图像生成方法的流程图三；
图5示出根据本公开实施例的图像生成方法的流程图四；
图6A-6C示出根据本公开实施例的图像生成方法的应用示意图；
图7示出根据本公开实施例的图像生成装置的框图一；
图8示出根据本公开实施例的图像生成装置的框图二；
图9示出根据本公开实施例的电子设备的结构示意图一；
图10示出根据本公开实施例的电子设备的结构示意图二。
具体实施方式
以下将参考附图详细说明本公开的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。
在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
另外，为了更好的说明本公开实施例，在下文的具体实施方式中给出了众多的具体 细节。本领域技术人员应当理解，没有某些具体细节，本公开实施例同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本公开实施例的主旨。
图1示出根据本公开实施例的图像生成方法的流程图一，如图1所示，所述方法包括：
在步骤S11中，提取第一图像的内容特征；
在步骤S12中，分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
在步骤S13中，至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
在步骤S14中，根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
根据本公开实施例的图像生成方法，可利用由第二图像的全图风格特征和对象风格特征确定的目标风格特征以及第一图像的内容特征，获得具有与内容特征对应内容且与目标风格特征对应风格的第三图像，可在对图像进行风格转换时，不仅使图像的全图风格发生转换，也可使图像中的对象所在局部图像块的风格进行适当地转换，使图像的全图和对象所在的区域协调，提高风格迁移的图像的逼真性，且可提高对图像中的对象的检测精度。进一步地，生成的第三图像具有与第一图像的内容特征对应的内容且与目标风格特征对应风格，扩充了具有与第一图像的内容特征对应的内容的图像样本的丰富性，特别是对于困难图像样本或少数图像样本，极大降低人工采集和标注成本。
在一种可能的实现方式中，所述方法可以由终端设备执行，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等，所述方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。或者，所述方法通过服务器执行。
在一种可能的实现方式中，第一图像和第二图像均可以是包括一个或多个对象的图像，所述对象可以是不同类别的对象，所述第一图像和/或所述第二图像中包括有以下至少一类对象：机动车、非机动车、人、交通标志、交通灯、树、动物、建筑物、障碍物。
在一种可能的实现方式中，第一图像和第二图像的图像风格不同，所述图像风格可包括图像的中的明暗、对比度、光照、色彩、艺术特色或美工等。在示例中，第一图像和第二图像可以是在不同的环境下拍摄的，例如，在白天、夜晚、雨中、雾中等环境下拍摄的图像，例如，第一图像为白天在某个地点拍摄的图像，第二图像为夜晚在另一个地点拍摄的图像。
在一种可能的实现方式中，在步骤S11中，可提取第一图像的内容特征。所述内容特征可用于表示第一图像中所包括的内容，作为一种示例，所述内容特征可以是第一图像中的对象的类别、形状、位置等信息中的至少一种，或者，所述内容特征还可以包括第一图像的背景等。作为另一种示例，所述内容特征可包括第一图像的全图的内容特征。在示例中，可通过特征提取器来提取第一图像的内容特征。实际应用中，所述特征提取器可以通过卷积神经网络等实现，本公开实施例对特征提取器的实现方式不做限制。所述内容特征可包括经特征提取器获取的第一图像的特征图或特征向量。
在一种可能的实现方式中，第一图像的内容特征可包括第一图像中的包括有对象的局部图像块的对象内容特征，可在第一图像中截取多个局部图像块，每个局部图像块可包括有一对象，并分别提取每个局部图像块的对象内容特征。例如，可通过特征提取器提取第一图像的每个局部图像块的对象内容特征。
在一种可能的实现方式中，在步骤S12中，可通过特征提取器来提取第二图像的全图风格特征和第二图像中包括有对象的局部图像块的对象风格特征。示例性的，所述特征提取器可以通过卷积神经网络等神经网络实现，本公开实施例对所述特征提取器的实现方式不做限制。例如，特征提取器可对第二图像和第二图像的局部图像块分别进行提取处理，分别获得全图风格特征和各对象的对象风格特征。其中，所述全图风格特征和对象风格特征可以为长度为8位的特征向量。本公开实施例对全图风格特征和对象风格特征的表示方式不做限制。
在一种可能的实现方式中，所述第二图像中包括有多个对象的情形下，提取所述第二图像中包括有对象的局部图像块的对象风格特征，包括：在所述第二图像中截取多个局部图像块，每个局部图像块包括有一对象；分别提取每个所述局部图像块的对象风格特征。
在示例中，可在第二图像中截取分别包括第二图像中的一个对象的局部图像块，并利用特征提取器分别提取每个局部图像块的对象风格特征。进一步地，在所述多个对象属于多个类别的情形下，提取所述第二图像中包括有对象的局部图像块的对象风格特征，还包括：融合相同类别对象的多个所述局部图像块提取的对象风格特征。例如，可对相同类别的多个对象的对象风格特征进行残差连接等处理，以对相同类别对象的多个对象风格特征进行融合。在示例中，可通过同样的方式，提取第一图像的全图风格特征和对象风格特征。
在示例中，若第一图像为白天在某个地点拍摄的图像，第二图像为夜晚在另一个地点拍摄的图像，第一图像的全图风格特征和对象风格特征可表达白天拍摄的图像的风格，例如，图像的明暗、对比度、光照、色彩均为白天拍摄的图像的风格，第一图像的全图风格特征表示第一图像的风格的信息，第一图像的对象风格特征表示第一图像中的对象所在区域(即，第一图像中的局部图像块)的风格的信息。第二图像的全图风格特征和对象风格特征可表达夜晚拍摄的图像的风格，第二图像的全图风格特征表示第二图像的风格的信息，第二图像的对象风格特征表示第二图像中的对象所在区域(即，第二图像中的局部图像块)的风格的信息。
在一种可能的实现方式中，在步骤S13中，至少根据第二图像的全图风格特征和对象风格特征，确定目标风格特征。所述第二图像和所述第一图像的风格不同，可根据第二图像的全图风格特征和对象风格特征确定目标风格特征，将第一图像的风格改变为与目标风格特征对应的风格且具有第一图像的内容的图像，即，第三图像，第三图像可包括风格变换后的第一图像全图(即，具有与第一图像全图的内容特征对应内容且与目标风格特征对应风格的图像)，以及风格变换后的局部图像块(即，具有与第一图像的局部图像块的对象内容特征对应内容且与目标风格特征对应风格的图像)。
在一种可能的实现方式中，步骤S13可包括：将所述全图风格特征融合到所述对象风格特征，得到所述目标风格特征。在示例中，第二图像的全图风格特征和对象风格特征均为特征向量，可对全图风格特征和对象风格特征进行残差连接等处理，以将全图风格特征融合到对象风格特征，得到目标风格特征。
通过这种方式，可通过将全图风格特征融合到对象风格特征，得到目标风格特征，在通过目标风格特征生成第三图像的过程中，使得第三图像具有与全图风格特征对应的风格，并使第三图像中的对象具有与对象风格特征对应的风格且与全图风格特征对应的风格协调，提高风格迁移的图像的逼真性。相对于将对象风格特征融合到全图风格特征，将全图风格特征融合到对象风格特征的方式获得的第三图像中的对象的风格与全图的风格之间的协调性更佳。
基于前述实施例，图2示出根据本公开实施例的图像生成方法的流程图二，如图2 所示，所述方法还包括：
在步骤S15中，提取所述第二图像中背景图像块的背景风格特征，其中，所述背景图像块为所述第二图像中除所述局部图像块之外的其他图像块。
在示例中，可通过特征提取器提取第二图像中背景图像块的背景风格特征。
则前述实施例中的步骤S13可包括：根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征。
通过这种方式，可根据全图风格特征、对象风格特征和背景风格特征，确定目标风格特征，使得第三图像具有与全图风格特征对应的风格，第三图像的背景具有与背景风格特征对应的风格，并使得第三图像中的对象具有与对象风格特征对应的风格。
在一种可能的实现方式中，所述根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征，包括：将所述全图风格特征融合到所述背景风格特征；将已经融合有所述全图风格特征的背景风格特征融合到所述对象风格特征，得到所述目标风格特征。
本实施例中，目标风格特征可以是将第二图像的全图风格特征融合到第二图像的背景风格特征，再将已经融合有第二图像的全图风格特征的背景风格特征融合到第二图像的对象风格特征，从而获得的目标风格特征。在其他实施方式中，目标风格特征还可以是将第二图像的全图风格特征融合到第二图像的对象风格特征，从而获得的目标风格特征。
示例性的，图3示出根据本公开实施例的风格特征融合示意图。如图3所示，第二图像的全图风格特征、背景风格特征和对象风格特征均为特征向量，可对全图风格特征和背景风格特征进行残差连接等处理，以将全图风格特征融合到背景风格特征，并将已经融合有全图风格特征的背景风格特征和对象风格特征进行残差连接等处理，以将已经融合有全图风格特征的背景风格特征融合到对象风格特征，得到目标风格特征。
通过这种方式，可通过将全图风格特征融合到背景风格特征再融合到对象风格特征，得到目标风格特征，在通过目标风格特征生成第三图像的过程中，使得第三图像具有与全图风格特征对应的风格，并使第三图像中的背景具有与背景风格特征对应的风格且与全图风格特征对应的风格协调，使第三图像中的对象具有与对象风格特征对应的风格且与全图风格特征对应的风格以及背景风格特征对应的风格协调，提高风格迁移的图像的逼真性。相对于将对象风格特征融合到背景风格特征再融合到全图风格特征，将全图风格特征融合到背景风格特征再融合到对象风格特征的方式获得的第三图像中的对象的风格、背景的风格以及全图的风格之间的协调性更佳。
在一种可能的实现方式中，在步骤S14中，可根据第一图像的内容特征和所述目标风格特征生成具有与第一图像的内容特征对应内容且与所述目标风格特征对应风格的第三图像。例如，可通过图像生成器根据第一图像的内容特征和所述目标风格特征生成所述第三图像。
本实施例中，第一图像的内容特征可包括第一图像的全图的内容特征以及第一图像的每个局部图像块的对象内容特征。则在一种可能的实现方式中，可将第一图像的全图的内容特征与所述目标风格特征进行残差连接，和/或，将第一图像的每个局部图像块的对象内容特征与所述目标风格特征进行残差连接。例如，可通过自适应实例标准化(Adaptive Instance Normalization，AdaIN)等方法来进行所述残差连接，获得具有第一图像的全图的内容特征且具有所述目标风格特征的特征信息，和/或，获得具有第一图像的每个局部图像块的对象内容特征且具有所述目标风格特征的特征信息。
在一种可能的实现方式中，可通过图像生成器对该特征信息(特征信息可以包括：第一图像的全图的内容特征与所述目标风格特征，和/或，第一图像的每个局部图像块的 对象内容特征与所述目标风格特征)进行解码处理，可生成具有第一图像的全图的内容特征对应的内容以及所述目标风格特征对应的风格的图像，和/或，生成具有第一图像的每个局部图像块的对象内容特征对应的内容以及所述目标风格特征对应的风格的局部图像块。
在一种可能的实现方式中，可将具有第一图像的全图的内容特征对应的内容以及所述目标风格特征对应的风格的图像，和/或，将第一图像的每个局部图像块的对象内容特征对应的内容以及所述目标风格特征对应的风格的局部图像块输入图像生成器，以获得具有第一图像的全图的内容特征对应的内容以及所述目标风格特征对应的风格的图像，和/或，获得具有第一图像的每个局部图像块的对象内容特征对应的内容以及所述目标风格特征对应的风格的局部图像块，上述图像和局部图像块均可称为第三图像。
在一种可能的实现方式中，如图4所示，步骤S14可包括：
步骤S141、经图像生成器根据所述内容特征和所述目标风格特征生成图像，经图像判别器判别所生成的图像的真实性；
步骤S142、基于所述图像判别器的判别结果和所述图像生成器生成图像之间的对抗，训练所述图像生成器；
步骤S143、经训练完成后的所述图像生成器生成所述第三图像。
本实施例中，生成的图像的内容特征应与第一图像的内容特征一致，生成的图像中的局部图像块的内容特征应与第一图像的局部图像块的内容特征一致，且生成的图像的风格应与目标风格特征一致。但生成的图像为图像生成器生成的图像，生成的图像的内容与第一图像或第一图像的局部图像块之间可能存在偏差，生成的图像的风格与目标风格特征之间也可能存在偏差；则可根据该偏差确定网络损失，基于网络损失训练图像生成器和特征提取器。
在一种可能的实现方式中，可将生成的图像输入图像判别器，经图像判别器判别所生成的图像的真实性；并基于图像判别器的判别结果和图像生成器生成图像之间的对抗，训练图像生成器。
在一种可能的实现方式中，可根据生成的图像和第一图像之间的差异确定第一综合网络损失。例如，可根据生成的图像和第一图像之间的对应的像素点之间的差异确定交叉熵损失，将所述交叉熵损失作为第一综合网络损失。
在另一种可能的实现方式中，若生成的图像为局部图像块，则可根据生成的局部图像块和第一图像的局部图像块之间的差异确定第二综合网络损失。例如，可根据生成的局部图像块和第一图像的局部图像块之间的对应的像素点之间的差异确定交叉熵损失，将所述交叉熵损失作为第二综合网络损失。
在一种可能的实现方式中，所述第一综合网络损失和第二综合网络损失均可用以下公式(1)表示：
Figure PCTCN2020076470-appb-000001
其中，L r表示第一综合网络损失或第二综合网络损失，
Figure PCTCN2020076470-appb-000002
表示生成的图像的像素点或生成的局部图像块的像素点，k表示第一图像的像素点或第一图像的局部图像块的像素点，
Figure PCTCN2020076470-appb-000003
表示
Figure PCTCN2020076470-appb-000004
与k的对应像素点之间的差的1范数。此外，
Figure PCTCN2020076470-appb-000005
还可表示生成的图像或生成的局部图像块的内容信息、风格信息等，k还可表示第一图像或第一图像的局部图像块的内容信息、风格信息等。
在一种可能的实现方式中，可将生成的图像或生成的局部图像块输入图像判别器进行判别处理，获得判别结果；其中，所述判别结果可包括对应于生成的图像的第一判别 结果，和/或，对应于生成的局部图像块的第二判别结果，第一判别结果和第二判别结果可用于分别表示生成的图像和生成的局部图像块的真实性。在示例中，所述第一判别结果和第二判别结果可以是概率的形式，例如，所述第一判别结果表示生成的图像为真实图像的概率为80％。
在一种可能的实现方式中，可根据上述第一综合网络损失和第二综合网络损失中的至少之一、以及上述第一判别结果和第二判别结果中的至少之一，对抗训练所述特征提取器、所述图像生成器以及所述图像判别器，即，调整特征提取器、图像生成器以及图像判别器的网络参数，直到第一训练条件和第二训练条件达到平衡状态；其中，所述第一训练条件例如：特征提取器和图像生成器的第一综合网络损失和/或第二综合网络损失达到最小化或小于设定第一阈值；所述第二训练条件例如：图像判别器输出的第一判别结果和/或第二判别结果为真实图像的概率最大化或大于设定第二阈值。
在一种可能的实现方式中，由于第一图像和第二图像为风格不同的两个图像，在训练过程中，可交换两个图像的位置，对特征提取器、图像生成器以及图像判别器进行训练，例如，在一次训练中，第一图像为图像A，第二图像为图像B，在另一次训练中，第一图像为图像B，第二图像为图像A，可将这两次训练的作为一个训练组，并将两次训练中的网络损失确定为特征提取器、图像生成器以及图像判别器的综合网络损失，或者将两次训练中的网络损失的平均值确定为特征提取器、图像生成器以及图像判别器的综合网络损失。
基于上述示例，在一种可能的实现方式中，所述综合网络损失可用以下公式(2)表示：
Figure PCTCN2020076470-appb-000006
其中，
Figure PCTCN2020076470-appb-000007
为所述一次训练中确定第一判别结果的对抗损失；
Figure PCTCN2020076470-appb-000008
为所述另一次训练中确定第一判别结果的对抗损失；
Figure PCTCN2020076470-appb-000009
为所述一次训练中确定第二判别结果的对抗损失；
Figure PCTCN2020076470-appb-000010
为所述另一次训练中确定第二判别结果的对抗损失；
Figure PCTCN2020076470-appb-000011
为在所述一次训练中，k表示第一图像的像素点且
Figure PCTCN2020076470-appb-000012
表示生成的图像的像素点的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000013
为在所述另一次训练中，k表示第一图像的像素点且
Figure PCTCN2020076470-appb-000014
表示生成的图像的像素点的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000015
为在所述一次训练中，k表示第一图像的内容信息且
Figure PCTCN2020076470-appb-000016
表示生成的图像的内容信息的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000017
为在所述另一次训练中，k表示第一图像的内容信息且
Figure PCTCN2020076470-appb-000018
表示生成的图像的内容信息的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000019
为在所述一次训练中，k表示第一图像的风格信息且
Figure PCTCN2020076470-appb-000020
表示生成的图像的风格信息的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000021
为在所述另一次训练中，k表示第一图像的风格信息且
Figure PCTCN2020076470-appb-000022
表示生成的图像的风格信息的情况下，确定的第一综合网络损失；
Figure PCTCN2020076470-appb-000023
为在所述一次训练中，k表示第一图像的局部图像块的像素点且
Figure PCTCN2020076470-appb-000024
表示生成的局部图像块的像素点的情况下，确定的第二综合网络损失；
Figure PCTCN2020076470-appb-000025
为在所述另一次训练中，k表示第一图像的局部图像块的像素点且
Figure PCTCN2020076470-appb-000026
表示生成的局部图像块的像素点的情况下，确定的第二综合网络损失；
Figure PCTCN2020076470-appb-000027
为在所述一次训练中，k表示第一图像的局部图 像块的内容信息且
Figure PCTCN2020076470-appb-000028
表示生成的局部图像块的内容信息的情况下，确定的第二综合网络损失；
Figure PCTCN2020076470-appb-000029
为在所述另一次训练中，k表示第一图像的局部图像块的内容信息且
Figure PCTCN2020076470-appb-000030
表示生成的局部图像块的内容信息的情况下，确定的第二综合网络损失；
Figure PCTCN2020076470-appb-000031
为在所述一次训练中，k表示第一图像的局部图像块的风格信息且
Figure PCTCN2020076470-appb-000032
表示生成的局部图像块的风格信息的情况下，确定的第二综合网络损失；
Figure PCTCN2020076470-appb-000033
为在所述另一次训练中，k表示第一图像的局部图像块的风格信息且
Figure PCTCN2020076470-appb-000034
表示生成局部图像块的风格信息的情况下，确定的第二综合网络损失。
在一种可能的实现方式中，可根据上述公式(2)确定的综合网络损失对抗训练特征提取器、图像生成器以及图像判别器。直到第一训练条件和第二训练条件达到平衡状态，即可终止训练，获得训练后的图像生成器和特征提取器。
在一种可能的实现方式中，可通过训练完成的图像生成器生成第三图像。例如，图像生成器可通过上采样等解码处理，对第一图像的内容特征(例如第一图像的全图的内容特征或第一图像的每个局部图像块的对象内容特征)以及所述目标风格特征进行解码处理，生成第三图像，生成的第三图像可具有与第一图像的内容特征对应的内容且与目标风格特征对应风格。在示例中，如果第一图像包括对图像中的各对象的标注(例如，类别标注等)，由于生成的第三图像具有与第一图像的内容特征对应的内容，因此，第三图像可具有对图像中的各对象的标注。
在一种可能的实施方式中，可通过两张一致的图像训练特征提取器和图像生成器，可提高训练效率，提升训练效果。其中，两张一致的图像表明前述第一图像和第二图像是内容一致的图像。
其中，在第一图像和第二图像是内容一致的图像的情况下，所述第二图像可以为对所述第一图像进行放缩处理获得的图像，即，第二图像与第一图像中的内容一致，且风格一致，只是分辨率与第一图像不一致。可通过第一图像以及第二图像训练特征提取器和图像生成器，可提高特征提取器和图像生成器对分辨率的普适性，以及提高特征提取器和图像生成器的鲁棒性。
根据本公开实施例的图像生成方法，一方面，可通过特征提取器和图像生成器生成与第一图像或者第一图像的局部图像块一致的图像来训练特征提取器和图像生成器，可提高训练效率，提升训练效果。另一方面，可利用由第二图像的全图风格特征和对象风格特征确定的目标风格特征以及第一图像的内容特征，获得具有与内容特征对应内容且与目标风格特征对应风格的第三图像，可在对图像进行风格转换时，不仅使图像的全图风格发生转换，也可使图像中的对象所在局部图像块的风格进行适当地转换，使图像的全图和对象所在的区域协调，提高风格迁移的图像的逼真性，且可提高对图像中的对象的检测精度。进一步地，生成的第三图像具有与第一图像的内容特征对应的内容且与目标风格特征对应风格，还可具有与第一图像相同的标注，扩充了具有与第一图像的内容特征对应的内容的图像样本的丰富性，特别是对于困难图像样本(如某种很难遇到的天气环境下采集到的图像，如极端天气条件)或少数图像样本(如某种采集较少环境下采集的图像，如夜晚采集的图像)，极大降低人工采集和标注成本。
基于前述实施例，在一种可能的实现方式中，如图4所示，所述方法还包括：
在步骤S16中，提取源图像的内容特征；将所述源图像的内容特征和所述目标风格特征输入训练完成后的所述图像生成器，得到第一目标图像，其中，所述第一目标图像具有与所述源图像的内容特征对应内容且与所述目标风格特征对应风格。
本实施例中，在特征提取器和图像生成器训练完成后，可通过特征提取器提取源图 像的内容特征，通过图像生成器将任意风格的源图像的风格转换为特定风格，输出具有与第一图像的内容特征对应内容且具有所述目标风格特征的第一目标图像。
基于前述实施例，图5示出根据本公开实施例的图像生成方法的流程图四，如图5所示，所述方法还包括：
在步骤S17中，提取源图像的内容特征；将所述源图像的内容特征、所述目标风格特征和随机噪声输入训练完成后的所述图像生成器，得到第二目标图像，其中：所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格。
在示例中，随机噪声可包括具有随机的内容或风格的图像提取到的随机内容噪声和/或随机风格特征，或者随机生成的各像素点的RGB值为随机值的图像的随机内容噪声和/或随机风格特征，可通过特征提取器提取上述图像的随机内容噪声和/或随机风格特征，所述随机风格特征对应的图像风格是随机的。例如，源图像的为白天拍摄的某地点的图像，随机噪声可以是人工随机生成的图像提取到的随机风格特征。例如，可从黑夜、傍晚、阴天、黎明、雨天、雪天等风格中的随机选择一种作为随机噪声的风格，并随机选择机动车、非机动车、人、交通标志、交通灯、树、动物、建筑物、障碍物中的至少一种作为内容以生成用于获得随机噪声的图像。
示例性的，图像生成器可根据源图像的内容特征、随机噪声(随机噪声可包括随机风格特征和/或随机内容特征)、目标风格特征中的至少一个生成第二目标图像，在示例中，可对源图像的内容特征和随机噪声的随机内容特征进行融合，获得与源图像和随机噪声对应的内容特征，还可将目标风格特征和随机噪声的随机风格特征进行融合，获得与目标风格特征和随机噪声对应的风格特征，图像生成器可根据上述特征生成与源图像对应的内容且与目标风格特征对应的风格的第二目标图像，或者生成与源图像和随机噪声对应的内容且与目标风格特对应的风格的第二目标图像，或者生成与源图像对应的内容且与目标风格特征和随机噪声对应的风格的第二目标图像，或者生成与源图像和随机噪声对应的内容且与目标风格特征和随机噪声对应的风格的第二目标图像，或者生成与源图像对应的内容且与随机噪声对应的风格的第二目标图像，或者生成与源图像和随机噪声对应的内容且与随机噪声对应的风格的第二目标图像。由此实现基于神经网络可以实现不止一种风格的图像，提高所生成图像的内容和/或风格的多样性。
采用本申请实施例的技术方案，通过特征提取器和图像生成器可对任意风格的图像进行风格转换的处理，可使转换后的图像的整体和目标对象所在的区域协调。
图6A-6C示出根据本公开实施例的图像生成方法的应用示意图，如图6A所示，第一图像与第二图像的风格不同，例如，第一图像为白天在某个地点拍摄的图像，第二图像为夜晚在另一个地点拍摄的图像。可对第一图像的全图和第一图像中包括有对象的局部图像块分别进行风格变换，获得第三图像，第三图像具有与第一图像内容相同且与根据全图风格特征和对象风格特征确定的目标风格特征对应风格。例如，如图6B所示，第一图像为白天拍摄的某个街道的图像(例如图6B中左上图中所示的图像)，可确定第 一图像中的包括有对象的局部图像块(例如图6B中左下图中所示的局部图像块)，并通过由全图风格特征和对象风格特征确定的目标风格特征对第一图像进行风格变换，获得该街道的夜晚风格的第三图像。
具体的，可如图6C所示，可通过特征提取器提取第一图像的全图风格特征、第一图像的局部图像块的对象风格特征、第二图像全图风格特征和第二图像的局部图像块的风格特征的对象风格特征；进一步地，还可提取第一图像中背景图像块的背景风格特征以及第二图像中背景图像块的背景风格特征。
具体的，参照图6C所示，一方面，可通过特征提取器获得第一图像的内容特征，在示例中，可对第一图像进行下采样处理，获得第一图像的特征图，并提取该特征图的内容特征，提取到的内容特征可包括第一图像全图的内容特征和第一图像中包括有对象的局部图像块的对象内容特征。
另一方面，可通过特征提取器分别提取第二图像的全图风格特征、第二图像中背景图像块的背景风格特征、以及所述第二图像中包括有对象的局部图像块的对象风格特征。在示例中，可对第二图像进行下采样处理，通过下采样处理后的第二图像提取出上述全图风格特征、背景风格特征以及对象风格特征。进而将所述全图风格特征、所述对象风格特征和所述背景风格特征进行融合，确定目标风格特征。
示例性的，将所述全图风格特征融合到所述背景风格特征；将已经融合有所述全图风格特征的背景风格特征融合到所述对象风格特征，得到所述目标风格特征。
进一步地，可根据第一图像全图的内容特征和/或第一图像中包括有对象的局部图像块的对象内容特征、以及上述目标风格特征生成第三图像。在一种可能的实现方式中，可通过训练后的图像生成器生成第三图像。
示例性的，可将第一图像全图的内容特征与所述目标风格特征进行残差连接处理，获得具有第一图像的全图的内容特征且具有所述目标风格特征的特征信息，和/或，可将第一图像中包括有对象的局部图像块的对象内容特征与所述目标风格特征进行残差连接处理，获得具有第一图像的每个局部图像块的对象内容特征且具有所述目标风格特征的特征信息；进一步可对获得的特征信息进行上采样处理，获得具有第一图像的全图的内容特征对应的内容且与所述目标风格特征对应的风格的图像，和/或，获得具有第一图像的每个局部图像块的对象内容特征对应的内容以及与所述目标风格特征对应的风格的局部图像块。
在一些可能的实施方式中，也可将第一图像的全图风格特征融合至第一图像的对象风格特征，得到第一图像的融合风格特征；或者将第一图像的全图风格特征融合至第一图像的背景风格特征，再将已经融合有所述全图风格特征的背景风格特征融合到第一图像的对象风格特征，得到第一图像的融合风格特征。其中，第一图像的融合风格特征的实现方式具体可参照前述实施例中的目标风格特征的实现方式，这里不再赘述。
进一步地，可将第一图像全图的内容特征与第一图像的融合风格特征进行残差连接处理，获得具有第一图像的全图的内容特征且具有所述第一图像的融合风格特征的特征信息，和/或，可将第一图像中包括有对象的局部图像块的对象内容特征与所述第一图像的融合风格特征进行残差连接处理，获得具有第一图像的每个局部图像块的对象内容特征且具有所述第一图像的融合风格特征的特征信息；进一步可对获得的特征信息进行上采样处理，获得具有第一图像的全图的内容特征对应的内容且与所述第一图像的融合风格特征对应的风格的图像，和/或，获得具有第一图像的每个局部图像块的对象内容特征对应的内容以及与所述第一图像的融合风格特征对应的风格的局部图像块。生成的图像应与第一图像完全一致，生成局部图像块应与第一图像的局部图像块完全一致。
图7示出根据本公开实施例的图像生成装置的框图一，如图7所示，所述装置包括：
第一提取模块11，用于提取第一图像的内容特征；
第二提取模块12，用于分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
确定模块13，用于至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
第一生成模块14，用于根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
在一种可能的实现方式中，所述确定模块13，用于将所述全图风格特征融合到所述对象风格特征，得到所述目标风格特征。
图8示出根据本公开实施例的图像生成装置的框图二，如图8所示，所述装置还包括：
第三提取模块15，用于提取所述第二图像中背景图像块的背景风格特征，其中，所述背景图像块为所述第二图像中除所述局部图像块之外的其他图像块；
所述确定模块13，用于根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征。
在一种可能的实现方式中，所述确定模块13，用于将所述全图风格特征融合到所述背景风格特征；将已经融合有所述全图风格特征的背景风格特征融合到所述对象风格特征，得到所述目标风格特征。
在一种可能的实现方式中，所述第一生成模块14，用于经图像生成器根据所述内容特征和所述目标风格特征生成图像，经图像判别器判别所生成的图像的真实性；基于所述图像判别器的判别结果和所述图像生成器生成图像之间的对抗，训练所述图像生成器；经训练完成后的所述图像生成器生成所述第三图像。
在一种可能的实现方式中，所述装置还包括：第二生成模块16；
所述第一提取模块11，用于提取源图像的内容特征；
所述第二生成模块16，用于将所述源图像的内容特征和所述目标风格特征输入训练完成后的所述图像生成器，得到第一目标图像，其中，所述第一目标图像具有与所述源图像的内容特征对应内容且与所述目标风格特征对应风格。
在一种可能的实现方式中，所述第一图像和/或所述第二图像中包括有以下至少一类对象：机动车、非机动车、人、交通标志、交通灯、树、动物、建筑物、障碍物。
在一种可能的实现方式中，所述第二提取模块12，用于所述第二图像中包括有多个对象的情形下，在所述第二图像中截取多个局部图像块，每个局部图像块包括有一对象；分别提取每个所述局部图像块的对象风格特征。
在一种可能的实现方式中，所述第二提取模块12，用于在所述多个对象属于多个类别的情形下，融合相同类别对象的多个所述局部图像块提取的对象风格特征。
在一种可能的实现方式中，所述装置还包括：第三生成模块17；
所述第一提取模块11，用于提取源图像的内容特征；
所述第三生成模块17，用于将所述源图像的内容特征、所述目标风格特征和随机噪声输入训练完成后的所述图像生成器，得到第二目标图像，其中：所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机 噪声对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格。
可以理解，本公开提及的上述各个方法实施例及装置实施例，在不违背原理逻辑的情况下，均可以彼此相互结合形成结合后的实施例，限于篇幅，本公开实施例不再赘述。
此外，本公开实施例还提供了图像生成装置、电子设备、计算机可读存储介质、程序，上述均可用来实现本公开提供的任一种方法，相应技术方案和描述和参见方法部分的相应记载，不再赘述。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述
本公开实施例还提出一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现本公开实施例上述方法。计算机可读存储介质可以是非易失性计算机可读存储介质。
本公开实施例还提出一种电子设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为本公开实施例上述方法。其中，电子设备可以被提供为终端、服务器或其它形态的设备。
图9是根据一示例性实施例示出的一种电子设备的结构示意图。例如，电子设备800可以是移动电话、计算机、数字广播终端、消息收发设备、游戏控制台、平板设备、医疗设备、健身设备或个人数字助理等终端。
参照图9，电子设备800可以包括以下一个或多个组件：处理组件802、存储器804、电源组件806、多媒体组件808、音频组件810、输入/输出(I/O)接口812、传感器组件814以及通信组件816。
处理组件802通常控制电子设备800的整体操作，诸如与显示、电话呼叫、数据通信、相机操作和记录操作相关联的操作。处理组件802可以包括一个或多个处理器820来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件802可以包括一个或多个模块，便于处理组件802和其他组件之间的交互。例如，处理组件802可以包括多媒体模块，以方便多媒体组件808和处理组件802之间的交互。
存储器804被配置为存储各种类型的数据以支持在电子设备800的操作。这些数据的示例包括用于在电子设备800上操作的任何应用程序或方法的指令、联系人数据、电话簿数据、消息、图片、视频等。存储器804可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(Static Random Access Memory，SRAM)，电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)，可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)，可编程只读存储器(Programmable Read-Only Memory，PROM)，只读存储器(Read Only Memory，ROM)，磁存储器，快闪存储器，磁盘或光盘。
电源组件806为电子设备800的各种组件提供电力。电源组件806可以包括电源管理系统、一个或多个电源、及其他与为电子设备800生成、管理和分配电力相关联的组件。
多媒体组件808包括在所述电子设备800和用户之间的提供一个输出接口的屏幕。 在一些实施例中，屏幕可以包括液晶显示器(Liquid Crystal Display，LCD)和触摸面板(Touch Panel，TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件808包括一个前置摄像头和/或后置摄像头。当电子设备800处于操作模式，如拍摄模式或视频模式时，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜系统或具有焦距和光学变焦能力。
音频组件810被配置为输出和/或输入音频信号。例如，音频组件810包括一个麦克风(Microphone，MIC)，当电子设备800处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器804或经由通信组件816发送。在一些实施例中，音频组件810还包括一个扬声器，用于输出音频信号。
I/O接口812为处理组件802和外围接口模块之间提供接口，上述外围接口模块可以是键盘、点击轮、按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。
传感器组件814包括一个或多个传感器，用于为电子设备800提供各个方面的状态评估。例如，传感器组件814可以检测到电子设备800的打开/关闭状态、组件的相对定位，例如所述组件为电子设备800的显示器和小键盘，传感器组件814还可以检测电子设备800或电子设备800一个组件的位置改变，用户与电子设备800接触的存在或不存在，电子设备800方位或加速/减速和电子设备800的温度变化。传感器组件814可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传感器组件814还可以包括光传感器，如金属氧化物半导体元件(Complementary Metal-Oxide Semiconductor，CMOS)或图像传感器或电荷耦合元件(Charge Coupled Device，CCD)图像传感器，用于在成像应用中使用。在一些实施例中，该传感器组件814还可以包括加速度传感器，陀螺仪传感器，磁传感器，压力传感器或温度传感器。
通信组件816被配置为便于电子设备800和其他设备之间有线或无线方式的通信。电子设备800可以接入基于通信标准的无线网络，如WiFi，2G或3G，或它们的组合。在一个示例性实施例中，通信组件816经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件816还包括近场通信(Near Field Communication，NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(Radio Frequency Identification，RFID)技术，红外数据协会(Infrared Data Association，IrDA)技术，超宽带(Ultra WideBand，UWB)技术，蓝牙(BlueTooth，BT)技术和其他技术来实现。
在示例性实施例中，电子设备800可以被一个或多个应用专用集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、数字信号处理设备(DSPD)、可编程逻辑器件(Programmable Logic Device，PLD)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器804，上述计算机程序指令可由电子设备800的处理器820执行以完成上述方法。
图10是根据一示例性实施例示出的一种电子设备结构示意图。例如，电子设备1900可以被提供为一服务器。参照图10，电子设备1900包括处理组件1922，其进一步包括 一个或多个处理器，以及由存储器1932所代表的存储器资源，用于存储可由处理组件1922的执行的指令，例如应用程序。存储器1932中存储的应用程序可以包括一个或一个以上的每一个对应于一组指令的模块。此外，处理组件1922被配置为执行指令，以执行上述方法。
电子设备1900还可以包括：电源组件1926被配置为执行电子设备1900的电源管理，有线或无线网络接口1950被配置为将电子设备1900连接到网络，和输入输出(I/O)接口1958。电子设备1900可以操作基于存储在存储器1932的操作系统，例如Windows ServerTM、Mac OS XTM、UnixTM、LinuxTM、FreeBSDTM或类似。
本公开实施例还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器1932，上述计算机程序指令可由电子设备1900的处理组件1922执行以完成上述方法。
本公开实施例可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(Static Random Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
这里参照根据本公开实施例的方法、装置(系统)和计算机程序产品的流程图和/ 或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。

Claims (22)

  1. 一种图像生成方法，包括：
    提取第一图像的内容特征；
    分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
    至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
    根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
  2. 根据权利要求1所述的方法，其中，所述根据所述全图风格特征和所述对象风格特征，确定所述目标风格特征，包括：
    将所述全图风格特征融合到所述对象风格特征，得到所述目标风格特征。
  3. 根据权利要求1所述的方法，其中，所述方法还包括：提取所述第二图像中背景图像块的背景风格特征，其中，所述背景图像块为所述第二图像中除所述局部图像块之外的其他图像块；
    所述至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征，包括：根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征。
  4. 根据权利要求3所述的方法，其中，所述根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征，包括：
    将所述全图风格特征融合到所述背景风格特征；
    将已经融合有所述全图风格特征的背景风格特征融合到所述对象风格特征，得到所述目标风格特征。
  5. 根据权利要求1-4中任一项所述的方法，其中，所述根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格，包括：
    经图像生成器根据所述内容特征和所述目标风格特征生成图像，经图像判别器判别所生成的图像的真实性；
    基于所述图像判别器的判别结果和所述图像生成器生成图像之间的对抗，训练所述图像生成器；
    经训练完成后的所述图像生成器生成所述第三图像。
  6. 根据权利要求5所述的方法，其中，所述方法还包括：
    提取源图像的内容特征；
    将所述源图像的内容特征和所述目标风格特征输入训练完成后的所述图像生成器，得到第一目标图像，其中，所述第一目标图像具有与所述源图像的内容特征对应内容且与所述目标风格特征对应风格。
  7. 根据权利要求1-6中任一项所述的方法，其中，所述第一图像和/或所述第二图像中包括有以下至少一类对象：机动车、非机动车、人、交通标志、交通灯、树、动物、建筑物、障碍物。
  8. 根据权利要求1-7中任一项所述的方法，其中，所述第二图像中包括有多个对象的情形下，提取所述第二图像中包括有对象的局部图像块的对象风格特征，包括：
    在所述第二图像中截取多个局部图像块，每个局部图像块包括有一对象；
    分别提取每个所述局部图像块的对象风格特征。
  9. 根据权利要求8中任一项所述的方法，其中，在所述多个对象属于多个类别的情形下，提取所述第二图像中包括有对象的局部图像块的对象风格特征，还包括：
    融合相同类别对象的多个所述局部图像块提取的对象风格特征。
  10. 根据权利要求5-9任一所述的方法，其中，所述方法还包括：
    提取源图像的内容特征；
    将所述源图像的内容特征、所述目标风格特征和随机噪声输入训练完成后的所述图像生成器，得到第二目标图像，其中：所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格。
  11. 一种图像生成装置，包括：
    第一提取模块，用于提取第一图像的内容特征；
    第二提取模块，用于分别提取第二图像的全图风格特征和所述第二图像中包括有对象的局部图像块的对象风格特征，其中，所述第二图像和所述第一图像的风格不同；
    确定模块，用于至少根据所述全图风格特征和所述对象风格特征，确定目标风格特征；
    第一生成模块，用于根据所述内容特征和所述目标风格特征生成第三图像，以使所述第三图像具有与所述内容特征对应内容且与所述目标风格特征对应风格。
  12. 根据权利要求11所述的装置，其中，所述确定模块，用于将所述全图风格特征融合到所述对象风格特征，得到所述目标风格特征。
  13. 根据权利要求11所述的装置，其中，所述装置还包括：
    第三提取模块，用于提取所述第二图像中背景图像块的背景风格特征，其中，所述背景图像块为所述第二图像中除所述局部图像块之外的其他图像块；
    所述确定模块，用于根据所述全图风格特征、所述对象风格特征和所述背景风格特征，确定所述目标风格特征。
  14. 根据权利要求13所述的装置，其中，所述确定模块，用于将所述全图风格特征融合到所述背景风格特征；将已经融合有所述全图风格特征的背景风格特征融合到所述对象风格特征，得到所述目标风格特征。
  15. 根据权利要求11-14中任一项所述的装置，其中，所述第一生成模块，用于经图像生成器根据所述内容特征和所述目标风格特征生成图像，经图像判别器判别所生成的图像的真实性；基于所述图像判别器的判别结果和所述图像生成器生成图像之间的对抗，训练所述图像生成器；经训练完成后的所述图像生成器生成所述第三图像。
  16. 根据权利要求15所述的装置，其中，所述装置还包括：第二生成模块；
    所述第一提取模块，用于提取源图像的内容特征；
    所述第二生成模块，用于将所述源图像的内容特征和所述目标风格特征输入训练完成后的所述图像生成器，得到第一目标图像，其中，所述第一目标图像具有与所述源图像的内容特征对应内容且与所述目标风格特征对应风格。
  17. 根据权利要求11-16中任一项所述的装置，其中，所述第一图像和/或所述第二 图像中包括有以下至少一类对象：机动车、非机动车、人、交通标志、交通灯、树、动物、建筑物、障碍物。
  18. 根据权利要求11-17中任一项所述的装置，其中，所述第二提取模块，用于所述第二图像中包括有多个对象的情形下，在所述第二图像中截取多个局部图像块，每个局部图像块包括有一对象；分别提取每个所述局部图像块的对象风格特征。
  19. 根据权利要求18所述的装置，其中，所述第二提取模块，用于在所述多个对象属于多个类别的情形下，融合相同类别对象的多个所述局部图像块提取的对象风格特征。
  20. 根据权利要求15-19中任一项所述的装置，其中，所述装置还包括：第三生成模块；
    所述第一提取模块，用于提取源图像的内容特征；
    所述第三生成模块，用于将所述源图像的内容特征、所述目标风格特征和随机噪声输入训练完成后的所述图像生成器，得到第二目标图像，其中：所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述目标风格特征和所述随机噪声对应的风格，或者，所述第二目标图像具有与所述源图像的内容特征对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格，或者所述第二目标图像具有与所述源图像的内容特征和所述随机噪声对应的内容、且所述第二目标图像具有与所述随机噪声对应的风格。
  21. 一种电子设备，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为：执行权利要求1至10中任意一项所述的方法。
  22. 一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现权利要求1至10中任意一项所述的方法。
PCT/CN2020/076470 2019-04-29 2020-02-24 图像生成方法及装置、电子设备及存储介质 WO2020220807A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2021564168A JP7394147B2 (ja) 2019-04-29 2020-02-24 画像生成方法及び装置、電子機器並びに記憶媒体
CN202080032107.8A CN113841179A (zh) 2019-04-29 2020-02-24 图像生成方法及装置、电子设备及存储介质
US17/452,388 US11900648B2 (en) 2019-04-29 2021-10-26 Image generation method, electronic device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910352792.1 2019-04-29
CN201910352792 2019-04-29

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/452,388 Continuation US11900648B2 (en) 2019-04-29 2021-10-26 Image generation method, electronic device, and storage medium

Publications (1)

Publication Number Publication Date
WO2020220807A1 true WO2020220807A1 (zh) 2020-11-05

Family

ID=73029598

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/076470 WO2020220807A1 (zh) 2019-04-29 2020-02-24 图像生成方法及装置、电子设备及存储介质

Country Status (4)

Country Link
US (1) US11900648B2 (zh)
JP (1) JP7394147B2 (zh)
CN (1) CN113841179A (zh)
WO (1) WO2020220807A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113434633A (zh) * 2021-06-28 2021-09-24 平安科技（深圳）有限公司 基于头像的社交话题推荐方法、装置、设备及存储介质
CN113469876A (zh) * 2021-07-28 2021-10-01 北京达佳互联信息技术有限公司 图像风格迁移模型训练方法、图像处理方法、装置及设备
JP7537035B2 (ja) 2021-03-31 2024-08-20 センスタイム グループ リミテッド 画像生成方法、装置、機器及び記憶媒体
JP7537034B2 (ja) 2021-03-31 2024-08-20 センスタイム グループ リミテッド 画像生成方法、装置、機器及び記憶媒体

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102706932B1 (ko) * 2020-02-10 2024-09-13 삼성전자주식회사 이미지의 생성 방법 및 그 전자 장치
WO2024090802A1 (ko) * 2022-10-25 2024-05-02 삼성전자주식회사 렌더링 스타일이 변경된 이미지를 획득하기 위한 전자 장치 및 방법
KR20240115657A (ko) * 2023-01-19 2024-07-26 삼성전자주식회사 전자 장치 및 이의 제어 방법
CN117729421B (zh) * 2023-08-17 2024-10-18 荣耀终端有限公司 图像处理方法、电子设备和计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106327539A (zh) * 2015-07-01 2017-01-11 北京大学 基于样例的图像重建方法及装置
CN108734653A (zh) * 2018-05-07 2018-11-02 商汤集团有限公司 图像风格转换方法及装置
CN108805803A (zh) * 2018-06-13 2018-11-13 衡阳师范学院 一种基于语义分割与深度卷积神经网络的肖像风格迁移方法
US20190057356A1 (en) * 2017-08-21 2019-02-21 Hirevue, Inc. Detecting disability and ensuring fairness in automated scoring of video interviews

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018132855A (ja) * 2017-02-14 2018-08-23 国立大学法人電気通信大学 画像スタイル変換装置、画像スタイル変換方法および画像スタイル変換プログラム
US10482639B2 (en) * 2017-02-21 2019-11-19 Adobe Inc. Deep high-resolution style synthesis
US10565757B2 (en) * 2017-06-09 2020-02-18 Adobe Inc. Multimodal style-transfer network for applying style features from multi-resolution style exemplars to input images
CN108537776A (zh) * 2018-03-12 2018-09-14 维沃移动通信有限公司 一种图像风格迁移模型生成方法及移动终端
CN109523460A (zh) * 2018-10-29 2019-03-26 北京达佳互联信息技术有限公司 图像风格的迁移方法、迁移装置和计算机可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106327539A (zh) * 2015-07-01 2017-01-11 北京大学 基于样例的图像重建方法及装置
US20190057356A1 (en) * 2017-08-21 2019-02-21 Hirevue, Inc. Detecting disability and ensuring fairness in automated scoring of video interviews
CN108734653A (zh) * 2018-05-07 2018-11-02 商汤集团有限公司 图像风格转换方法及装置
CN108805803A (zh) * 2018-06-13 2018-11-13 衡阳师范学院 一种基于语义分割与深度卷积神经网络的肖像风格迁移方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIN, XING ET AL.: "Enhanced Image Style Transferring Method with Primary Structure Maintained", PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE, vol. 31, no. 11, 15 November 2018 (2018-11-15), ISSN: 1003-6059 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7537035B2 (ja) 2021-03-31 2024-08-20 センスタイム グループ リミテッド 画像生成方法、装置、機器及び記憶媒体
JP7537034B2 (ja) 2021-03-31 2024-08-20 センスタイム グループ リミテッド 画像生成方法、装置、機器及び記憶媒体
CN113434633A (zh) * 2021-06-28 2021-09-24 平安科技（深圳）有限公司 基于头像的社交话题推荐方法、装置、设备及存储介质
CN113434633B (zh) * 2021-06-28 2022-09-16 平安科技（深圳）有限公司 基于头像的社交话题推荐方法、装置、设备及存储介质
CN113469876A (zh) * 2021-07-28 2021-10-01 北京达佳互联信息技术有限公司 图像风格迁移模型训练方法、图像处理方法、装置及设备
CN113469876B (zh) * 2021-07-28 2024-01-09 北京达佳互联信息技术有限公司 图像风格迁移模型训练方法、图像处理方法、装置及设备

Also Published As

Publication number Publication date
JP7394147B2 (ja) 2023-12-07
CN113841179A (zh) 2021-12-24
JP2022530518A (ja) 2022-06-29
US20220044054A1 (en) 2022-02-10
US11900648B2 (en) 2024-02-13

Similar Documents

Publication Publication Date Title
WO2020220807A1 (zh) 图像生成方法及装置、电子设备及存储介质
TWI728621B (zh) 圖像處理方法及其裝置、電子設備、電腦可讀儲存媒體和電腦程式
TWI740309B (zh) 圖像處理方法及裝置、電子設備和電腦可讀儲存介質
WO2021031506A1 (zh) 图像处理方法及装置、电子设备和存储介质
WO2021196401A1 (zh) 图像重建方法及装置、电子设备和存储介质
TWI732338B (zh) 文本序列的識別方法、電子設備和電腦可讀存儲介質
WO2021051857A1 (zh) 目标对象匹配方法及装置、电子设备和存储介质
WO2020199704A1 (zh) 文本识别
CN112001321B (zh) 网络训练、行人重识别方法及装置、电子设备和存储介质
WO2021031609A1 (zh) 活体检测方法及装置、电子设备和存储介质
TWI759647B (zh) 影像處理方法、電子設備，和電腦可讀儲存介質
WO2022011892A1 (zh) 网络训练方法及装置、目标检测方法及装置和电子设备
WO2020155711A1 (zh) 图像生成方法及装置、电子设备和存储介质
TW202105244A (zh) 圖像處理方法及裝置、電子設備和電腦可讀儲存介質
WO2021036382A1 (zh) 图像处理方法及装置、电子设备和存储介质
CN110532956B (zh) 图像处理方法及装置、电子设备和存储介质
WO2021208666A1 (zh) 字符识别方法及装置、电子设备和存储介质
JP2021512378A (ja) アンカー決定方法及び装置、電子機器並びに記憶媒体
TW202127369A (zh) 網路訓練方法、圖像生成方法、電子設備及電腦可讀儲存介質
CN110659690B (zh) 神经网络的构建方法及装置、电子设备和存储介质
WO2022141969A1 (zh) 图像分割方法及装置、电子设备、存储介质和程序
TWI738349B (zh) 圖像處理方法及圖像處理裝置、電子設備和電腦可讀儲存媒體
CN111242303A (zh) 网络训练方法及装置、图像处理方法及装置
CN111523599B (zh) 目标检测方法及装置、电子设备和存储介质
CN112597944A (zh) 关键点检测方法及装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20799227

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021564168

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.02.22)

122 Ep: pct application non-entry in european phase

Ref document number: 20799227

Country of ref document: EP

Kind code of ref document: A1