CN113869429A

CN113869429A - Model training method and image processing method

Info

Publication number: CN113869429A
Application number: CN202111155326.8A
Authority: CN
Inventors: 杨威; 叶晓青; 陈曲; 谭啸; 孙昊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-09-29
Filing date: 2021-09-29
Publication date: 2021-12-31

Abstract

The utility model provides a model training method and an image processing method, which relate to the field of artificial intelligence, in particular to the image processing and deep learning technology, and the specific implementation scheme is as follows: obtaining a training sample; inputting the training sample into an illumination style migration model, and generating an illumination style migration image of the sample image by the illumination style migration model according to the sample image and the sample reference image; obtaining style similarity of the sample reference image and the illumination style image, adjusting model parameters of the illumination style migration model based on the style similarity until the training end condition is met, and determining the illumination style migration model with the model parameters adjusted for the last time as a target illumination style migration model. Therefore, the convergent target illumination style migration model can be acquired, the illumination style of the sample reference image can be accurately and reliably migrated into the sample image, and meanwhile, a foundation is laid for image processing based on the target illumination style migration model.

Description

Model training method and image processing method

Technical Field

The present disclosure relates to the field of computer technology, and more particularly to the field of artificial intelligence, and in particular to image processing, machine learning, and deep learning techniques.

Background

In the related art, when the illumination style is migrated, an image fusion method based on a fusion retrieval technology is usually adopted for processing, but such methods often have technical problems of large calculation amount, large storage space occupation, poor accuracy and the like. Therefore, how to obtain a convergent illumination binning migration model through training and efficiently and accurately perform illumination style migration based on the illumination binning migration model has become one of important research directions.

Disclosure of Invention

The disclosure provides a model training method and an image processing method.

According to an aspect of the present disclosure, there is provided a model training method, including:

acquiring a training sample, wherein the training sample comprises a sample image and a sample reference image corresponding to the sample image, and the sample image and the sample reference image have different illumination styles;

inputting the training sample into an illumination style migration model, and generating an illumination style migration image of the sample image by the illumination style migration model according to the sample image and the sample reference image;

obtaining style similarity of the sample reference image and the illumination style image, adjusting model parameters of the illumination style migration model based on the style similarity until the training end condition is met, and determining the illumination style migration model with the model parameters adjusted for the last time as a target illumination style migration model.

According to another aspect of the present disclosure, there is provided an image processing method including:

acquiring an image to be processed;

inputting the image to be processed into a target illumination style migration model, and outputting the target illumination style migration image of the image to be processed by the target illumination style migration model, wherein the target illumination style migration model is a model trained by the model training method of the first aspect.

acquiring an image to be processed and a reference image corresponding to the image to be processed;

acquiring a first illumination style corresponding to the image to be processed and a second illumination style corresponding to the reference image;

and acquiring a target illumination style migration image of the image to be processed according to the first illumination style and the second illumination style.

According to another aspect of the present disclosure, there is provided a model training apparatus including:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a training sample, the training sample comprises a sample image and a sample reference image corresponding to the sample image, and the sample image and the sample reference image have different illumination styles;

the generation module is used for inputting the training sample into an illumination style migration model, and the illumination style migration model generates an illumination style migration image of the sample image according to the sample image and the sample reference image;

and the determining module is used for acquiring the style similarity of the sample reference image and the illumination style image, adjusting the model parameters of the illumination style migration model based on the style similarity until the training end condition is met, and determining the illumination style migration model with the model parameters adjusted for the last time as a target illumination style migration model.

According to another aspect of the present disclosure, there is provided an image processing apparatus including:

the acquisition module is used for acquiring an image to be processed;

and the output module is used for inputting the image to be processed into a target illumination style migration model and outputting the target illumination style migration image of the image to be processed by the target illumination style migration model, wherein the target illumination style migration model is a model trained by adopting the model training method of the first aspect.

the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring an image to be processed and a reference image corresponding to the image to be processed;

the second acquisition module is used for acquiring a first illumination style corresponding to the image to be processed and a second illumination style corresponding to the reference image;

and the third acquisition module is used for acquiring a target illumination style migration image of the image to be processed according to the first illumination style and the second illumination style.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the model training method of the first aspect of the disclosure or the image processing method of the second or third aspect.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the model training method of the first aspect of the present disclosure or the image processing method of the second or third aspect.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, characterized in that the computer program, when executed by a processor, implements the model training method according to the first aspect of the present disclosure or the image processing method according to the second aspect or the third aspect.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

FIG. 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

FIG. 7 is a schematic illustration of an image processing method;

FIG. 8 is a schematic illustration of another image processing method;

FIG. 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

FIG. 10 is a block diagram of a model training apparatus for implementing the model training method of an embodiment of the present disclosure;

fig. 11 is a block diagram of an image processing apparatus for implementing an image processing method of an embodiment of the present disclosure;

fig. 12 is a block diagram of an image processing apparatus for implementing another image processing method of the embodiment of the present disclosure;

FIG. 13 is a block diagram of an electronic device for implementing a model training method and an image processing method according to embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The following briefly describes the technical field to which the disclosed solution relates:

computer Technology (Computer Technology) is very extensive and can be roughly divided into several aspects of Computer system Technology, Computer machine component Technology, Computer component Technology and Computer assembly Technology. The computer technology comprises the following steps: the basic principle of the operation method, the design of an arithmetic Unit, an instruction system, the design of a Central Processing Unit (CPU), the pipeline principle and the application thereof in the CPU design, a storage system, a bus and input and output.

AI (Artificial Intelligence) is a subject for studying a computer to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of a human being, and has a technology at a hardware level and a technology at a software level. Artificial intelligence hardware techniques generally include computer vision techniques, speech recognition techniques, natural language processing techniques, and learning/deep learning thereof, big data processing techniques, knowledge-graph techniques, and the like.

Image Processing technology (Image Processing) is a technology for Processing Image information by a computer. The method mainly comprises the steps of image digitization, image enhancement and restoration, image data coding, image segmentation, image identification and the like.

Machine Learning (ML), which is a fundamental approach for making a computer have intelligence, is a core of artificial intelligence and is a fundamental approach for making a computer have intelligence, and is used for specially researching how a computer simulates or realizes human Learning behaviors to acquire new knowledge or skills and reorganizing an existing knowledge structure to continuously improve the performance of the computer.

DL (Deep Learning), a new research direction in the field of machine Learning, is introduced into machine Learning to make it closer to the original goal, artificial intelligence. Deep learning is the intrinsic law and representation hierarchy of learning sample data, and information obtained in the learning process is very helpful for interpretation of data such as characters, images and sounds. The final aim of the method is to enable the machine to have the analysis and learning capability like a human, and to recognize data such as characters, images and sounds. Deep learning is a complex machine learning algorithm, and achieves the effect in speech and image recognition far exceeding the prior related art.

A model training method and an image processing method of the embodiments of the present disclosure are described below with reference to the drawings.

Fig. 1 is a schematic diagram according to a first embodiment of the present disclosure. It should be noted that the main execution body of the model training method of this embodiment is a model training device, and the image processing device may specifically be a hardware device, or software in a hardware device, or the like. The hardware devices are, for example, terminal devices, servers, and the like.

As shown in fig. 1, the model training method proposed in this embodiment includes the following steps:

s101, obtaining a training sample, wherein the training sample comprises a sample image and a sample reference image corresponding to the sample image, and the sample image and the sample reference image have different illumination styles.

It should be noted that the sample image and the sample reference image have different illumination styles, wherein the illumination condition of the sample reference image is better than that of the sample image.

In general, the lighting conditions in the day are better than those in the night, and the lighting conditions in the sunny day are better than those in the cloudy day, so that for a group of training samples, the images shot at night can be used as sample images, and the images shot in the day can be used as corresponding sample reference images; an image taken on a cloudy day may be used as a sample image, and an image taken on a sunny day may be used as a corresponding sample reference image.

It should be noted that the model training method provided by the present disclosure may be applicable to training of multiple illumination style migration models, and further, the illumination style migration models are also applicable to multiple application scenarios. Therefore, in the present disclosure, for different application scenarios, the order of obtaining the sample image and the sample reference image in the training sample is not limited.

Taking an application scene for positioning based on the illumination style migration model as an example, an image taken at a time when the illumination condition is good may be acquired as a sample reference image, and then an image taken at a time when the illumination condition is poor may be acquired as a sample image according to the sample reference image.

For example, any image taken during the day may be acquired from the original map as the corresponding sample reference image. Further, an image photographed at night may be acquired as a sample image from the sample reference image.

S102, inputting the training sample into the illumination style migration model, and generating an illumination style migration image of the sample image according to the sample image and the sample reference image by the illumination style migration model.

In the embodiment of the disclosure, the illumination style migration model may migrate the illumination style of the sample reference image into the sample image to generate the illumination style migration image of the sample image.

For example, a training sample is input into the illumination style migration model, wherein the sample reference image is an image shot in the day, the sample image is an image shot at night, and the illumination grids of the image shot in the day are migrated into the image shot at night by the illumination style migration model to generate an illumination style migration image of the image shot at night.

S103, obtaining style similarity of the sample reference image and the illumination style image, adjusting model parameters of the illumination style migration model based on the style similarity until the training end condition is met, and determining the illumination style migration model with the model parameters adjusted for the last time as a target illumination style migration model.

The style similarity refers to a similarity degree between the illumination style of the sample reference image and the illumination style of the illumination style migration image.

In the embodiment of the disclosure, feature extraction can be performed on the sample reference image and the illumination style migration image to obtain the style features of the illumination style migration image and the style features of the sample reference image, so as to obtain the style similarity between the two style features.

The training stopping condition can be set according to the actual situation. For example, the style similarity between the sample reference image and the illumination style image can be set to reach a preset style similarity threshold; for example, the training stop condition may be set such that the number of times of adjustment of the parameter of the lighting style migration model reaches a preset number threshold.

According to the model training method disclosed by the embodiment of the disclosure, a training sample is obtained, the training sample is input into an illumination style migration model, an illumination style migration image of a sample image is generated by the illumination style migration model according to the sample image and a sample reference image, so that the style similarity of the sample reference image and the illumination style image is obtained, model parameters of the illumination style migration model are adjusted based on the style similarity until a training end condition is met, and the illumination style migration model after the model parameters are adjusted for the last time is determined as a target illumination style migration model. Therefore, the illumination style migration model is trained by acquiring the sample image and the sample reference image with different illumination styles, and the model parameters are adjusted based on the similarity, so that the converged target illumination style migration model can accurately and reliably migrate the illumination style of the sample reference image to the sample image, and meanwhile, a foundation is laid for image processing based on the target illumination style migration model.

Further, in the embodiment of the present disclosure, the lighting style migration model may be a Generative Adaptive Networks (GAN) model.

The following explains the case where the lighting style migration model is a GAN model.

It should be noted that the generative confrontation network is a deep learning model. The model includes (at least) two networks in a framework: generating (generating) network and discriminating (discriminating) network; wherein a network is generated for randomly generating observation data based on given implicit information; and the judging network is used for judging based on the observation data, such as true and false judgment, similarity judgment and the like.

In this case, in the embodiment of the present disclosure, the generation network is configured to generate the illumination style transition image of the sample image according to the sample image and the sample reference image, and the determination network is configured to perform the style similarity determination on the sample reference image and the illumination style transition image.

Further, model parameters of the GAN model may be adjusted based on the style similarity to obtain the target GAN model.

Fig. 2 is a schematic diagram according to a second embodiment of the present disclosure.

As shown in fig. 2, taking an illumination style migration model as a GAN model as an example, the model training method provided by this embodiment includes the following steps:

s201, obtaining a training sample, wherein the training sample comprises a sample image and a sample reference image corresponding to the sample image, and the sample image and the sample reference image have different illumination styles.

As a possible implementation manner, as shown in fig. 3, on the basis of the foregoing embodiment, the specific process of acquiring the training sample in the step S201 includes the following steps:

s301, acquiring a sample reference image and a shooting angle, a shooting position and a first shooting time corresponding to the sample reference image.

It should be noted that, in the present disclosure, specific manners of obtaining the sample reference image and the shooting angle, the shooting position, and the first shooting time corresponding to the sample reference image are not limited, and may be selected according to actual situations.

Alternatively, any one of the images taken in the daytime may be acquired from the original map as a corresponding sample reference image, and then the corresponding photographing angle, photographing position, and first photographing time when the sample reference image is photographed may be acquired.

S302, second shooting time is obtained, image acquisition is carried out according to the shooting angle, the shooting position and the second shooting time, and a sample image is obtained, wherein the first shooting time and the second shooting time correspond to different illumination styles respectively.

Alternatively, the second shooting time may be taken as night, and then image acquisition may be performed according to the shooting angle, the shooting position, and the second shooting time. That is, it is possible to perform photographing at a photographing angle toward the sample reference image at night according to the photographing position of the sample reference image, thereby completing photographing of the night scene image (sample image).

Further, in order to further enrich training data and improve the model training effect, image acquisition is performed according to the shooting angle, the shooting position and the second shooting time so as to obtain a sample image, and shooting parameters such as different exposure degrees and focusing conditions of the camera can be adjusted in the process of shooting at night.

As a possible implementation manner, as shown in fig. 4, on the basis of the foregoing embodiment, a specific process of performing image acquisition according to a shooting angle, a shooting position, and a second shooting time in the foregoing steps to obtain a sample image includes the following steps:

s401, acquiring at least one shooting parameter, wherein the shooting parameter at least comprises one of the following parameters: exposure level and focus condition.

The shooting parameters refer to parameters of an acquisition device such as a camera when an image is shot.

It should be noted that in the present disclosure, the shooting parameters at least include an exposure degree and a focusing condition, and optionally, may further include an aperture, a shutter, sensitivity, a focal length, and the like.

And S402, acquiring an image according to the shooting parameters, the shooting angle, the shooting position and the second shooting time aiming at each shooting parameter to obtain a sample image.

In the embodiment of the disclosure, after the shooting parameters, the shooting angle, and the second shooting time are obtained, image acquisition may be performed for each shooting parameter by using a fixed shooting angle, a fixed shooting position, and a fixed second shooting time, so as to obtain a plurality of sample images.

In this case, the plurality of acquired sample images correspond to the same sample reference image.

For example, for a sample reference image a, if the shooting parameters are a1, a2, and A3, respectively, in this case, the following three sample images can be acquired: a sample image B with shooting parameters of A1, a shooting angle of B, a shooting position of C and a second shooting time of D, a sample image C with shooting parameters of A2, a shooting angle of B, a shooting position of C and a second shooting time of D, and a sample image D with shooting parameters of A3, a shooting angle of B, a shooting position of C and a second shooting time of D.

S202, performing countermeasure training on a generation network and a judgment network in the GAN model based on the training samples, wherein the generation network is used for generating illumination style migration images of the sample images according to the sample images and the sample reference images, and the judgment network is used for judging style similarity of the illumination style images according to the sample reference images.

In the disclosed embodiments, the sample image and the sample reference image may be input to a generation network to output an illumination style migration image of the sample image.

As a possible implementation manner, as shown in fig. 5, on the basis of the foregoing embodiment, a specific process of generating an illumination style migration image of a sample image according to the sample image and a sample reference image in the foregoing steps includes the following steps:

s501, separating the illumination style of the sample reference image to obtain a reference illumination style corresponding to the sample reference image.

The step of separating the illumination style of the sample reference image refers to a process of separating the reference illumination style of the sample reference image from the sample reference image.

And S502, transferring the reference illumination style to the sample image to obtain an illumination style transferred image.

In the embodiment of the present disclosure, after the reference illumination style is obtained, the reference illumination style may be migrated to the sample image to obtain an illumination style migrated image.

In the present disclosure, in the process of transferring the reference illumination style to the sample image, at least the brightness of the reference illumination style is transferred to the sample image, so that the brightness of the sample image is close to the brightness of the sample reference image. Optionally, parameters such as color temperature and color difference of the reference illumination style can be migrated into the sample image.

In the illumination style transition image, the rest of the image is kept unchanged except for the change of the illumination style. That is, the lighting style transition image changes only the lighting style of the sample image, and does not change the shot content (e.g., buildings, people, etc.) presented by the sample image.

Further, the illumination style migration image and the sample reference image of the sample image generating the network output may be input to a discrimination network to output a style similarity between the sample reference image and the illumination style image.

As a possible implementation manner, as shown in fig. 6, on the basis of the foregoing embodiment, a specific process of acquiring a style similarity between the illumination style migration image and the sample reference image in the foregoing step includes the following steps:

s601, inputting the illumination style migration image and the sample reference image into a discrimination network, and performing feature extraction on the illumination style migration image and the sample reference image by the discrimination network to obtain a first style feature corresponding to the illumination style migration image and a second style feature corresponding to the sample reference image.

The first style characteristic refers to an illumination style characteristic corresponding to the illumination style migration image.

And the second style characteristic refers to the illumination style characteristic corresponding to the sample reference image.

After the illumination style migration image and the sample reference image are input to the discrimination network, feature extraction may be performed by using a convolution layer in the discrimination network to obtain the first style feature and the second style feature.

For example, since the perspective of the illumination style migration image and the sample reference image are very close, the similarity calculation can be performed by using the equishallow convolutional Neural networks of VGG16(Visual Geometry Group) and ResNet18(Residual Neural Network) in the discriminant Network. Alternatively, the shallow features of the illumination style migration image and the sample reference image, i.e., the first style feature and the second style feature, may be obtained by acquiring the first n layers of the shallow convolutional neural network.

S602, obtaining style similarity between the illumination style migration image and the sample reference image according to the first style characteristic and the second style characteristic.

In an embodiment of the present disclosure, after the first style feature and the second style feature are obtained, the style similarity between the migrated image and the sample reference image may be illuminated.

It should be noted that, in the present disclosure, a specific manner for obtaining the style similarity between the illumination style migration image and the sample reference image is not limited, and may be selected according to an actual situation.

Optionally, after the shallow layer feature is obtained by using the shallow convolutional neural network, a cosine similarity or other similarity calculation may be performed according to the first style feature and the second style feature.

And S203, based on the style similarity, adjusting model parameters of the GAN model to obtain the target GAN model.

In the embodiment of the disclosure, after the style similarity is obtained, the loss function of the GAN model can be obtained based on the style similarity, and then the model parameters are adjusted according to the loss function.

Optionally, in the process of adjusting the model parameters of the GAN model, the style similarity obtained each time may be compared with a style similarity threshold. Optionally, in response to that the style similarity does not reach a preset style similarity threshold, obtaining a loss function according to the style similarity, and adjusting model parameters of the illumination style migration model according to the loss function; optionally, in response to that the style similarity reaches a preset style similarity threshold, stopping training, and using the GAN model after the model parameters are adjusted this time as a target GAN model.

Wherein, the loss function can be a cross entropy loss function, etc. Further, the model parameters of the GAN model can be adjusted by training in a back-propagation manner according to the loss function.

According to the model training method disclosed by the embodiment of the disclosure, the sample images can be acquired by adopting a plurality of different shooting parameters, so that the training samples are enriched, the model training effect is improved, and the robustness of the target GAN model is improved. Meanwhile, similarity calculation can be carried out through a shallow convolution neural network in the discrimination network, so that the time for model training is shortened, and the model training efficiency is further improved. Therefore, based on the model training method provided by the disclosure, the convergent GAN model and other illumination style migration models are trained and obtained in a mode of simple training process, high efficiency and high reliability.

Fig. 7 is a schematic diagram according to a seventh embodiment of the present disclosure. It should be noted that the execution subject of the image processing method of the present embodiment is an image processing apparatus, and the image processing apparatus may specifically be a hardware device, or software in a hardware device, or the like. The hardware devices are, for example, terminal devices, servers, and the like.

As shown in fig. 7, the image processing method proposed in this embodiment includes the following steps:

and S701, acquiring an image to be processed.

The image to be processed may be any image. Alternatively, it may be any image that is not consistent with the expected lighting style.

For example, for a positioning application scene, if any image is an image with a poor lighting style, for example, an image captured at night, the image may be regarded as an image to be processed.

For another example, for an image processing application scene, if the expected lighting style is an image of a night shooting style, in such a case, an image acquired in any one of the day shooting styles may be used as the image to be processed.

S702, inputting the image to be processed into the target illumination style migration model, and outputting the target illumination style migration image of the image to be processed by the target illumination style migration model.

It should be noted that, as a possible implementation manner, when trying to perform the illumination style migration processing on the image to be processed, the processing may be performed based on the target illumination style migration model, in this case, only the image to be processed needs to be acquired, and a reference image corresponding to the image to be processed does not need to be acquired.

Wherein the target illumination style migration model is a trained convergence model.

In the embodiment of the disclosure, the image to be processed may be input into the target illumination style migration model, and the target illumination style is migrated into the image to be processed by the target illumination style migration model, so as to obtain a target illumination style migration image of the image to be processed.

According to the image processing method disclosed by the embodiment of the disclosure, the image to be processed can be input into the target illumination style migration model by acquiring the image to be processed, and the target illumination style migration image of the image to be processed is output, so that the illumination style of the image to be processed can be migrated through the converged target illumination style migration model, the illumination style migration effect is ensured, the utilization rate of the image to be processed is further improved, and the user experience is improved.

As one possible implementation, the target lighting style migration model may be a target GAN model. Alternatively, the image to be processed may be input to the target GAN model, and the target lighting style migration image of the image to be processed may be output by the generation network in the target GAN model.

According to the image processing method disclosed by the embodiment of the disclosure, the image to be processed can be input into the target GAN model by acquiring the image to be processed, and the target illumination style migration image of the image to be processed is output by the generation network in the target GAN model, so that the illumination style of the image to be processed can be migrated through the target GAN model, the illumination style migration effect is ensured, the utilization rate of the image to be processed is further improved, and the user experience is improved.

Fig. 8 is a schematic diagram according to an eighth embodiment of the present disclosure. It should be noted that the execution subject of the image processing method of the present embodiment is an image processing apparatus, and the image processing apparatus may specifically be a hardware device, or software in a hardware device, or the like. The hardware devices are, for example, terminal devices, servers, and the like.

As shown in fig. 8, the image processing method proposed in this embodiment includes the following steps:

s801, acquiring an image to be processed and a reference image corresponding to the image to be processed.

It should be noted that, as a possible implementation manner, when trying to perform the illumination style migration processing on the image to be processed, the processing may not be performed based on the target illumination style migration model, and in this case, the reference image corresponding to the image to be processed needs to be acquired while the image to be processed is acquired.

The image to be processed may be any image. Alternatively, it may be any image that is inconsistent with the expected lighting style; the reference image corresponding to the image to be processed can be any image inconsistent with the illumination style of the image to be processed. Alternatively, it may be any image consistent with the expected lighting style.

S802, a first illumination style corresponding to the image to be processed and a second illumination style corresponding to the reference image are obtained.

It should be noted that, in the present disclosure, specific ways of obtaining the first illumination style corresponding to the image to be processed and the second illumination style corresponding to the reference image are not limited, and may be selected according to actual situations.

Optionally, the image to be processed may be separated to obtain a corresponding first illumination style, and the reference image may be separated to obtain a corresponding second illumination style.

And S803, acquiring a target illumination style migration image of the image to be processed according to the first illumination style and the second illumination style.

It should be noted that, in the present disclosure, a specific manner of obtaining the target illumination style migration image of the image to be processed according to the first illumination style and the second illumination style is not limited, and may be selected according to an actual situation.

Optionally, a second illumination parameter corresponding to the second illumination style may be obtained, and a first illumination parameter corresponding to the first illumination style may be adjusted according to the second illumination parameter, so as to obtain a target illumination style migration image of the image to be processed.

According to the image processing method, the image to be processed and the reference image corresponding to the image to be processed are obtained, the first illumination style corresponding to the image to be processed and the second illumination style corresponding to the reference image are obtained, and then the target illumination style migration image of the image to be processed is obtained according to the first illumination style and the second illumination style.

Fig. 9 is a schematic diagram according to a ninth embodiment of the present disclosure.

As shown in fig. 9, the image processing method proposed in this embodiment includes the following steps:

step S801 in the above embodiment includes the following steps S901 to S902.

And S901, acquiring the image to be processed and the shooting angle and the shooting position corresponding to the image to be processed.

The shooting angle comprises shooting height, shooting direction, shooting distance and the like, and the shooting height is divided into flat shooting, bent shooting, upward shooting and the like; the shooting direction is divided into a front angle, a side angle, an oblique side angle, a back angle and the like; the shooting distance is one of the elements that determine the scene type.

Here, the photographing position refers to position information of an image, such as relative position information or absolute position information.

S902, acquiring target shooting time, and acquiring a reference image shot at the target shooting time according to a shooting angle and a shooting position corresponding to the image to be processed, wherein the shooting angle and the shooting position corresponding to the image to be processed are consistent with the shooting angle and the shooting position corresponding to the reference image.

The target shooting time refers to the shooting time corresponding to the reference image.

It should be noted that, in the present disclosure, a specific manner for acquiring the target shooting time is not limited, and may be selected according to actual situations.

Alternatively, the target photographing time may be directly acquired using a smart photographing apparatus.

In the embodiment of the disclosure, after the target shooting time is obtained, the reference image shot at the target shooting time may be obtained according to the shooting angle and the shooting position corresponding to the image to be processed, where the shooting time of the image to be processed is not consistent with that of the reference image, and the shooting angle is consistent with that of the shooting position.

For example, when the obtained target time is 13:00, the shooting angle corresponding to the image to be processed is X, and the shooting position is Y, the reference image P based on the target time being 13:00, the shooting angle being X, and the shooting position being Y may be obtained.

Step S802 in the above embodiment includes the following steps S903 to S904.

And S903, separating the illumination style of the image to be processed to obtain a first illumination style corresponding to the image to be processed.

It should be noted that, in the present disclosure, a specific manner of separating the illumination style of the image to be processed to obtain the first illumination style corresponding to the image to be processed is not limited, and may be selected according to an actual situation.

Optionally, the style migration model may be used to separate the illumination style of the image to be processed, so as to obtain a first illumination style corresponding to the image to be processed.

And S904, separating the illumination style of the reference image to obtain a second illumination style corresponding to the reference image.

It should be noted that, in the present disclosure, the specific manner of separating the illumination style of the reference image to obtain the second illumination style corresponding to the reference image is not limited, and may be selected according to the actual situation.

Optionally, the illumination style of the reference image may be separated by using a style migration model to obtain a first illumination style corresponding to the reference image.

Step S803 in the above embodiment includes the following steps S905 to S906.

S905, acquiring a first illumination parameter of the image to be processed according to the first illumination style, and acquiring a second illumination parameter of the reference image according to the second illumination style.

In the embodiment of the disclosure, a first illumination parameter of an image to be processed can be acquired, and a second illumination parameter of a reference image can be acquired according to the first illumination parameter.

S906, adjusting the first illumination parameter according to the second illumination parameter, and taking the adjusted image to be processed as the target illumination style migration image.

Optionally, the first illumination parameter may be adjusted according to the second illumination parameter, so that the first illumination parameter is consistent with or as close as possible to the second illumination parameter, so as to obtain a target illumination style migration image of the image to be processed.

According to the image processing method, the image to be processed and the reference image corresponding to the image to be processed are obtained, the first illumination style corresponding to the image to be processed and the second illumination style corresponding to the reference image are obtained, and the target illumination style migration image of the image to be processed is obtained according to the first illumination style and the second illumination style.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, and do not violate the good customs of the public order.

Corresponding to the model training methods provided in the above several embodiments, an embodiment of the present disclosure further provides a model training device, and since the model training device provided in the embodiment of the present disclosure corresponds to the model training methods provided in the above several embodiments, the implementation manner of the model training method is also applicable to the model training device provided in the embodiment, and is not described in detail in the embodiment.

FIG. 10 is a schematic diagram of a model training apparatus according to an embodiment of the present disclosure.

As shown in fig. 10, the model training apparatus 1000 includes: an obtaining module 1010, a generating module 1020, and a determining module 1030, wherein:

an obtaining module 1010, configured to obtain a training sample, where the training sample includes a sample image and a sample reference image corresponding to the sample image, and the sample image and the sample reference image have different illumination styles;

a generating module 1020, configured to input the training sample into an illumination style migration model, and generate an illumination style migration image of the sample image according to the sample image and the sample reference image by the illumination style migration model;

a determining module 1030, configured to obtain style similarities of the sample reference image and the illumination style image, adjust a model parameter of the illumination style migration model based on the style similarities until a training end condition is met, and determine the illumination style migration model after the model parameter is adjusted for the last time as a target illumination style migration model.

Wherein the generating module 1020 is further configured to:

performing countermeasure training on a generation network and a discrimination network in the GAN model based on the training sample, wherein the generation network is used for generating an illumination style migration image of the sample image according to the sample image and the sample reference image, and the discrimination network is used for performing style similarity discrimination on the illumination style image according to the sample reference image;

and adjusting the model parameters of the GAN model based on the style similarity so as to obtain a target GAN model.

Wherein, the obtaining module 1010 is further configured to:

acquiring the sample reference image and a shooting angle, a shooting position and first shooting time corresponding to the sample reference image;

and acquiring second shooting time, and acquiring an image according to the shooting angle, the shooting position and the second shooting time to acquire the sample image, wherein the first shooting time and the second shooting time correspond to different illumination styles respectively.

Wherein, the obtaining module 1010 is further configured to:

acquiring at least one shooting parameter, wherein the shooting parameter at least comprises one of the following parameters: exposure degree and focusing condition;

and for each shooting parameter, acquiring an image according to the shooting parameter, the shooting angle, the shooting position and the second shooting time to obtain the sample image.

Wherein the generating module 1020 is further configured to:

separating the illumination style of the sample reference image to obtain a reference illumination style corresponding to the sample reference image;

and transferring the reference illumination style to the sample image to obtain the illumination style transferred image.

Wherein, the obtaining module 1010 is further configured to:

inputting the illumination style migration image and the sample reference image into the discrimination network, and performing feature extraction on the illumination style migration image and the sample reference image by the discrimination network to obtain a first style feature corresponding to the illumination style migration image and a second style feature corresponding to the sample reference image;

and acquiring the style similarity between the illumination style migration image and the sample reference image according to the first style characteristic and the second style characteristic.

Wherein, the obtaining module 1010 is further configured to:

and in response to the fact that the style similarity does not reach a preset style similarity threshold value, obtaining a loss function according to the style similarity, and adjusting the model parameters of the illumination style migration model according to the loss function.

According to the model training device disclosed by the embodiment of the disclosure, the illumination style migration model is trained by acquiring the sample image and the sample reference image with different illumination styles, and the model parameters are adjusted based on the similarity, so that the converged target illumination style migration model can accurately and reliably migrate the illumination style of the sample reference image to the sample image, and meanwhile, a foundation is laid for image processing based on the target illumination style migration model.

In correspondence with the image processing methods provided by the above-mentioned several embodiments, an embodiment of the present disclosure further provides an image processing apparatus, and since the image processing apparatus provided by the embodiment of the present disclosure corresponds to the image processing methods provided by the above-mentioned several embodiments, the implementation manner of the image processing method is also applicable to the image processing apparatus provided by the embodiment, and is not described in detail in the embodiment.

Fig. 11 is a schematic configuration diagram of an image processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 11, the image processing apparatus 1100 includes: an acquisition module 1110 and an output module 1120. Wherein:

an obtaining module 1110 configured to: obtaining an image to be processed

An output module 1120 configured to: inputting the image to be processed into a target illumination style migration model, and outputting the target illumination style migration image of the image to be processed by the target illumination style migration model, wherein the target illumination style migration model is a model trained by the model training method of the first aspect.

According to the image processing device disclosed by the embodiment of the disclosure, the image to be processed is acquired, then the image to be processed is input into the target illumination style migration model, and the target illumination style migration image of the image to be processed is output, so that the illumination style of the image to be processed can be migrated through the converged target illumination style migration model, the illumination style migration effect is ensured, the utilization rate of the image to be processed is further improved, and the user experience is improved.

Fig. 12 is a schematic configuration diagram of an image processing apparatus according to another embodiment of the present disclosure.

As shown in fig. 12, the image processing apparatus 1200 includes: a first acquisition module 1210, a second acquisition module 1220, and a third acquisition module 1230. Wherein:

a first obtaining module 1210, configured to obtain an image to be processed and a reference image corresponding to the image to be processed;

a second obtaining module 1220, configured to obtain a first illumination style corresponding to the image to be processed and a second illumination style corresponding to the reference image;

a third obtaining module 1230, configured to obtain a target illumination style migration image of the to-be-processed image according to the first illumination style and the second illumination style.

Wherein, the first obtaining module 1210 is further configured to:

acquiring the image to be processed and a shooting angle and a shooting position corresponding to the image to be processed;

and acquiring target shooting time, and acquiring the reference image shot at the target shooting time according to the shooting angle and the shooting position corresponding to the image to be processed, wherein the shooting angle and the shooting position corresponding to the image to be processed are consistent with the shooting angle and the shooting position corresponding to the reference image.

The second obtaining module 1220 is further configured to:

separating the illumination style of the image to be processed to obtain the first illumination style corresponding to the image to be processed;

and separating the illumination style of the reference image to obtain the second illumination style corresponding to the reference image.

The third obtaining module 1230 is further configured to:

acquiring a first illumination parameter of the image to be processed according to the first illumination style, and acquiring a second illumination parameter of the reference image according to the second illumination style;

and adjusting the first illumination parameter according to the second illumination parameter, and taking the adjusted image to be processed as the target illumination style migration image.

According to the image processing device disclosed by the embodiment of the disclosure, the image to be processed and the reference image corresponding to the image to be processed are obtained, the first illumination style corresponding to the image to be processed and the second illumination style corresponding to the reference image are obtained, and then the target illumination style migration image of the image to be processed is obtained according to the first illumination style and the second illumination style.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

Fig. 13 illustrates a schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 13, the apparatus 1300 includes a computing unit 1301 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)1302 or a computer program loaded from a storage unit 1308 into a Random Access Memory (RAM) 1303. In the RAM 1303, various programs and data necessary for the operation of the device 1300 can also be stored. The calculation unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input/output (I/O) interface 1305 is also connected to bus 1304.

A number of components in the device 1300 connect to the I/O interface 1305, including: an input unit 1306 such as a keyboard, a mouse, or the like; an output unit 1307 such as various types of displays, speakers, and the like; storage unit 1308, such as a magnetic disk, optical disk, or the like; and a communication unit 1309 such as a network card, modem, wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Computing unit 1301 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of computing unit 1301 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 1301 performs the respective methods and processes described above, such as a model training method or an image processing method. For example, in some embodiments, the model training or image processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1308. In some embodiments, some or all of the computer program may be loaded onto and/or installed onto device 1300 via ROM 1302 and/or communications unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the model training or image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the model training method or the image processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), the internet, and blockchain networks.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements a model training method or an image processing method as described above.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A model training method, comprising:

2. The training method of claim 1, wherein the lighting style migration model is a generative confrontation network (GAN) model, the method further comprising:

3. The training method of claim 1, wherein said obtaining training samples comprises:

4. The training method of claim 3, wherein the acquiring the image according to the shooting angle, the shooting position, and the second shooting time to obtain the sample image comprises:

5. The training method of claim 2 or 3, wherein the generating of the illumination style migration image of the sample image from the sample image and the sample reference image comprises:

6. The training method of claim 5, wherein the method further comprises:

7. The training method of claim 6, wherein said adjusting model parameters of said lighting style migration model based on said style similarities comprises:

8. An image processing method comprising:

acquiring an image to be processed;

inputting the image to be processed into a target illumination style migration model, and outputting the target illumination style migration image of the image to be processed by the target illumination style migration model, wherein the target illumination style migration model is a model trained by the model training method according to any one of claims 1 to 7.

9. The processing method of claim 8, wherein the target lighting style migration model is a target-generating confrontation network (GAN) model, the method further comprising:

and inputting the image to be processed into the target GAN model, and outputting the target illumination style migration image of the image to be processed by a generation network in the target GAN model.

10. An image processing method comprising:

11. The processing method according to claim 10, wherein said acquiring an image to be processed and a reference image corresponding to the image to be processed comprises:

12. The processing method according to claim 10, wherein the obtaining a first lighting style corresponding to the image to be processed and a second lighting style corresponding to the reference image comprises:

13. The processing method according to claim 10 or 12, wherein the obtaining of the target illumination style migration image of the image to be processed according to the first illumination style and the second illumination style comprises:

14. A model training apparatus comprising:

15. The training apparatus of claim 14, wherein the generating means is further configured to:

16. The training device of claim 14, wherein the obtaining module is further configured to:

17. The training device of claim 16, wherein the obtaining module is further configured to:

18. The training apparatus of claim 15 or 16, wherein the generating means is further configured to:

19. The training device of claim 18, wherein the obtaining module is further configured to:

20. The training device of claim 19, wherein the determining module is further configured to:

21. An image processing apparatus comprising:

the acquisition module is used for acquiring an image to be processed;

an output module, configured to input the image to be processed into a target illumination style migration model, and output the target illumination style migration image of the image to be processed by the target illumination style migration model, where the target illumination style migration model is a model trained by using the model training method according to any one of claims 1 to 7.

22. The processing apparatus of claim 21, wherein the output module is further configured to:

23. An image processing apparatus comprising:

24. The processing apparatus of claim 23, wherein the first obtaining module is further configured to:

25. The processing apparatus of claim 23, wherein the second obtaining means is further configured to:

26. The processing apparatus according to claim 23 or 25, wherein the third obtaining means is further configured to:

27. An electronic device comprising a processor and a memory;

wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for implementing the method of any one of claims 1-7 or claims 8-9 or claims 10-13.

28. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7 or claims 8-9 or claims 10-13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7 or claims 8-9 or claims 10-13.