WO2021112350A1

WO2021112350A1 - Method and electronic device for modifying a candidate image using a reference image

Info

Publication number: WO2021112350A1
Application number: PCT/KR2020/006445
Authority: WO
Inventors: Ashish Chopra; Rajan Dahiya; Bapi Reddy KARRI
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2019-12-05
Filing date: 2020-05-15
Publication date: 2021-06-10

Abstract

Embodiments herein provide a method for modifying a candidate image by an electronic device (100). The method includes obtaining the candidate image and displaying, by the electronic device (100), a plurality of reference images corresponding to the candidate image. Further, the method includes detecting, by the electronic device (100), at least one reference image selected from the plurality of reference images and generating, by the electronic device (100), a plurality of versions of the at least one selected reference image, where each version of the at least one selected reference image comprises variable visual parameter. Furthermore, the method includes modifying, by the electronic device (100), at least one portion of the candidate image using at least one version of the plurality of versions of the at least one selected reference image; and storing, by the electronic device (100), the modified candidate image.

Description

METHOD AND ELECTRONIC DEVICE FOR MODIFYING A CANDIDATE IMAGE USING A REFERENCE IMAGE

The embodiments herein relate to performing actions in an electronic device. More particularly relates to a method and electronic device for modifying portion of candidate image using version of reference image.

Generally users of electronic devices share a huge amount of content on various platforms like social networking sites, blogging sites, news applications, etc. The content includes multimedia data such as text information, images and videos. The users of the electronic devices usually edit the images before sharing the same on the various platforms by using various photo editing tools such as for example filters, effects, overlays, etc to enhance the content in the images aesthetically.

The various photo editing tools generally modify a candidate image by transferring an image style of a reference image to the candidate image completely, as shown in FIG. 1. However, a user has no choice in determining the image style transformation of the candidate image as the photo editing tools is applied to the entire candidate image. Further, in case the user wants to modify only a portion of the candidate image using a portion of the reference image, then the photo editing tools does not provide such an option to the user. Therefore, the existing methods do not provide user flexibility to determine the kind of modification to the candidate image that may be desired by the user.

The above information is presented as background information only to help the reader to understand the present invention. Applicants have made no determination and make no assertion as to whether any of the above might be applicable as prior art with regard to the present application.

The principal object of the embodiments herein is to provide a method for modifying portion of candidate image using version of reference image in an electronic device.

Another object of the embodiments herein is to display a plurality of reference images corresponding to the candidate image.

Another object of the embodiments herein is to generate a plurality of versions of at least one selected reference image which comprises variable visual parameter.

Another object of the embodiments herein is to apply a first Generative Adversarial Network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image.

Another object of the embodiments herein is to apply a second Generative Adversarial Network (GAN) model to modify at least one portion of the candidate image based on at least one version of the at least one reference image.

In accordance with an aspect of the present disclose, an embodiment herein discloses a method for modifying a candidate image by an electronic device (100). The method comprises obtaining the candidate image; identifying at least one reference image selected from a plurality of reference images associated with the candidate image; generating a plurality of versions of the at least one reference image, wherein each version of the at least one reference image comprises variable visual parameter; modifying at least one portion of the candidate image using at least one version of the plurality of versions; and displaying the modified candidate image.

In an embodiment, the plurality of reference images associated with the candidate image is identified by segmenting the candidate image into a plurality of segments; extracting features from each of the plurality segments of the candidate image; determining a pattern of the candidate image based on the extracted features; and identifying the plurality of reference images based on the pattern of the candidate image.

In an embodiment, the method for generating the plurality of versions of the at least one selected reference image comprises providing the at least one reference image to a first Generative Adversarial Network (GAN) model; and generating the plurality of versions of the at least one selected reference image using the first GAN model (166).

In an embodiment, the variable visual parameter is at least one of a color, a light, an intensity and a gradient.

In an embodiment, the method for modifying the candidate image using at least one version of the plurality of versions comprises providing the at least one version of the plurality of versions of the at least one reference image to a second GAN model (168); identifying the at least one portion of the candidate image to be modified; and applying the second GAN model (168) to the at least one portion of the candidate image based on the at least one version of the plurality of versions.

In accordance with another aspect of the present disclosure, an embodiment herein discloses a method for modifying a candidate image by an electronic device (100). The method comprises obtaining at least one reference image associated with the candidate image; applying a first Generative Adversarial Network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image; applying a second Generative Adversarial Network (GAN) model to modify at least one portion of the candidate image based on at least one version of the at least one reference image; and storing the modified candidate image.

In accordance with another aspect of the present disclosure, an embodiment herein discloses an electronic device (100) for modifying a candidate image. The electronic device (100) comprises a memory (120); and at least one processor (160) coupled to the memory (120). The at least one processor (160) is configured to: obtain the candidate image; identify at least one reference image selected from a plurality of reference images associated with the candidate image; generate a plurality of versions of the at least one reference image, wherein each version of the at least one reference image comprises variable visual parameter; modify at least one portion of the candidate image using at least one version of the plurality of versions; and display the modified candidate image.

In accordance with another aspect of the present disclosure, an embodiment herein discloses an electronic device (100) for modifying a candidate image. The electronic device (100) comprises a memory (120); and at least one processor (160) coupled to the memory (120). The at least one processor (160) is configured to: obtain at least one reference image associated with the candidate image; apply a first Generative Adversarial Network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image; apply a second Generative Adversarial Network (GAN) model to modify at least one portion of the candidate image based on at least one version of the at least one reference image; and display the modified candidate image.

These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.

This invention is illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

FIG. 1 is an example illustrating a method for modifying a candidate image using a reference image in an electronic device, according to a prior art;

FIG. 2 is a block diagram of the electronic device for modifying the candidate image using the reference image, according to an embodiment as disclosed herein;

FIG. 3A is a flow chart illustrating a method for modifying the candidate image using the reference image in the electronic device, according to an embodiment as disclosed herein;

FIG. 3B is a flow chart illustrating a method for modifying the candidate image using the reference image in the electronic device, according to another embodiment as disclosed herein;

FIG. 4 is an example illustrating a method for performing an image segmentation of the candidate image by a candidate image processing engine of the electronic device, according to an embodiment as disclosed herein;

FIG. 5 is an example illustrating a plurality of reference images corresponding to the candidate image recommended by the electronic device, according to an embodiment as disclosed herein;

FIG. 6A is an example illustrating a color graph used to generate a plurality of versions of the reference image selected by the user in the electronic device, according to an embodiment as disclosed herein;

FIG. 6B is an example illustrating the plurality of versions of the reference image generated by the electronic device, according to an embodiment as disclosed herein;

FIG. 6C is an example illustrating a color bar used to generate the plurality of versions of the reference image selected by the user in the electronic device, according to an embodiment as disclosed herein;

FIG. 7 is an example illustrating the plurality of versions of the reference image generated by the electronic device based on different perspective angles, according to an embodiment as disclosed herein;

FIG. 8 is an example illustrating a color and texture extracting technique performed by the electronic device, according to an embodiment as disclosed herein;

FIG. 9A is an example illustrating an overview of the method for modifying the candidate image based on the plurality of versions of the reference image by the electronic device, according to an embodiment as disclosed herein;

FIG. 9B illustrates an example of a model architecture of a convolution stage of the encoder-decoder of a second GAN model, according to an embodiment as disclosed herein;

FIG. 9C illustrates an example of a style transfer pipeline architecture of modifying the candidate image using at least one of the versions of the reference image by the electronic device, according to an embodiment as disclosed herein;

FIG. 10A is an example illustrating a modification of at least one portion of the candidate image using at least one reference image by the electronic device, according to an embodiment as disclosed herein;

FIG. 10B is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device, according to an embodiment as disclosed herein;

FIG. 10C is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device, according to an embodiment as disclosed herein; and

FIG. 10D is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device, according to an embodiment as disclosed herein.

Various embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. In the following description, specific details such as detailed configuration and components are merely provided to assist the overall understanding of these embodiments of the present disclosure. Therefore, it should be apparent to those skilled in the art that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness. Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Herein, the term "or" as used herein, refers to a non-exclusive or, unless otherwise indicated. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

As is traditional in the field, embodiments may be described and illustrated in terms of blocks which carry out a described function or functions.　These blocks, which may be referred to herein as units, engines, manager, modules or the like, are physically implemented by analog and/or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits and the like, and may optionally be driven by firmware and/or software. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.　The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block.　Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the disclosure.　Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the disclosure.

Accordingly the embodiments herein provide a method for modifying a candidate image using an electronic device (100). The method includes receiving, by the electronic device (100), the candidate image and displaying, by the electronic device (100), a plurality of reference images corresponding to the candidate image. Further, the method includes detecting, by the electronic device (100), at least one reference image selected from the plurality of reference images and generating, by the electronic device (100), a plurality of versions of the at least one selected reference image, where each version of the at least one selected reference image comprises variable visual parameter. Furthermore, the method includes modifying, by the electronic device (100), at least one portion of the candidate image using at least one version of the plurality of versions of the at least one selected reference image; and storing, by the electronic device (100), the modified candidate image.

In the conventional methods and systems, an image style of the reference image is superimposed over the candidate image completely and the user is not provided any flexibility to select only a portion of the candidate image for modification.

Unlike to the conventional methods and systems, in the proposed method the electronic device (100) allows the user to select a portion of the candidate image for modification based on the selected reference image.

Unlike to the conventional methods and systems, in the proposed method the electronic device (100) a plurality of versions of the reference image is generated and the user is allowed to select at least one version of the plurality of versions for modifying the at least one portion of the candidate image.

Referring now to the drawings, and more particularly to FIGS. 2 through 10D, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments.

FIG. 2 is a block diagram of the electronic device (100) for modifying portion of candidate image using version of reference image, according to an embodiment as disclosed herein.

Referring to the FIG. 2, the electronic device (100) can be, for example, a mobile phone, a smart phone, Personal Digital Assistant (PDA), a tablet, a wearable device, or the like.　In an embodiment, the electronic device (100) includes a memory (120), a display (140) and a processor (160).

In an embodiment, the memory (120) can include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory (120) may, in some examples, be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted that the memory (120) is non-movable. In some examples, the memory (120) is configured to store larger amounts of information than the memory (120). In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache).

In an embodiment, the display (140) is configured to display a candidate image and allow the user to select at least one portion of the candidate image to be modified by the electronic device (100). The display (140) is also configured to display plurality of reference images corresponding to the candidate image. The display (140) is also configured to display plurality of versions of the modified candidate image on the electronic device (100).

In an embodiment, the processor (160) includes a candidate image processing engine (162), a reference image generation engine (164), a first generative adversarial network (GAN) model (166) and a second generative adversarial network (GAN) model (168).

The candidate image processing engine (162) is configured to receive the candidate image and perform image segmentation of the candidate image based on at least one of colors and light intensities. The image segmentation of the candidate image is performed to determine a number of segments of colors which are present in the candidate image and extract features from each of the segments of the candidate image. For example, the segments in the candidate image may include sky, water, green patches, buildings, etc. The candidate image processing engine (162) includes a convolution neural network (CNN) which performs the image segmentation.

The reference image generation engine (164) is configured to receive the segmented candidate image from the candidate image processing engine (162). Further, the reference image generation engine (164) is configured to determine a pattern of the candidate image based on the extracted features of each of the segments of the candidate image and generate the plurality of reference images from the electronic device (100) based on the pattern of the candidate image.

The first GAN model (166) is configured to receive the at least one reference image selected by the user from the plurality of images provided by the reference image generation engine (164). The first GAN model (166) performs the image segmentation on the at least one reference image selected by the user and determines plurality of colors in the at least one reference image. Further, a color graph is applied on the selected reference image to generate a plurality of versions of the reference image selected by the user in the electronic device (100). Each version of the at least one selected reference image comprises variable visual parameter such as for example variation in intensity of color and light variations. The first GAN model (166) is configured to generate the plurality of versions of the reference image based on the color graph by mapping the functions of the selected reference image and the color graph (as described in FIGS. 6A-6C).

Further, the first GAN model (166) is also configured to generate a plurality of versions of the reference image based on different perspective angles i.e., by rotating the perspective angle of a region of interest/object in the reference image to generate the plurality of versions of the reference image (as described in FIG. 7).

The second GAN model (168) is configured to determine the at least one portion of the candidate image selected by the user for modification. Further, the second GAN model (168) is configured to is configured to modify the at least one portion of the candidate image based on at least one version of the at least one reference image. Therefore, the proposed method provides multiple versions of the modified candidate image based on the multiple versions of the reference image.

Although the FIG. 2 shows the hardware elements of the electronic device (100) but it is to be understood that other embodiments are not limited thereon. In other embodiments, the bendable device (100) may include less or more number of elements. Further, the labels or names of the elements are used only for illustrative purpose and does not limit the scope of the invention. One or more components can be combined together to perform same or substantially similar function.

FIG. 3A is a flow chart illustrating a method for modifying the candidate image using the reference image in the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 3A, at step 310a the electronic device (100) obtains the candidate image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to obtain the candidate image. The candidate image may be obtained by capturing an image using a capturing device of the electronic device, or receiving the image from an external electronic device.

At step 320a the electronic device (100) identifies at least one reference image selected from a plurality of reference images. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to display a plurality of reference images associated with the candidate image, and identified at least one reference image selected from the plurality of reference images.

At step 330a the electronic device (100) generates a plurality of versions of the at least one reference image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to generate the plurality of versions of the at least one reference image. Each version of the at least one reference image comprises variable visual parameter, such as a color, a light, an intensity, and a gradient.

At step 340a the electronic device (100) modifies at least one portion of the candidate image using at least one version of the plurality of versions. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to modify at least one portion of the candidate image using the at least one version of the at least one reference image.

At step 350a the electronic device (100) displays the modified candidate image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to display the modified candidate image, and the memory (120) can be configured to store the modified candidate image.

The various actions, acts, blocks, steps, or the like in the method may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the invention.

FIG. 3B is a flow chart illustrating a method for modifying the candidate image using the reference image in the electronic device (100), according to another embodiment as disclosed herein.

Referring to the FIG. 3B, at step 310b the electronic device (100) obtains the at least one reference image associated with the candidate image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to obtain the at least one reference image associated with the candidate image.

At step 320b the electronic device (100) applies the first generative adversarial network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to apply the first generative adversarial network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image.

At step 330b the electronic device (100) applies the second generative adversarial network (GAN) model to modify at least one portion of the candidate image based on at least one version of the at least one reference image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to apply the second generative adversarial network (GAN) model to modify the at least one portion of the candidate image based on at least one version of the at least one reference image.

At step 340b the electronic device (100) stores the modified candidate image. For example, in the electronic device (100) as illustrated in the FIG. 2, the at least one processor (160) can be configured to display the modified candidate image, and the memory (120) can be configured to store the modified candidate image.

FIG. 4 is an example illustrating the method for performing the image segmentation of the candidate image by the candidate image processing engine (162) of the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 4, consider a candidate image which comprises a landscape of mountains, sky, clouds, grass, etc. The image segmentation is performed by the candidate image processing engine (162) using the Convolution Neural network (CNN) which receives the candidate image as the input and provides a segmented version of the candidate image as the output.

The CNN is part of the candidate image processing engine (162) which performs the image segmentation of the candidate image. The CNN performs the image segmentation based on a plurality of classes using which the CNN has been trained. Therefore, the segmented candidate image will have segments corresponding to the plurality of classes of the CNN. The plurality of classes may include for example sky, clouds, human beings, waterfall, mountain, grass-patches, rivers, buildings, etc. The segments are the individual components and classes based on which the candidate image is divided.

The proposed method for modifying the candidate image using the reference image in the electronic device (100) includes performing image segmentation of both the candidate image and the reference image. The FIG. 4 illustrates the image segmentation of the candidate image, which is applicable to the reference image as well and hence the same procedure may be considered for the same.

FIG. 5 is an example illustrating the plurality of reference images corresponding to the candidate image recommended by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 5, the electronic device (100) generates the segmented version of the candidate image as described in the FIG. 4. The electronic device (100) determines the plurality of reference images which are stored in the electronic device (100) which have a similar content as the segmented candidate image based on the clustering of the content into the plurality of classes. Further, the electronic device (100) recommends and displays the plurality of reference images based on the clustering of the content of the segmented candidate image. The user is allowed to select at least one reference image of the plurality of reference images to be used for modifying the candidate image.

FIG. 6 is an example illustrating a color graph used to generate the plurality of versions of the reference image selected by the user in the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 6A, the color graph comprising a plurality of colors in color temperature in Kelvin, is provided. The color graph is applied on the selected reference image to generate the plurality of versions of the reference image selected by the user in the electronic device (100). The plurality of version represents a plurality of intensity of color and light variations. The first GAN model (166) is used to generate the plurality of versions of the reference image based on the color graph by mapping the functions of the selected reference image and the color graph.

FIG. 6B is an example illustrating the plurality of versions of the reference image generated by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 6B in conjunction with the FIG. 6A, the first GAN model (166) is provided with the reference image selected by the user as the input. The first GAN model (166) is configured to generate the plurality of versions of the reference image based on the intensity of the color i.e., color differences determined using the color graph, as shown in FIG. 6B. Therefore, a first version of the reference image may have a highlight of yellow color, a second version of the reference image may have a highlight of blue color, a third version of the reference image may have a highlight of orange color, etc based on the various intensities of the color graph.

FIG. 6C is an example illustrating a color bar used to generate the plurality of versions of the reference image selected by the user in the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 6C, in the proposed method the first GAN model (166) comprises plurality of layers for detecting the light colors and the intensities on the basis of the segmentation of the content in the reference image. In the example, the first GAN model (166) generates five different versions of the reference image selected by the user which may be used for modifying the candidate image.

The plurality of versions of the selected reference image includes an outdoor share, an evening sun, tungsten, sunrise/sunset and candle flame. Unlike to the conventional methods and systems, in the proposed method each of the plurality of versions of the selected reference image have different hues of color intensities and light components which provides an enhanced number of options to the user for modifying the candidate image.

FIG. 7 is an example illustrating the plurality of versions of the reference image generated by the electronic device (100) based on different perspective angles, according to an embodiment as disclosed herein.

Referring to the FIG. 7, the proposed method includes the generation of the plurality of versions of the reference image based on the different perspective angles such as for example θ1, θ2, θ3, etc. The electronic evice (100) rotates the perspective angle of the region of interest in the reference image to generate the plurality of versions of the reference image. In the example, the proposed method applies eight transformations to the region/object of interest to obtain the plurality of versions of the reference image. The number of transformations required to be applied to the reference image depends on the region/object of interest, as the number of transformations can be increased up to a point where the region/object of interest starts to distort.

The plurality of versions of the reference image generated by varying the perspective angle of the region of interest may be used by the user to modify the candidate image. Further, the plurality of versions of the reference image is also stored in the electronic device (100) which may be used by the user.

Hence, based on the FIGS. 6A-6C and the FIG. 7, in the example provided a total of 5 versions of the reference image is generated based on the variations in the color intensities and 8 versions of the reference image is generated based on the variations in the perspective angles. Therefore, a total of 40 versions of the reference image are generated using the first GAN model (166) by the electronic device (100).

FIG. 8 is an example illustrating a color and texture extracting technique performed by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 8, at step 1, one of the candidate image and the reference image selected by the user is received by the electronic device (100). Further, the electronic device (100) extracts the color and texture for one of the candidate image and the reference image selected by the user. At step 2, the color and texture is extracted by using multiple levels of texture filter such as level-level filter, edge-edge filter, ripple-ripple filter, sport-spot filter etc. Further, at step 3, the filtered images are obtained and at step 4, texture-energy maps are formed for the filtered images. At step 5, the normalized maps are formed for the images and at step 6, the features if the images such as the gradient, the color and the texture of the image are obtained. Further, the code for feature and texture extraction is as below:

find color extract

int maxHist(int row[])

{

// Create an empty stack. The stack holds indexes of

// hist[] array/ The bars stored in stack are always

// in increasing order of their heights.

stack<int> result;

int top_val; // Top of stack

int max_area = 0; // Initialize max area in current

// row (or histogram)

int area = 0; // Initialize area with current top

// Run through all bars of given histogram (or row)

int i = 0;

while (i < C)

{

// If this bar is higher than the bar on top stack,

// push it to stack

if (result.empty() || row[ result.top() ] <= row[i])

result.push(i++);

else

{

// If this bar is lower than top of stack, then

// calculate area of rectangle with stack top as

// the smallest (or minimum height) bar. 'i' is

// 'right index' for the top and element before

// top in stack is 'left index

top_val = row[ result.top() ];

result.pop();

area = top_val * i;

if (!result.empty())

area = top_val * (i - result.top() - 1 );

max_area = max(area, max_area);

}

while (!result.empty())

{

top_val = row[ result.top() ];

result.pop();

area = top_val * i;

if (!result.empty())

area = top_val * (i - result.top() - 1 );

max_area = max(area, max_area);

}

return max_area;

}

// Returns area of the largest rectangle with all 1s in the image A[][]

int maxRectangle(int A[][C])

// Calculate area for first row and initialize it as

// result

int result = maxHist(A[0]);

// iterate over row to find maximum rectangular area

// considering each row as histogram

for (int i = 1; i < R; i++)

{

for (int j = 0; j < C; j++)

// if A[i][j] is 1 then add A[i -1][j]

if (A[i][j]) A[i][j] += A[i - 1][j];

// Update result if area with current row (as last row)

// of rectangle) is more

result = max(result, maxHist(A[i]));

}

return result;

}

FIG. 9A is an example illustrating an overview of the method for modifying the candidate image based on the plurality of versions of the reference image by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 9A, at step 1, the at least one portion of the candidate image to be modified is selected by the user. At step 2, the electronic device (100) recommends the plurality of reference images which are similar to the candidate image and allows the user to select at least one reference image of the plurality of reference images, which is used to modify the candidate image. Further, the electronic device (100) generates the plurality of versions of the reference image selected by the user (as shown in FIG. 9A).

Further, at step 3, the electronic device (100) generates the plurality of versions of the modified candidate image by applying each of the versions of the plurality of versions of the reference image.

The second GAN model (168) includes a Rectified Unet based generator (168a) and a RESNET based discriminator (168b). The Rectified Unet based generator (168a) is configured to produce enhanced candidate images using the at least one version of the plurality of versions of the reference image. The RESNET based discriminator (168b) is configured to identify whether the generated candidate image and the at least one version of the plurality of versions of the reference image relate to each other over an expected distribution. Both the Rectified Unet based generator (168a) and the RESNET based discriminator (168b) are trained at equal phase but in the final deployment only the Rectified Unet based generator (168a) is used.

The Rectified Unet based generator (168a) is trained on both adversarial losses i.e., L2_loss (to account for the difference between the reference image and candidate image) and L1_loss (modified version) to account for noise produced in the candidate image. The images generated using the Rectified Unet based generator (168a) are chosen based on the L2 distance from the reference image selected by the user.

The loss L2 is calculated by selecting an average of l2 distance between the top images generated using the Rectified Unet based generator (168a) and the reference image selected by the user.

The RESNET based discriminator (168b) is trained on cross entropy loss for performing the classification. The RESNET model helps to approximate complex functions by stacking a series of residual blocks.

In the proposed method, in order to check whether a given image pair consisting of the reference image selected by the user and the modified candidate image is real or fake can be considered as a classification problem. Hence, for the classification problem the real image pair with class label - 1 and the modified image pair with class label - 0 is considered.

Further, the classification issues are addressed using the RESNET model which provides better classification accuracy and adaptability with variable input size. The Noise reduction pipeline is based on conditional generative adversarial networks.

Further, for other error terms the image is selected with the least l2 distance as reference:

error_generator= error_{discriminator} + loss_L1 + loss_L2

FIG. 9B illustrates an example of the model architecture of the convolution stage of the encoder-decoder of the second GAN model (168), according to an embodiment as disclosed herein.

Referring to the FIG. 9B, at convolution stage multiple layers are added for detecting light colors and intensities on the basis of objects at the encoder, as shown in FIG. 9B. The decoder stage includes up-sampling the encoded image and concatenating the encoded image with the low-level features of the input image.

FIG. 9C illustrates an example of a style transfer pipeline architecture of modifying the candidate image using at least one of the versions of the reference image by the electronic device (100), according to an embodiment as disclosed herein.

The second GAN model (168) includes the encoder-decoder architecture that utilizes an encoder network E to map an input candidate image x onto a latent representation z = E(x). A generative decoder G then plays the role of a painter and generates the modified output image y = G(z) from the sketchy content representation z.

To train E and G, a a standard adversarial discriminator　D　is used to distinguish the stylized output　G(E(xi))　from real examples　yj ∈ Y.　A single image　y0　is given with a set　Y　of at least one reference image　yj ∈ Y.

The transformed image loss is defined as:

where C × H × W is the size of the image x and for training T is initialized with uniform weights,

where λ controls the relative importance of adversarial loss.

FIG. 10A is an example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 10A, consider the candidate image comprising a landscape with green grass along a road. At step 1, the user selects a portion of the candidate image comprising the green grass as the region of interest to be modified by the electronic device (100).

At step 2, the electronic device (100) performs the image segmentation of the candidate image and automatically determines plurality of reference images which are related to the candidate image. Further, the electronic device (100) displays the plurality of reference images on the screen of the electronic device (100) and allows the user to select the at least one reference image from the plurality of reference images to be used to modify the region of interest in the candidate image.

At step 3, the user selects a reference image from the plurality of reference images, where the reference image comprises similar landscape as the candidate image which includes green grass with white flowers along a road. Since, at step 1, the user had selected the portion of the candidate image comprising the green grass as the region of interest to be modified, the electronic device (100) automatically modifies the green grass in the candidate image with the effects of the green grass with white flowers using the plurality of versions of the reference image and presents to the user, as shown in step 4. Further, at step 5, the user may select one version of the modified image to be used for example to publish in a social networking site, etc. Further, all the versions of the modified candidate image will be available to the user in the electronic device (100).

FIG. 10B is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 10B, consider the candidate image comprising a landscape with cloudy sky, green grass and a tree. At step 1, the user selects a portion of the candidate image comprising the cloudy sky as the region of interest to be modified by the electronic device (100).

At step 3, the user selects a reference image from the plurality of reference images, where the reference image comprises similar landscape as the candidate image which includes a scene of a road along with a sunny sky with clouds. Since, at step 1, the user had selected the portion of the candidate image comprising the cloudy sky as the region of interest to be modified, the electronic device (100) automatically modifies the cloudy sky of the candidate image with the effects of the sunny sky with clouds using the plurality of versions of the reference image and presents to the user, as shown in step 4. Further, at step 5, the user may select one version of the modified candidate image to be used. Further, all the versions of the modified candidate image will be available to the user in the electronic device (100).

FIG. 10C is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 10C, consider the candidate image comprising a snow capped mountains with bright sunny sky. At step 1, the user selects a portion of the candidate image comprising the bright sunny sky as the region of interest to be modified by the electronic device (100).

At step 3, the user selects a reference image from the plurality of reference images, where the reference image comprises a canyon like structure with an evening sky. Since, at step 1, the user had selected the portion of the candidate image comprising the bright sunny sky as the region of interest to be modified, the electronic device (100) automatically modifies the bright sunny sky of the candidate image using the plurality of versions of the evening sky of the reference image and presents to the user, as shown in step 4. Further, at step 5, the user may select one version of the modified candidate image to be used. Further, all the versions of the modified candidate image will be available to the user in the electronic device (100).

FIG. 10D is another example illustrating the modification of at least one portion of the candidate image using at least one reference image by the electronic device (100), according to an embodiment as disclosed herein.

Referring to the FIG. 10D, consider the candidate image comprising a lady standing in front of tall and lush green grass. At step 1, the user selects a portion of the candidate image comprising the tall and lush green grass as the region of interest to be modified by the electronic device (100).

At step 3, the user selects a reference image from the plurality of reference images, where the reference image comprises similar landscape as the candidate image which includes green grass along the mountains. Since, at step 1, the user had selected the portion of the candidate image comprising the tall and lush green grass as the region of interest to be modified, the electronic device (100) automatically modifies the selected the portion of the candidate image using the plurality of versions of the reference image and presents to the user, as shown in step 4. Further, at step 5, the user may select one version of the modified candidate image to be for example shared over a messaging platform, etc. Further, all the versions of the modified candidate image will be available to the user in the electronic device (100).

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.

Claims

A method for modifying a candidate image by an electronic device, the method comprising:

obtaining the candidate image;

identifying at least one reference image selected from a plurality of reference images associated with the candidate image;

generating a plurality of versions of the at least one reference image, wherein each version of the at least one reference image comprises variable visual parameter;

modifying at least one portion of the candidate image using at least one version of the plurality of versions; and

displaying the modified candidate image.
The method of claim 1, wherein the plurality of reference images associated with the candidate image is identified by:

segmenting the candidate image into a plurality of segments;

extracting features from each of the plurality of segments of the candidate image;

determining a pattern of the candidate image based on the extracted features; and

identifying the plurality of reference images based on the pattern of the candidate image.
The method of claim 2, wherein generating the plurality of versions of the at least one reference image comprises:

providing the at least one reference image to a first Generative Adversarial Network (GAN) model; and

generating the plurality of versions of the at least one reference image using the first GAN model.
The method of claim 1, wherein the variable visual parameter is at least one of a color, a light, an intensity and a gradient.
The method of claim 1, wherein modifying the candidate image using at least one version of the plurality of versions comprising:

providing the at least one version of the plurality of versions of the at least one reference image to a second GAN model;

identifying the at least one portion of the candidate image to be modified; and

applying the second GAN model to the at least one portion of the candidate image based on the at least one version of the plurality of versions.
A method for modifying a candidate image by an electronic device, the method comprising:

obtaining at least one reference image associated with the candidate image;

applying a first Generative Adversarial Network (GAN) model on the at least one reference image to generate a plurality of versions of the at least one reference image;

applying a second Generative Adversarial Network (GAN) model to modify at least one portion of the candidate image based on at least one version of the at least one reference image; and

displaying the modified candidate image.
The method of claim 6, wherein obtaining the at least one reference image comprising:

identifying at least one portion of the candidate image to be modified;

displaying a plurality of reference images associated with the candidate image; and

determining the at least one reference image among the plurality of reference images, based on user selection.
The method of claim 6, wherein applying the first GAN model on the at least one reference image to generate the plurality of versions of the reference image comprising:

providing the at least one reference image to the first GAN model;

identifying at least one visual parameter associated with the at least one reference image; and

generating the plurality of versions of the at least one reference image using the first GAN model based on the at least one visual parameter.
The method of claim 8, wherein the at least one visual parameter is at least one of a color, a light, an intensity and a gradient.
The method of claim 6, wherein applying the second GAN model to modify at least one portion of the candidate image based on at least one version of the at least one reference image comprising:

determining the at least one portion of the candidate image to be modified;

applying the at least one version of the at least one reference image to the at least one portion of the candidate image using the second GAN model; and

modifying the at least one portion of the candidate image based on at the least one version of the at least one reference image.
An electronic device for modifying a candidate image, the electronic device comprising:

a memory; and

at least one processor coupled to the memory and configured to:

obtain the candidate image;

identify at least one reference image selected from a plurality of reference images associated with the candidate image;

generate a plurality of versions of the at least one reference image, wherein each version of the at least one reference image comprises variable visual parameter;

modify at least one portion of the candidate image using at least one version of the plurality of versions; and

display the modified candidate image.
The electronic device of claim 11, wherein the at least one processor is further configured to:

segment the candidate image into a plurality of segments;

extract features from each of the plurality segments of the candidate image;

determine a pattern of the candidate image based on the extracted features; and

identify the plurality of reference images based on the pattern of the candidate image.
The electronic device of claim 11, wherein the at least one processor is further configured to:

provide the at least one reference image to a first Generative Adversarial Network (GAN) model; and

generate the plurality of versions of the at least one reference image using the first GAN model.
The electronic device of claim 11, wherein the variable visual parameter is at least one of a color, a light, an intensity and a gradient.
The electronic device of claim 11, wherein the at least one processor is further configured to:

provide the at least one version of the plurality of versions of the at least one reference image to a second GAN model;

identify the at least one portion of the candidate image to be modified; and

apply the second GAN model to the at least one portion of the candidate image based on the at least one version of the plurality of versions.