WO2024117433A1

WO2024117433A1 - Method and electronic device for performing color correction

Info

Publication number: WO2024117433A1
Application number: PCT/KR2023/009947
Authority: WO
Inventors: Sandeep Singh SPALL; Choice CHOUDHARY
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2022-11-30
Filing date: 2023-07-12
Publication date: 2024-06-06

Abstract

In an embodiment, a method for performing color correction by an electronic device is provided. The method may include obtaining first image set comprising a plurality of images including at least a part of a first object from a first camera sensor. The method may include obtaining second image set comprising a plurality of images including at least a part of a second object from a second camera sensor, placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained. The method may include performing color correction based on the first image set and the second image set.

Description

METHOD AND ELECTRONIC DEVICE FOR PERFORMING COLOR CORRECTION

The disclosure provided a method and an electronic device for performing color correction.

Digital cameras are a very important tool nowadays. With the advancement of technology, the digital camera has now become pocket-sized that one can carry anywhere. Further, the digital cameras are now widely incorporated into mobile devices like smartphones, tablets, laptops, etc. with the same or more capabilities and features as dedicated digital cameras. There are several other advantages of the digital camera, including ease of use and instant review, which means one can immediately review captured images and also erase them. The digital camera includes an image signal processor (ISP) to transform a raw image from sensor into a high-quality digital image. The ISP performs multiple functions, which include noise reduction, color interpolation, lens shading correction, defective pixel correction, gamma correction, local tone mapping, auto exposure, color correction or white-balance, and auto focus. The noise reduction is performed by applying de-noising techniques on the digital image that erase noise created depending on behavior & type of data and provides noise free image. The color interpolation also known as demosaicing which includes receiving Bayer inputs from the image sensor, converting raw image typically captured using a Bayer color filter array (CFA) into a color RGB image. The lens shading correction is applied to improve brightness and color non-uniformity towards the image periphery. The defective pixel correction is performed to correct defective pixels on the image sensor. The gamma correction compensates for the nonlinearity of relative intensity as the frame buffer value changes in output displays. The local tone mapping combines different exposures together in order to increase the local contrast within disparate regions of an HDR scene. The auto exposure performs automatic fine tuning of the image brightness according to the amount of light that reaches the camera sensor. The color correction is performed to improve the quality of the digital image. Generally, the color correction is performed to ensure proper color fidelity in the captured digital image by changing the color, brightness, contrast, and color temperature of the digital image or part of the digital image. The Auto focus automatically adjusts the sharpness of the image, which improves the image definition.

In order to perform color correction of the digital images, adjustment of white balance is important. Various techniques have been proposed for performing color correction including adjustment of white balance of digital image.

Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the existing method and system for providing an auto-color correction and auto-focus in an image.

In an embodiment, a method for performing color correction by an electronic device is provided. The method may include obtaining first image set including at least a part of a first object from a first camera sensor. The method may include obtaining second image set including at least a part of a second object from a second camera sensor, placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained. The method may include performing color correction based on the first image set and the second image set.

In an embodiment, an electronic device for providing performing color correction may comprise a memory configured to store instructions, at least one processor configured to execute the instructions. The at least one processor is configured to execute the instructions to obtain first image set including at least a part of a first object from a first camera sensor. The at least one processor is configured to execute the instructions to obtain second image set including at least a part of a second object from a second camera sensor, placed at an angular position with respect to the first camera sensor simultaneously with the time the first image set obtained. The at least one processor is configured to execute the instructions to perform color correction based on the first image set and the second image set.

In an embodiment, a computer-readable storage medium, storing instructions for executing the method is provided. The method may include obtaining first image set including at least a part of a first object from a first camera sensor. The method may include obtaining second image set including at least a part of a second object from a second camera sensor, placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained. The method may include performing color correction based on the first image set and the second image set.

The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described earlier, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

The accompanying drawings, which are incorporated herein and constitute a part of this disclosure, illustrate exemplary embodiments, and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components.

Figure 1 illustrates a flow diagram showing a method for providing an auto-color correction and auto-focus in an image captured by a mobile device.

Figure 2 illustrates a block diagram of a system for providing the auto-color correction and auto-focus in the image captured by the mobile device.

Figure 3 illustrates a flow diagram showing a method for processing the captured plurality of images for performing color correction of the image.

Figure 4A illustrates a flow diagram showing a method for determining region of interest in the plurality of images.

Figure 4B and Figure 4C illustrate a block diagram of an encoder decoder architecture of DNN for parsing the plurality of small units for determining the region of interest.

Figure 5A illustrates a flow diagram showing a method for generating the wide view image.

Figure 5B illustrates an exemplary embodiment for generating the wide view image.

Figure 5C illustrates an architecture of DNN for generating the wide view image.

Figure 5D illustrates a subset of the architecture for generating the wide view image.

Figure 6 illustrates a block diagram of a perspective tone mapping module.

Figure 7A illustrates a flow diagram showing a method for performing color correction of the wide view image by the perspective tone mapping module.

Figure 7B illustrates a pictorial representation of color target including predefined index color classification.

Figure 7C illustrates a pictorial representation of color boxes in hue saturation value domain for performing preferred color correction.

Figure 7D illustrates a pictorial representation of hue of the color box of the image captured from first camera.

Figure 7E illustrates a pictorial representation of saturation of the color box of the image captured from first camera and from the second camera.

Figure 7F illustrates a pictorial representation of value of the color box of the image captured from first camera and from the second camera.

Figure 8 illustrates a block diagram of a recommendation module for providing one or more recommendations to a user.

Figure 9 illustrates a flow diagram showing a method for determining the one or more recommendations.

Figure 10A illustrates a diagram for obtaining ROI in an embodiment of the disclosure.

Figure 10B illustrates a part of a diagram for obtaining ROI in an embodiment of the disclosure.

Figure 11 illustrates diagram for performing color correction in an embodiment of the disclosure.

Figure 12 illustrates a flow chart for performing color correction in an embodiment of the disclosure.

Figure 13 illustrates a block diagram of an electronic device for performing color correction in an embodiment of the disclosure.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to one skilled in the art that these specific details are only exemplary and not intended to be limiting. Additionally, it may be noted that the systems and/or methods are shown in block diagram form only in order to avoid obscuring the present disclosure. It is to be understood that various omissions and substitutions of equivalents may be made as circumstances may suggest or render expedient to cover various applications or implementations without departing from the spirit or the scope of the present disclosure. Further, it is to be understood that the phraseology and terminology employed herein are for the purpose of clarity of the description and should not be regarded as limiting.

Furthermore, in the present description, references to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The appearance of the phrase "in one embodiment" in various places in the specification is not necessarily referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Further, the terms "a" and "an" used herein do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced items. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described, which may be requirements for some embodiments but not for other embodiments.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "transmit," "receive," and "communicate," as well as derivatives thereof, encompass both direct and indirect communication. The terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation. The term "or" is inclusive, meaning and/or. The phrase "associated with," as well as derivatives thereof, means to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like.

Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.

As used here, terms and phrases such as "have," "may have," "include," or "may include" a feature (like a number, function, operation, or component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Also, as used here, the phrases "A or B," "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Further, as used here, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices. A first component may be denoted a second component and vice versa without departing from the scope of this disclosure.

It will be understood that, when an element (such as a first element) is referred to as being (operatively or communicatively) "coupled with/to" or "connected with/to" another element (such as a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that, when an element (such as a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (such as a second element), no other element (such as a third element) intervenes between the element and the other element.

As used here, the phrase "configured (or set) to" may be interchangeably used with the phrases "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specifically designed in hardware to." Rather, the phrase "configured to" may mean that a device can perform an operation together with another device or parts. For example, the phrase "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations. At least one embodiment of the disclosure may be combined.

The terms and phrases as used here are provided merely to describe some embodiments of this disclosure but not to limit the scope of other embodiments of this disclosure. It is to be understood that the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. All terms and phrases, including technical and scientific terms and phrases, used here have the same meanings as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined here. In some cases, the terms and phrases defined here may be interpreted to exclude embodiments of this disclosure.

Examples of an "electronic device" according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch) including flexible display. Note that, according to various embodiments of this disclosure, an electronic device may be one or a combination of the above-listed devices. According to some embodiments of this disclosure, the electronic device may be a flexible device. The electronic device disclosed here is not limited to the above-listed devices and may include new electronic devices depending on the development of technology.

In the following description, electronic devices are described with reference to the accompanying drawings, according to various embodiments of this disclosure. As used here, the term "user" may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

Definitions for other certain words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many if not most instances, such definitions apply to prior as well as future uses of such defined words and phrases.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope.

Referring to Figure 1, a flow diagram showing a method (100) providing an auto-color correction and auto-focus in an image captured by a mobile device is disclosed. The method may be explained in conjunction with the system disclosed in Figure2. In the flow diagram, each block may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the drawings. For example, two blocks shown in succession in Figure 1 may be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Any process descriptions or blocks in flowcharts should be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the example embodiments in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. In addition, the process descriptions or blocks in flow charts should be understood as representing decisions made by a hardware structure such as a state machine. The flow diagram starts at step (102) and proceeds to step (104).

At first, a plurality of images of a primary object from a first camera and the plurality of images from a second camera are captured simultaneously, at step 102. The plurality of images includes images captured from different angles under same lighting condition by the first camera and the second camera simultaneously by moving the mobile device in different directions. It should be noted that the images can be captured in all lighting conditions and therefore, the method is applicable in all the lighting conditions. In one embodiment, the second camera is placed at an angular position with respect to the first camera in the mobile device and hence the plurality of images captured from the second camera is different from the plurality of images captured from the first camera. In one exemplary embodiment, the first camera is a back camera which is used for capturing primary object and the second camera is a front camera of the mobile device. In an exemplary embodiment, the first camera is the front camera which is used for capturing primary object and the second camera is the back camera of the mobile device.

In an embodiment, a plurality of the images obtained by first camera sensor may be a first image set and a plurality of the images obtained by second camera sensor may be a second image set. In an embodiment, each of the first image set obtained from the first camera sensor and each of the second image set obtained from the second camera sensor corresponds to a time point. Accordingly, one of the first image set and one of the second image set may be captured simultaneously. The second image set may be obtained from the second camera sensor, placed at an angular position with respect to the first camera sensor simultaneously with the time the first image set obtained.

In an embodiment, the first camera may be the first camera sensor, and the second camera may be the second camera sensor. The first camera, the second camera, the first camera sensor, and the second camera sensor are sensors for obtaining at least one image by capturing a scene of at least one time point, and may be included in at least one device.

The mobile device used herein may include, but is not limited to, a mobile phone, smart phone, a laptop, a tablet computer, a wearable device, a communication device, or any other computing device or equipment, capable of implementing the features of the present disclosure. Thereafter, the captured plurality of images is processed, at step 104, for performing color correction of the image focusing the primary object. The color correction of the image focusing on the primary object can be explained in detail using Figure 3.

An embodiment of the disclosure include a method of performing color correction based on first image set obtained by a first camera sensor and simultaneously second image set obtained by a second camera sensor. Further, the embodiment of the disclosure include placement of the second camera at an angular position with respect to the first camera in the electronic device.

Meanwhile, in the disclosure, an operation performed using a first image set may be performed using second image set, and an operation performed using the second image set may be performed using the first image set.

Referring to Figure 2 a block diagram of a system (200) for providing the auto-color correction and auto-focus in the image captured by the mobile device is illustrated, in accordance with one or more exemplary embodiments of the present disclosure. The system comprises a region of interest determining module (202) for capturing a plurality of images of a primary object from a first camera, dividing each of the captured plurality of images into a plurality of small units using color code, and performing parsing of the plurality of small units for determining a region of interest (ROI) including the primary object.

The system further comprises an immense net module (204) configured for determining standard red green blue of each of the plurality of images captured from the first camera, generating blending weights for each of the plurality of images, and merging the plurality of images based on the blending weights and providing the wide view image including the primary object. The system further comprises a perspective tone mapping module (206) configured for performing color correction of the received wide view image.

At least one of the plurality of modules may be implemented through an AI model. A function associated with AI may be performed through the non-volatile memory, the volatile memory, and the processor. In one embodiment, the auto-color correction and auto-focus of the image may be performed by one or more processing units including a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an AI-dedicated processor such as a neural processing unit (NPU).

The one or more processing units control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the one or more memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or AI model of the desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and/or may be implemented through a separate server/system.

The AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through the calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

Referring to Figure 3 a flow diagram showing a method for processing the captured plurality of images for performing color correction of the image is illustrated, in accordance with one or more exemplary embodiments of the present disclosure. As illustrated, a region of interest is determined, at step 302, in the plurality of images captured from the first camera. In one embodiment, the region of interest includes the region which focuses on the primary object. The determination of the region of interest in the plurality of images can be explained in conjunction with Figure 4A, Figure 4B, and Figure 10.

Referring to Figure 4A, a flow diagram showing a method for determining region of interest (ROI) in the plurality of images is illustrated, in accordance with one or more exemplary embodiments of the present disclosure. Each of the plurality of images captured from the first camera is divided, at step 402, into a plurality of small units. In one embodiment, each of the captured plurality of images is divided into the plurality of small units using color code. It should be noted that the ROI is a subset of an image which requires color correction and focus. In an embodiment, the ROI is determined in each of the plurality of images captured from the first camera and the second camera so that color correction can be applied on the determined ROI based on light captured from the first camera and adjusted on image from the second camera or vice-versa.

Successively, parsing of the plurality of small units is performed for determining the region of interest, at step 404. In one embodiment, the parsing is performed using a deep neural network (DNN). The DNN is an artificial neural network (ANN) with additional depth and an increased number of hidden layers between input and output layers. The DNN is generally a feedforward network in which data flows from the input layer to the output layer without looping back. Initially, the DNN creates a map of virtual neurons and assigns weights to connections between them. The weights and inputs are multiplied and return an output between 0 and 1. In case the network do not accurately recognize a particular pattern, the weights get adjusted and certain parameters get more influential, until the DNN determines correct mathematical manipulation to fully process the data. The DNN has a symmetric architecture and is conceptually a more reasonable network. In one embodiment, encoder decoder architecture of the DNN may be used for parsing the plurality of small units, as illustrated in Figure4B.

Referring to Figure 4B, a block diagram of the encoder decoder architecture of the DNN for parsing the plurality of small units for determining the region of interest (ROI) is illustrated. As illustrated in Figure 4B, a bunch of convolution layers are utilized for performing downsampling using pooling and upsampling using unpooling inside the architecture in order to parse all the small units to locate the ROI. In one embodiment, the method for downsampling used herein is max pooling. The max-pooling operation takes largest response from each small unit of a feature map. In an embodiment, a method known as stride convolution can also be used for the downsampling.

Further, the method for upsampling based on pooling and convolution includes unpooling. Max Unpooling performs upsampling and remembers indices of largest elements before the max pooling and utilizes later on when the Max Unpooling is performed to place elements in positions of each small units where they are previously located before max pooling,

As illustrated, the input of 3 X H X W is given to the encoder-decoder, wherein 3 is a dimension of the frame, H is height of the frame, and W is a width of the frame. At first, downsampling of the received input frame is performed. The first two layers performs downsampling and provides high resolution frame of dimension D1, height (h/2), and width (w/2), the next three layers provides medium resolution frame of dimension D2, height (h/4), and width (w/4), and the next two layers provides low resolution frame of dimension D3, height (h/4), and width (w/4). After successful downsampling, upsampling of the low resolution frame of dimension D3, height (h/4), and width (w/4) is performed in next three layers, which provides medium resolution frame of dimension D2, height (h/4), and width (w/4) and again upsampling is performed in next two layers to provide high resolution frame of dimension D1, height (h/2), and width (w/2) and finally the frame of height (H) and width (W) is received at the output.

Referring to Figure 4C, a block diagram of the encoder decoder architecture of the DNN for parsing the plurality of small units for determining the region of interest (ROI) is illustrated. As illustrated in Figure 4C, feature maps disclosing input (4x4) downsampled to provide output (2x2) again fed for upsampling to provide the output (4x4). It should be noted that the output of downsampling prior to input for upsampling may have passed through other layers of the encoder and decoder, causing values to change. Meanwhile, the method of up sampling or down sampling is not limited to the disclosed example. Also, the ROI can be determined by a simpler or more complex neural network.

Successively, a wide view image is generated, at step 304. In one embodiment, the wide view image may include the primary object and generated by merging the plurality of images captured from the first camera, which can be explained in conjunction with Figure 5A-D.

Meanwhile, in an embodiment of the disclosure, an object (e.g., primary object, first object, and second object, etc.) is included in an image and is not limited to a certain object or person. It can also include anything that can be identified, such as anything (e.g., cloud, or grass) that appears in the background. Referring to Figure 5A a flow diagram showing a method for generating the wide view image is illustrated, in accordance with one or more exemplary embodiments of the disclosure. The method illustrates, a standard red green blue of each of the plurality of images captured from the first camera is determined, at step 502. In one embodiment, the standard red green blue of each of the plurality of images of the primary object is determined. It should be noted that the plurality of images of the primary object is taken by moving the camera in different directions under the same light condition.

Successively, blending weights are generated, at step 504, for each of the plurality of images. Thereafter, the plurality of images are merged based on the blending weights and the wide view image is provided, at step 506. In one embodiment, the wide view image is generated by the immense net module (204) which performs the following functions for

Wherein

is corrected image, dot (.) is Hadamard product,

weighting map for the

image rendered with the ci WB setting,

is sRGB of plurality of images.

Wherein w and h refer to the width and height of the plurality of images

Wherein

is mapped image to target WB setting,

is polynomial kernel function that projects the R, G, B channels into a higher-dimensional space, and

is initial high-resolution image rendered with the fixed WB setting.

In one embodiment, the immense net module (204) utilizes the DNN architecture to estimate the values of Wi for a given set of the plurality of images Ici(down). The DNN accepts the plurality of images and rendered with the predefined WB settings, The DNN learns to produce proper weighting maps Wi.

Reconstruction loss function:

Wherein,

is Squared Frobenius norm, Pcorr is extracted training patches from ground-truth sRGB plurality of images,

is input sRGB plurality of images rendered with the ci WB setting,

is blending weighting map generated for Pci.

In order to avoid producing out-of-gamut colors in the reconstructed image, a cross-channel softmax operator is applied to output tensor of the DNN network before computing loss, which is

Wherein,

regularization term to the produced weighting maps to encourage the DNN network to produce smooth weights,

is blending weighting map produced for Pci,

is a convolution operator,

with

is 3X3 horizontal Sobel filters, and

with

is 3X3 vertical Sobel filters.

Final Loss is computed as

Wherein L is a final loss,

is a scaling factor to control the contribution of Ls to the final loss.

Final image is generated as described in the following equation:

Wherein,

is final image and

is upsampled weighting maps.

In an embodiment, the electronic device may obtain first image set from the first camera sensor. For example, the electronic device may obtain multiple images in 4 direction (left, right, up, down) and center to obtain more illumination for balancing the scene illumination from back camera sensor. The electronic device may obtain the information associated with perspective axes help to capture 4 reverse direction and center from front camera (right, left, down, up). In this case, the electronic device may obtain total 10 images (5 images from the back camera, and 5 images from the front camera). In this case, the electronic device may obtain the same or similar scene object colors under same or similar light from the different perspective views by moving electronic device in different directions. The electronic device may correct white balancing by linearly blending among the different white balance settings images using the DNN network.

In an embodiment, the electronic device may obtain multiple images from multiple angles by moving the electronic device up, down, left and light. And, the electronic device may obtain blending weights for white balancing by analyzing light angles, different backgrounds, etc. Also, the electronic device may obtain a wide view image based on the blending weights. Meanwhile, as a type of color correction, the disclosure describe white balancing as an example, but is not limited thereto.

Referring to Figure 5B an exemplary embodiment for generating the wide view image is illustrated. As illustrated, the plurality of images taken from different direction is given to the DNN. The DNN on receiving the plurality of images determines the blending weights and then finally provides the final wide image by merging the plurality of images based on the determined blending weights. In one embodiment, the immense net module (204) utilizes a architecture of the DNN for generating the wide view image, which can be explained in conjunction with Figure 5C. The architecture of the DNN may be a GridNet architecture, but, it is not limited to.

Referring to Figure 5C, the architecture of the DNN for generating the wide view image is illustrated. As illustrated the architecture consists of six columns and four rows. Each column consists of either downsampling unit (illustrated in diagonal dotted and solid hatched) or upsampling unit (illustrated in diagonal check hatched). In one embodiment, the first three columns are downsampling units and last three are upsampling units. Each row consists of residual unit (illustrated in diagonal solid hatched). It should be noted that each residual unit except the first one produces features with the same dimensions of the input feature. For the first three columns, the dimensions of each feature received from the upper row are reduced by two, while the number of output channels is duplicated, as illustrated in the downsampling unit. Further, it is illustrated that the upsampling unit increases the dimensions of the received features by two in the last three columns and the last residual unit produces output weights with k channels. The more detail of a part of the architecture (510) is illustrated in Figure5D

Referring to Figure 5D, a subset of the architecture for generating the wide view image is illustrated. As illustrated each of the residual units (512, 516), downsampling units (514), and upsampling units (518) comprises a parametric rectified linear unit or PReLU, which is an activation function that generalizes a traditional rectified unit with a slope for negative values. Further, the number of input/output channels, stride, and padding size of each convolution layer is illustrated in the Figure 5D. The symbol c refers to the number of channels in each convolution layer. The first residual unit of the DNN, as illustrated in the GridNet architecture, accepts concatenated input images with 3k channels where k refers to the number of images rendered with k WB settings. It should be noted that each residual unit except the first one produces features with the same dimensions of the input feature.

Successively, color correction of the wide view image is performed, at step 304. In one embodiment, the color correction of the wide view image is performed using the perspective tone mapping (206), which can be explained in conjunction with Figure 6.

Referring to Figure 6, a block diagram of the perspective tone mapping module (206) is illustrated, in accordance with one or more exemplary embodiments of the disclosure.

In an embodiment, perspective tone mapping (PTM) is to obtain more illumination feature from a plurality of cameras placed at different angle perspective axis. For example, the electronic device may clip the image obtain from front camera sensor and back camera sensor based on color code. And, the electronic device may check each pixel's neighborhood from back camera and decides whether the pixel needs lightening based on similar tone mapping from front camera sensor.

In an embodiment, the plurality of cameras included in the electronic device may associated with each other spatially. For example, the front and back camera sensor are related to each other spatially at 180 degree. The cameras simultaneously generate at least one sets of perspective and timely related images. The electronics can, through automatic scene analysis, regenerate the white balance for color correction according to axis calibrated in both cameras. In an embodiment, incident light reflected from the object from first camera (back camera) may be compensated with the incident light captured from the second camera sensor (front camera) placed at Perspective Axes.

In an embodiment, the electronic device mat obtain multiple images in all directions of front and back camera. And the electronic device may perform color clipping recognition by semantic segmentation from front and back camera image. And the electronic device may create a color box in the HSV domain for second image set and a color box in the HSV domain for first image set. A predefined percentage of weight may be applied between the HSV center and first boundary. The electronic device may use the color box for the second image set for correct the color box for the first image set. The illumination of the second image set may increase or decrease. The interpolation may be applied so that the weight gradually decrease from first boundary to second boundary using genetic algorithm fuzzy logic. The electronic device may change the pixels of any images of the first image set (e.g., center image) based on the corrected color box. Using semantic segmentation, the electronic device may determine the range of appearance of color parameters by deep learning using through the AI segmentation.

In an embodiment, for the perspective tone mapping, the electronic device may perform preferred color correction. For example, in the HSV domain pixels within a specific range changes to the desired direction. For example, the human skin is skin toned, the sky is bluer, the plant is greener, etc. For example, the skin tone tuning may include skin segmentation, local tone mapping, and perspective tone mapping.

Referring to Figure 6, the perspective tone mapping module (206) comprises a segmentation sub-module (602) configured for performing color segmentation of the plurality of images from the first camera and simultaneously from the second camera. The perspective tone mapping module (206) further comprises a preferred color correction sub-module (604) configured for receiving region of interest including the primary object from the region of interest determining module (202) and correcting color of the region of interest of the image by utilizing the color segments and color target including predefined index color classification. The method can be explained in conjunction with Figure 7A-F.

Referring to Figure 7A, a flow diagram showing a method for performing color correction of the wide view image is illustrated, in accordance with one or more exemplary embodiments of the disclosure. The method discloses the plurality of images from the first camera and simultaneously from the second camera is captured, at step 702. Successively, color segmentation of the captured plurality of images is performed and color segments are provided, at step 704. Thereafter, a preferred color correction (PCC) for correcting the color of the image focusing the primary object is performed, at step 706, by utilizing the color segments and color target including predefined index color classification, which is illustrated in Figure 7B.

In one embodiment, the PCC is performed using steps which include creating a color box in hue saturation value (HSV) domain for the plurality of images, which can be explained in conjunction with Figure 7C-F.

As illustrated in Figure 7C, a pictorial representation of color boxes in hue saturation value (HSV) domain for performing preferred color correction is illustrated, in accordance with one or more exemplary embodiments of the disclosure. As illustrated, the color box is created separately for the plurality of images captured from the first camera and the second camera in the HSV domain. In the HSV domain, HC is illustrated in Figure 7D. As illustrated in Figure 7D, the HC refers to hue center, HB1 refers to hue boundary 1st, and HB2 refers to hue boundary 2nd.

Further, the SC is illustrated in Figure 7E, the SC refers to saturation center, SB1 refers to saturation boundary 1st, and SB2 refers to saturation boundary 2nd.

VC is illustrated in Figure 7F. As illustrated, the VC refers to value center, VB1 refers to value boundary 1st, and VB2 refers to value boundary 2nd.

Further, a predefined percentage of weight is applied between the HSV center and first boundary and the color box created is corrected for the plurality of images captured from the first camera using the color box created for the second camera. Thereafter, interpolation is applied to decrease weight from first boundary to second boundary, and changing pixels in the color box created for the first camera based on parameters of same color class captured by the second camera and changing position of the color box created for the first camera based on color box created for the second camera. The electronic device may perform perspective tone mapping for providing a wide view image. For example, the electronic device may perform color segmentation based on the first image set and the second image set. The color segmentation may be similar with the semantic segmentation of computer vision for object and color recognition. The electronic device may perform preferred color correction (PCC) for the color correction of the image focusing at least one object based on the color segmentation and color target including predefined index color classification. For example, the electronic device may obtain a color box in hue saturation value (HSV) domain for the first image set and a color box in HSV domain for the second image set. The electronic device may adjust the color box for the first image set using the color box for the second image set.

In one embodiment, the system further comprises a recommendation module (800) for providing one or more recommendations to a user, which can be explained in conjunction with Figure8.

Referring to Figure8, a block diagram of the recommendation module for providing one or more recommendations to the user is illustrated, in accordance with one or more exemplary embodiments of the present disclosure. As illustrated, the recommendation module (800) provides the one or more recommendations based on light luminance in an environment capturing the image. In one embodiment, the light luminance in the environment is determined by detecting position of the light source based on the determined difference value, determined gyro information, and the region of interest computed in the plurality of images captured from the first camera. The one or more recommendations include a recommendation to displace the primary object or a recommendation to the user to move in any direction with respect to the primary object.

In an embodiment, the electronic device may obtain multiple frames in all directions of cameras. The electronic device may obtain gyroscope information associated with the first image set and the second image set. And, the electronic device may analyze the obtained image illumination and detection of light sources. And the electronic device may obtain the recognized light sources. And, the electronic device may obtain position of light sources by reducing of point number creating light source of a single area. And the electronic device may obtain recommendation to move the electronic device to left, right, up, down direction based on light illumination obtained.

In an embodiment, the electronic device may obtain one or more recommendations, including a recommendation to displace the object or a recommendation to the user to move in any direction with respect to the first object or the second object to a user, based on light luminance in an environment of the electronic device. And the electronic device may provide the one or more recommendations.

In an embodiment, the electronic device may obtain gyroscope information associated with each of the first image set from the first camera and the second image set from the second camera. And the electronic device may detect position of the light source based on the gyroscope information, the first image set, and the second image set. The electronic device may determine the one or more recommendations based on the detected position of the light source.

The recommendation module (800) can be explained in conjunction with Figure 9.

Referring to Figure 9, a flow diagram showing a method for determining the one or more recommendations is illustrated, in accordance with one or more exemplary embodiments of the present disclosure. As illustrated, gyro information associated with each of the plurality of images captured from the first camera and simultaneously from the second camera is determined, at step 902. Successively, luminance of each of the plurality of images is computed, at step 904. Successively, one point of high luminance value from multiple points in each of the plurality of images is selected, at 906, for detecting light source of high intensity. Successively, a group of points related to the detected light source and difference in luminance values among them is determined, at step 908. Successively, position of the light source is detected, at step 910, based on the determined difference value, determined gyro information, and a region of interest (ROI) computed in the plurality of images captured from the first camera. Thereafter, the one or more recommendations are determined, at step 912, based on the detected position of the light source.

In an embodiment, the electronic device may obtain input frame (1010) same with input image (e.g., center image in the first image set or each of the first image set). And, the electronic device may divide or split the input image to a plurality of small units (1020). The electronic device may obtain class information (1030) associate with the each object in the frame using a DNN for classifying at least one object included in the frame according to the characteristics of the at least one object. A detailed description of the class information (1030) will be described in detail with reference to Figure 10B. The electronic device may obtain a labeled map (1040) indicating classification according to characteristics of at least one objects included in the first image set based on the plurality of small units for determining the ROI, and deep neural network for classifying objects contained in the first image set. The electronic device may obtain the color palette (1050) of the input frame (1010) based on the labeled map (1040). And the electronic device may obtain at least one ROI (1060) of the input frame (1010) based on the labeled map (1040) or color palette (1050).

In an embodiment, the electronic device may determine a region of interest (ROI) in the first image set from the first camera. The electronic device may perform white balance correction based on the light source from the second image set and the ROI in the first image set.

In an embodiment, the electronic device may divide each of the first image set into a plurality of small units. The electronic device may obtain a labeled map indicating classification according to characteristics of at least one objects included in the first image set based on the plurality of small units for determining the ROI, and deep neural network for classifying objects contained in the first image set. The electronic device may obtain the ROI based on the labeled map.

Meanwhile, in an embodiment of the disclosure, the step for obtaining the ROI is not limited thereto, and the ROI may be obtained using only some of the steps disclosed above.

In an embodiment, the electronic device may obtain class information (1030) associate with the each object in the frame using a DNN for classifying at least one object included in the frame according to the characteristics of the at least one object. In an embodiment, the electronic device may identify human, purse, plants, sideway, and building. The electronic device may obtain a plurality of maps corresponding to the input frame. The plurality of frames indicate each of the identified objects. For example, in the human map, a position corresponding to at least one small unit in which a human is identified may be indicated by 1, and a position corresponding to at least one small unit in which the human is not identified may be indicated by 0. Purse map, plants map, sideway map, and building map may be generated in the same or similar manner as the human map. Meanwhile, the characteristics of the objects and the manner for generating each map are not limited to the disclosed examples.

In an embodiment, the electronic device may comprise the first camera sensor and the second camera sensor. The first camera sensor may be a back camera sensor. The second camera sensor may be a front camera sensor. The electronic device may include angle information between the first camera and the second camera. The angle between the first camera and the second camera may be 180 degrees, but is not limited thereto. The electronic device may also include gyroscope information of the electronic device with turn rate and position corresponding to the time point the images was captured. In an embodiment, the electronic device may include the DNN for color correction based on the blending weight from a plurality of images and the gyroscope information. Meanwhile, color correction through DNN may be performed in a server, and the electronic device may provide output images by obtaining only intermediate, or result images.

In an embodiment, the electronic device may obtain the information associated with the incident light (e.g., back light) (1110) reflected from a first object from the first camera sensor captured or obtained in all directions by moving the electronic device in multiple directions. And the electronic device may obtain the gyroscope information of each of first image set obtained from the first camera sensor.

In an embodiment, the electronic device may capture images (up, down, left, right) for illuminant from the back camera sensor. The electronic device may obtain the images by zoom in or zoom out for illuminant from different perspective from the back camera sensor. The electronic device may identify at least one object or a part of the at least one object. And the electronic device may obtain the information associated with blending weight from DNN used to predict the blending weight from different images (e.g., first image set).

In an embodiment, the electronic device may obtain the information associated with the incident light (e.g., front light) (1120) reflected from the second object from the second camera sensor captured or obtained in all directions by moving the electronic device in multiple directions. And, the electronic device may obtain the gyroscope information of each of second image set obtained from the second camera.

In an embodiment, the electronic device may capture images (up, down, left, right) for illuminant from the front camera sensor. The electronic device may identify at least one object or a part of the at least one object. And the electronic device may obtain information associated with blending weight from DNN used to predict the blending weight from different images (e.g., second image set).

In an embodiment, the electronic device may obtain the information associated with blending weight from the DNN using the first image set and the DNN using the second image set. The electronic device may obtain the image applied blending light (1130) with back light (1110) and front light (1120).

In an embodiment, the electronic device may obtain the corrected white balancing image based on the information associated with blending weight by linearly blending among the first image set and the second image set. The electronic device may generate blending weights corresponding to each of the first image set. The electronic device may provide a wide view image based on merging the first image set by using the blending weights.

For example, the electronic device may perform color clipping recognition by semantic segmentation from the first image set and the second image set. And, the electronic device may obtain small units by dividing or splitting at least one images. The electronic device may generate a color box in the HSV domain for first image set. And, the electronic device may generate a color box in the HSV domain for second image set. And the electronic device may perform color correction for the color box for the first image set, by using the color box for the second image set.

At step S1210, the electronic device may obtain first image set including at least a part of a first object from a first camera sensor. In an embodiment, the electronic device may obtain a main image from a first camera sensor. And the electronic device may obtain at least one more images by slightly adjusting an angle or position of the main image from the first camera sensor. The first image set may be the main image from the first camera sensor and at least one more images from the first camera sensor. The first image set may include images captured from different angles under same lighting condition by the first camera by moving the electronic device in different directions. The first image set may obtain at least a part of a first object.

At step S1220, the electronic device may obtaining second image set including at least a part of a second object from a second camera sensor, placed at an angular position with respect to the first camera sensor simultaneously with the time the second image set obtained. In an embodiment, the electronic device may obtain a main image from a second camera sensor. And the electronic device may obtain at least one more images by slightly adjusting an angle or position of the main image from the second camera sensor. The second image set may be the main image from the second camera sensor and at least one more images from the second camera sensor. The second image set may obtain at least a part of a second object. The second image set may include images captured from different angles under same lighting condition by the second camera by moving the electronic device in different directions simultaneously with the time the first image set captured or within the predetermined time. And the main image from the second camera sensor may be corresponding to the main image from a first camera sensor and obtained simultaneously with the time the main image from the first camera obtained. And, the at least one image from the second camera sensor may be corresponding to the at least one images from the first camera sensor and obtained simultaneously with the time the first image set obtained.

Meanwhile, the first object and the second object are not necessarily limited to any object or person, and represent objects included in the image, and are not limited to the disclosed examples.

Meanwhile, the angle between the first camera and the second camera may be 180 degrees, but is not limited thereto. According to the angle, the first object and the second object may be the same or different.

At step S1230, the electronic device may perform color correction based on the first image set and the second image set.

In an embodiment, the electronic device may perform color correction by correcting the color of an image (e.g., the main image from the first camera sensor, the main image from the second camera sensor, one of other images, and an image obtained within a predefined time from the first camera sensor or the second camera sensor) based on the first image set and the second image set. For the color correction the electronic device may use at least one of detecting ROI, Immerse Net, perspective tone mapping, and recommendation or not.

Meanwhile, detailed descriptions for color correction are omitted because it is redundant.

In an embodiment, the electronic device (1300) for performing color correction is include a memory (1310), at least one processor (1320), sensors (1330) including first camera sensor (1332) and second camera sensor (1334). The memory (1310) stores an application program executable by the at least one processor (1320) to cause the at least one processor (1320) to perform at least one step of the method described above. In embodiments, a system or electronic device with a storage medium may be provided. Software program codes capable of implementing the functions of any one of the above embodiments are stored in the storage medium, capable of making a computer (or a central processing unit (CPU) or a microprocessor unit (MPU)) of the system or the electronic device read out and execute the program codes stored in the storage medium. Furthermore, some or all of actual operations may be completed by an operating system or the like running in the computer through instructions based on the program codes. The program codes read out from the storage medium may also be written into a memory provided in an extension board inserted into the computer or into a memory provided in an extension unit connected to the computer. Then, an instruction based on the program codes causes a CPU or the like installed on the extension board or the extension unit to perform some or all of the actual operations, to realize the functions of any one of the embodiments of the above method.

In an embodiment, the memory (1310) may be implemented by various storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory, and a programmable program read-only memory (PROM). The at least one processor (1320) may be implemented to include one or multiple central processing units or one or multiple field programmable gate arrays. The field programmable gate arrays are integrated with one or multiple central processing unit cores. In embodiments, the central processing unit or central processing unit core may be implemented as a CPU or an MCU.

In an embodiment, the at least one processor (1320) may be operable to perform the above examples. Also, the at least one processor (1320) may perform operation performed by at least one of region of interest determining module (202), immense net module (204), the perspective tone mapping module (206), segmentation sub-module (602), and preferred color correction sub-module (604). The at least one of modules may be interact with either the first camera sensor, the second camera sensor, or memory. Detailed descriptions are omitted because it is redundant.

In an embodiment, a method (100) for providing an auto-color correction and auto-focus in an image captured by a mobile device is provided. The method may include capturing a plurality of images of a primary object from a first camera and the plurality of images from a second camera simultaneously, wherein the second camera is placed at an angular position with respect to the first camera in the mobile device. The method may include processing the captured plurality of images for performing color correction of the image focusing the primary object.

In an embodiment, the method may include providing one or more recommendations to a user, based on light luminance in an environment capturing the image, the one or more recommendations include a recommendation to displace the primary object or a recommendation to the user to move in any direction with respect to the primary object.

In an embodiment, the method may include determining gyro information associated with each of the plurality of images captured from the first camera and simultaneously from the second camera. The method may include computing luminance of each of the plurality of images. The method may include selecting one point of high luminance value from multiple points in each of the plurality of images for detecting light source of high intensity. The method may include determining a group of points related to the detected light source and difference in luminance values among them. The method may include detecting position of the light source based on the determined difference value, determined gyro information, and a region of interest (ROI) computed in the plurality of images captured from the first camera. The method may include determining the one or more recommendations based on the detected position of the light source.

In an embodiment, the method may be applicable in all lighting conditions.

In an embodiment, the plurality of images may include images captured from different angles under same lighting condition by the first camera and the second camera simultaneously by moving the mobile device in different directions.

In an embodiment, the plurality of images captured from the second camera may be different from the plurality of images captured of the primary object from the first camera.

In an embodiment, the first camera may be a back camera and the second camera may be a front camera of the mobile device, or vice-versa.

In an embodiment, the method may include determining a region of interest (ROI) in the plurality of images captured from the first camera, wherein the ROI includes the region which focuses on the primary object. The method may include generating a wide view image including the primary object by merging the plurality of images captured from the first camera. The method may include performing color correction of the wide view image using perspective tone mapping.

In an embodiment, the method may include dividing each of the captured plurality of images into a plurality of small units using color code. The method may include performing parsing of the plurality of small units for determining the ROI, wherein the parsing is performed using deep neural network.

In an embodiment, the wide view image may be generated by performing steps including at least one of determining standard red green blue of each of the plurality of images captured from the first camera, generating blending weights for each of the plurality of images, merging the plurality of images based on the blending weights and providing the wide view image including the primary object.

In an embodiment, the perspective tone mapping for performing color correction of the wide view image including at least one of capturing the plurality of images from the first camera and simultaneously from the second camera, performing color segmentation of the captured plurality of images and providing color segments, and performing preferred color correction (PCC) for correcting the color of the image focusing the primary object by utilizing the color segments and color target including predefined index color classification.

In an embodiment, the PCC is performed using at least one of creating a color box in hue saturation value (HSV) domain for the plurality of images, wherein the color box is created separately for the plurality of images captured from the first camera and the second camera, applying a predefined percentage of weight between the HSV center and first boundary, correcting the color box created for the plurality of images captured from the first camera using the color box created for the second camera, applying interpolation to decrease weight from first boundary to second boundary, and changing pixels in the color box created for the first camera based on parameters of same color class captured by the second camera and changing position of the color box created for the first camera based on color box created for the second camera.

In an embodiment, a system (200) for providing an auto-color correction and auto-focus for an image captured by a mobile device. The system (200) may comprise a region of interest determining module (202) for capturing a plurality of images of a primary object from a first camera, dividing each of the captured plurality of images into a plurality of small units using color code, and performing parsing of the plurality of small units for determining a region of interest (ROI) including the primary object, an immense net module (204) for determining standard red green blue of each of the plurality of images captured from the first camera, generating blending weights for each of the plurality of images, and merging the plurality of images based on the blending weights and providing the wide view image including the primary object, and a perspective tone mapping module (206), for performing color correction of the wide view image.

In an embodiment, the system may comprise a recommendation module (800) for providing one or more recommendations to a user, based on light luminance in an environment capturing the image, the one or more recommendations include a recommendation to displace the primary object or a recommendation to the user to move in any direction with respect to the primary object.

In an embodiment, the recommendations module (800) may determine one or more recommendations by performing steps including at least one of determining gyro information associated with each of the plurality of images captured from the first camera and simultaneously from the second camera, computing luminance of each of the plurality of images, selecting one point of high luminance value from multiple points in each of the plurality of images for detecting light source of high intensity, determining a group of points related to the detected light source and difference in luminance values among them, detecting position of the light source based on the determined difference value, determined gyro information, and the ROI computed in the plurality of images captured from the first camera, and determining the one or more recommendations based on the detected position of the light source.

In an embodiment, the system may be applicable in all lighting conditions.

In an embodiment, the second camera may be placed at an angular position with respect to the first camera in the mobile device and the plurality of images captured from the second camera may be different from the plurality of images captured of the primary object from the first camera.

In an embodiment, the immense net module (204) utilizes a deep neural network for generating blending weights for each of the plurality of images.

In an embodiment, the perspective tone mapping module (206) may comprise at least one of a segmentation sub-module (602), configured for performing color segmentation of the captured plurality of images from the first camera and simultaneously from the second camera, and a preferred color correction sub-module (604), configured for receiving the ROI including the primary object from the region of interest determining module (202) and correcting color of the ROI of the image by utilizing the color segments and color target including predefined index color classification.

In an embodiment, a method for performing color correction by an electronic device (1300) is provided. The method may include obtaining first image set including at least a part of a first object from a first camera sensor (1332) (S1210). The method may include obtaining second image set including at least a part of a second object from a second camera (1334) sensor, placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained (S1220). The method may include performing color correction based on the first image set and the second image set (S1230).

In an embodiment, the method may include obtaining one or more recommendations, including a recommendation to displace the object or a recommendation to the user to move in any direction with respect to the first object or the second object to a user, based on light luminance in an environment of the electronic device (1300). The method may include providing the one or more recommendations.

In an embodiment, the method may include obtaining gyroscope information associated with each of the first image set from the first camera sensor (1332) and the second image set from the second camera sensor (1334). The method may include detecting position of the light source based on the gyroscope information, the first image set, and the second image set. The method may include determining the one or more recommendations based on the detected position of the light source.

In an embodiment, the first image set may include images captured from different angles under same lighting condition by the first camera sensor (1332) by moving the electronic device (1330) in different directions. The second image set may include images captured from different angles under same lighting condition by the second camera sensor (1334) by moving the electronic device (1330) in different directions simultaneously with the time the first image set obtained.

In an embodiment, the first camera sensor (1332) may be a back camera and the second camera sensor (1334) may be a front camera of the electronic device (1300).

In an embodiment, the method may include determining a region of interest (ROI) in the first image set from the first camera sensor (1332). In an embodiment, the method may include performing white balance correction based on the light source from the second image set and the ROI in the first image set.

In an embodiment, the method may include dividing each of the first image set into a plurality of small units. The method may include obtaining a labeled map indicating classification according to characteristics of at least one objects included in the first image set based on the plurality of small units for determining the ROI, and deep neural network for classifying objects contained in the first image set. The method may include obtaining the ROI based on the labeled map.

In an embodiment, the method may include generating blending weights corresponding to each of the first image set. The method may include providing a wide view image based on merging the first image set by using the blending weights.

In an embodiment, the method may include performing perspective tone mapping for providing a wide view image.

In an embodiment, the method may include performing color segmentation based on the first image set and the second image set. The method may include performing preferred color correction (PCC) for the color correction of the image focusing at least one object based on the color segmentation and color target including predefined index color classification.

In an embodiment, the method may include obtaining a color box in hue saturation value (HSV) domain for the first image set and a color box in HSV domain for the second image set. The method may include adjusting the color box for the first image set using the color box for the second image set.

In an embodiment, an electronic device (1300) for providing performing color correction may comprise a memory (1310) configured to store instructions, at least one processor (1320) configured to execute the instructions. The at least one processor (1320) is configured to execute the instructions to obtain first image set including at least a part of a first object from a first camera sensor (1332). The at least one processor (1320) is configured to execute the instructions to obtain second image set including at least a part of a second object from a second camera sensor (1334), placed at an angular position with respect to the first camera sensor simultaneously with the time the first image set obtained. The at least one processor (1320) is configured to execute the instructions to perform color correction based on the first image set and the second image set.

In an embodiment, the at least one processor (1320) is configured to execute the instructions to obtain one or more recommendations, including a recommendation to displace the object or a recommendation to the user to move in any direction with respect to the first object or the second object to a user, based on light luminance in an environment of the electronic device (1300). The at least one processor (1320) is configured to execute the instructions to provide the one or more recommendations.

In an embodiment, the at least one processor (1320) is configured to generate blending weights corresponding to each of the first image set. The at least one processor (1320) is configured to execute the instructions to provide a wide view image based on merging the first image set by using the blending weights.

In an embodiment, a computer-readable storage medium, storing instructions for executing the method is provided. The method may include obtaining first image set including at least a part of a first object from a first camera sensor (1332) (S1210). The method may include obtaining second image set including at least a part of a second object from a second camera (1334) sensor, placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained (S1220). The method may include performing color correction based on the first image set and the second image set (S1230).It has thus been seen that the system and method for providing the auto-color correction and auto-focus in the image according to the disclosure achieve the purposes highlighted earlier. Such a system and method can in any case undergo numerous modifications and variants, all of which are covered by the same innovative concept, moreover, all of the details can be replaced by technically equivalent elements. The scope of protection of the invention is therefore defined by the attached claims.

Claims

A method for performing color correction by an electronic device (1300), the method comprising:

obtaining first image set comprising a plurality of images including at least a part of a first object from a first camera sensor (1332) (S1210);

obtaining second image set comprising a plurality of images including at least a part of a second object from a second camera sensor (1334), placed at an angular position with respect to the first camera sensor, simultaneously with the time the first image set obtained (S1220); and

performing color correction based on the first image set and the second image set (S1230).
The method of claim 1, further comprising:

obtaining one or more recommendations, including a recommendation to displace the object or a recommendation to the user to move in any direction with respect to the first object or the second object to a user, based on light luminance in an environment of the electronic device (1300); and

providing the one or more recommendations.
The method of claim 2, wherein the obtaining of the one or more recommendations comprising:

obtaining gyroscope information associated with each of the first image set from the first camera sensor (1332) and the second image set from the second camera sensor (1334);

detecting position of the light source based on the gyroscope information, the first image set, and the second image set; and

determining the one or more recommendations based on the detected position of the light source.
The method of any one of claims 1 to 3, wherein the first image set include images captured from different angles under same lighting condition by the first camera sensor (1332) by moving the electronic device (1330) in different directions; and

wherein the second image set include images captured from different angles under same lighting condition by the second camera sensor (1334) by moving the electronic device (1330) in different directions simultaneously with the time the first image set obtained.
The method of any one of claims 1 to 4, wherein the first camera sensor (1332) is a back camera and the second camera sensor (1334) is a front camera of the electronic device (1300).
The method of any one of claims 1 to 5, wherein the performing of the color correction comprising:

determining a region of interest (ROI) in the first image set from the first camera sensor (1332); and

performing white balance correction based on the light source from the second image set and the ROI in the first image set.
The method of claim 6, wherein the determining of the ROI comprising:

dividing each of the first image set into a plurality of small units;

obtaining a labeled map indicating classification according to characteristics of at least one objects included in the first image set based on the plurality of small units for determining the ROI, and deep neural network for classifying objects contained in the first image set; and

obtaining the ROI based on the labeled map.
The method of any one of claims 1 to 7, wherein the performing of color correction comprising:

generating blending weights corresponding to each of the first image set; and

providing a wide view image based on merging the first image set by using the blending weights.
The method of any one of claims 1 to 8, wherein the performing of color correction comprising:

performing perspective tone mapping for providing a wide view image.
The method of claim 9, wherein the performing of perspective tone mapping comprising:

performing color segmentation based on the first image set and the second image set; and

performing preferred color correction (PCC) for the color correction of the image focusing at least one object based on the color segmentation and color target including predefined index color classification.
The method of claim 10, wherein the performing of PCC comprising:

obtaining a color box in hue saturation value (HSV) domain for the first image set and a color box in HSV domain for the second image set; and

adjusting the color box for the first image set using the color box for the second images.
An electronic device (1300) for providing performing color correction, comprising:

a memory (1310) configured to store instructions; and

at least one processor (1320) configured to execute the instructions to:

obtain first image set including at least a part of a first object from a first camera sensor (1332);

obtain second image set including at least a part of a second object from a second camera sensor (1334), placed at an angular position with respect to the first camera sensor simultaneously with the time the first image set obtained; and

perform color correction based on the first image set and the second image set.
The electronic device (1300) of claim 12, wherein the at least one processor (1320) is configured to execute the instructions to:

obtain one or more recommendations, including a recommendation to displace the object or a recommendation to the user to move in any direction with respect to the first object or the second object, based on light luminance in an environment of the electronic device (1300); and

provide the one or more recommendations.
The electronic device (1300) of any one of claims 12 to 13, wherein the at least one processor (1320) is configured to execute the instructions to:

generate blending weights corresponding to each of the first image set; and

provide a wide view image based on merging the first image set by using the blending weights.
A computer-readable storage medium, storing instructions for executing the method of any one of claims 1 to 11.