KR20240073714A - Electronic device and method for increasing the resolution of digital bokeh image - Google Patents

Abstract

According to one embodiment, an electronic device may include at least one processor and at least one camera. The at least one processor may acquire an image through the at least one camera. The at least one processor may generate a depth map including depth data corresponding to a plurality of objects. The at least one processor may receive a user input for identifying an object that will not be blurred among the plurality of objects. The at least one processor may identify an improvement area based on the object and the depth map corresponding to the user input. The at least one processor may identify a focus area based on the object and the depth map corresponding to the user input. The at least one processor may identify whether blur information in the improvement area is greater than or equal to a reference value. The at least one processor may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The at least one processor may perform blur processing on portions of the image excluding the focus area. The at least one processor may generate an output image based on a synthesis of the focus area and a portion of the image excluding the focus area.

Description

Electronic device and method for increasing the resolution of digital bokeh image {ELECTRONIC DEVICE AND METHOD FOR INCREASING THE RESOLUTION OF DIGITAL BOKEH IMAGE}

Embodiments of the present disclosure relate to an electronic device and method for increasing the resolution of a digital bokeh image.

As smartphones become smaller, the thickness of the camera module has decreased. As camera module thickness decreases, the importance of digital bokeh technology has increased. Digital bokeh is a technology that separates the subject and the background and combines a blurred background area image with an object area image corresponding to the subject.

The above information may be provided as background art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing can be applied as prior art to the present disclosure.

According to one embodiment, an electronic device may include at least one processor and at least one camera. The at least one processor may acquire an image through the at least one camera. The at least one processor may generate a depth map including depth data corresponding to a plurality of objects. The at least one processor may receive a user input for identifying an object that will not be blurred among the plurality of objects. The at least one processor may identify an improvement area based on the object and the depth map corresponding to the user input. The at least one processor may identify a focus area based on the object and the depth map corresponding to the user input. The at least one processor may identify whether blur information in the improvement area is greater than or equal to a reference value. The at least one processor may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The at least one processor may perform blur processing on portions of the image excluding the focus area. The at least one processor may generate an output image based on a synthesis of the focus area and a portion of the image excluding the focus area.

According to one embodiment, a method performed by an electronic device may include acquiring an image through at least one camera. The method may include generating a depth map including depth data corresponding to a plurality of objects. The method may include receiving a user input to identify an object that will not be blurred among the plurality of objects. The method may include identifying an improvement area based on the depth map and the object corresponding to a user input. The method may include identifying a focus area based on the depth map and the object corresponding to the user input. The method may include an operation of identifying whether blur information in the improvement area is greater than or equal to a reference value. The method may include performing image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The method may include performing blur processing on a portion of the image excluding the focus area. The method may include generating an output image based on a synthesis of the focus area and a portion of the image excluding the focus area.

1 is a block diagram of an electronic device in a network environment, according to embodiments.
2 is a block diagram illustrating a camera module, according to various embodiments.
3 shows an example of digital bokeh, according to embodiments.
Figure 4 shows an example of an output image generated without image processing to reduce blur information, according to embodiments.
5 shows an example of a depth map, according to embodiments.
Figure 6 shows an example of a reference distance range for identifying blur information in a focus area, according to embodiments.
FIG. 7 illustrates a flow of operations of an electronic device for generating an output image based on blur information of a focus area, according to embodiments.
FIG. 8 illustrates a flow of operations of an electronic device for identifying blur information when acquiring an image, according to embodiments.
9 illustrates a flow of operations of an electronic device for performing digital bokeh while acquiring an image, according to embodiments.
FIG. 10 illustrates a flow of operations of an electronic device for performing digital bokeh after acquiring an image, according to embodiments.

Terms used in the present disclosure are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this disclosure. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this disclosure, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in the present disclosure cannot be interpreted to exclude embodiments of the present disclosure.

In various embodiments of the present disclosure described below, a hardware approach method is explained as an example. However, since various embodiments of the present disclosure include technology using both hardware and software, the various embodiments of the present disclosure do not exclude software-based approaches.

Terms used in the description below refer to montaging (e.g., compositing, merging, combination), and terms that refer to a part of the image (e.g., image region). )), a term referring to a part of the obtained image (improvement area, region for improvement, region for resolution enhancement), Terms referring to specified values (reference value, threshold value), etc. are provided as examples for convenience of explanation. Accordingly, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meaning may be used. In addition, terms such as '... part', '... base', '... water', and '... body' used hereinafter mean at least one shape structure or a unit that processes a function. It can mean.

In addition, in the present disclosure, the expressions greater than or less than may be used to determine whether a specific condition is satisfied or fulfilled, but this is only a description for expressing an example, and the description of more or less may be used. It's not exclusion. Conditions written as ‘more than’ can be replaced with ‘more than’, conditions written as ‘less than’ can be replaced with ‘less than’, and conditions written as ‘more than and less than’ can be replaced with ‘greater than and less than’. In addition, hereinafter, 'A' to 'B' means at least one of the elements from A to (including A) and B (including B).

Before describing the embodiments of the present disclosure, terms necessary to describe the operations of the electronic device according to the embodiments are defined. The object may be a part of the image that corresponds to the subject. The enhancement area is a target of image processing to improve resolution and may be a portion of the image surrounding the object. The object may be designated by user input. An enhancement object may be a portion of an image contained within the enhancement area. The focus area may be a portion of the image that is not blurred when applying digital bokeh. A focus object may be a portion of an image included in the focus area. Blur information may be an indicator of how blurry a part of an image is. The depth map may include depth data corresponding to a plurality of objects in the image.

Hereinafter, various embodiments disclosed in this document will be described with reference to the attached drawings. For convenience of explanation, the sizes of components shown in the drawings may be exaggerated or reduced, and the present invention is not necessarily limited to what is shown.

1 is a block diagram of an electronic device in a network environment, according to embodiments.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or operations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a auxiliary processor 123, the auxiliary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. This learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (eg, program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 provides a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 to communicate within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band), for example, to achieve a high data transfer rate. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO (full dimensional MIMO)), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, connected to the plurality of antennas by the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Figure 2 is a block diagram 200 illustrating a camera module 180, according to various embodiments. Referring to FIG. 2, the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (e.g., buffer memory), or an image signal processor. It may include (260). The lens assembly 210 may collect light emitted or reflected from a subject that is the target of image capture. Lens assembly 210 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 210. In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 have the same lens properties (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly is different from another lens assembly. It may have one or more lens properties that are different from the lens properties of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 220 may include one or more light emitting diodes (eg, red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to one embodiment, the image sensor 230 is one image sensor selected from among image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, and the same It may include a plurality of image sensors having different properties or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 240 responds to movement of the camera module 180 or an electronic device including the same (e.g., the electronic device 101 of FIG. 1), and operates at least one lens or image sensor included in the lens assembly 210. 230 may be moved in a specific direction or the operation characteristics of the image sensor 230 may be controlled (e.g., read-out timing may be adjusted). This allows to compensate for at least some of the negative effects of the movement on the image being captured. According to one embodiment, the image stabilizer 240 uses a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 to stabilize the camera module 180 or the electronic device 101. ) can detect the above movement. According to one embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer. The memory 250 may at least temporarily store at least a portion of the image acquired through the image sensor 230 for the next image processing task. For example, when image acquisition is delayed due to the shutter or when multiple images are acquired at high speed, the acquired original image (e.g., Bayer-patterned image or high-resolution image) is stored in the memory 250. , the corresponding copy image (e.g., low-resolution image) may be previewed through a display device (e.g., the display module 160 of FIG. 1). Thereafter, when a specified condition is satisfied (eg, user input or system command), at least a portion of the original image stored in the memory 250 may be obtained and processed, for example, by the image signal processor 260. According to one embodiment, the memory 250 may be configured as at least a part of a memory (eg, memory 130 of FIG. 1) or as a separate memory that operates independently.

The image signal processor 260 may perform one or more image processes on an image acquired through the image sensor 230 or an image stored in the memory 250. The one or more image processes may include, for example, depth map creation, three-dimensional modeling, panorama creation, feature point extraction, image compositing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring ( Additionally or alternatively, the image signal processor 260 may include blurring, sharpening, or softening, and may include at least one of the components included in the camera module 180 (eg, an image sensor). (230)), the image processed by the image signal processor 260 may be stored back in the memory 250 for further processing. It may be provided as an external component of the camera module 180 (e.g., memory 130, display device 160, electronic device 102, electronic device 104, or server 108 of FIG. 1). According to one embodiment, the image signal processor 260 may be configured as at least a part of a processor (eg, processor 120 of FIG. 1) or may be configured as a separate processor that operates independently of the processor 120. When the signal processor 260 is configured as a separate processor from the processor 120, at least one image processed by the image signal processor 260 is displayed on the display device as is or after additional image processing by the processor 120. It can be displayed through (160).

According to one embodiment, the electronic device 101 may include a plurality of camera modules 180, each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera and at least another one may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.

3 shows an example of digital bokeh, according to embodiments.

Referring to FIG. 3, according to one embodiment, the first image 301 may be an image acquired through a camera. The first image 301 may include a first object 303, a second object 305, and a background area 307. The first object 303 may be a part of an image corresponding to a dog doll. The second object 305 may be a part of the image corresponding to a USB (universal serial bus) port line. The background area 307 may be a part of the image excluding the first object 303 and the second object 305. The second image 309 may be an output image obtained through digital bokeh processing on the first image 301. The second image 309 may include a first object 311, a second object 313, and a background area 315. The first object 311 may be a part of an image corresponding to a dog doll. The second object 313 may be a part of the image corresponding to the USB port line. The background area 315 may be a part of the image excluding the first object 311 and the second object 313.

According to one embodiment, the at least one processor (e.g., processor 120 of FIG. 1) may acquire an image including a plurality of objects through a camera (e.g., camera module 180 of FIG. 1). there is. For example, the at least one processor 120 may acquire an image including the first object 303, the second object 305, and the background area 307.

According to one embodiment, the at least one processor 120 may generate a depth map including depth data corresponding to a plurality of objects. For example, the at least one processor 120 may generate depth data corresponding to the first object 303, depth data corresponding to the second object 305, and objects in the background area 307. A depth map including depth data can be obtained. The at least one processor 120 may obtain depth data corresponding to a plurality of objects included in the image through the depth map. The depth data may be the distance from the subject corresponding to the object to an electronic device including the camera 180 (eg, the electronic device 101 of FIG. 1). The depth data may be obtained through a depth sensor. For example, the at least one processor 120 may transmit a first signal to obtain depth data through a depth sensor. The at least one processor 120 may receive at least a portion of a second signal in which the transmitted first signal is reflected by one of the plurality of objects through the depth sensor. The depth sensor may be a time of flight (TOF) sensor. For example, the depth data may be obtained based on images acquired from a plurality of cameras. The at least one processor 120 may acquire a first image from a first camera. The at least one processor 120 may acquire a second image from a second camera. The at least one processor 120 may compare the first image and the second image to obtain depth data corresponding to a plurality of objects. The closer the subject is to the electronic device 101, the greater the difference between the positions of the object corresponding to the subject in the first image and the object corresponding to the subject in the second image compared to the subject farther from the electronic device 101. You can. For example, the depth data may be identified based on images acquired through a single camera. The at least one processor 120 may obtain depth data from the image through the convolution neural network (CNN). However, embodiments of the present disclosure are not limited thereto.

According to one embodiment, the at least one processor 120 may receive a user input for identifying an object that will not be blurred among a plurality of objects. For example, the user may touch an area on the display corresponding to the first object 303. The first object 303 may be designated as an object that will not be blurred among a plurality of objects included in the image 301. The first object 303 may be designated as an object that appears to be in focus in digital bokeh. The digital bokeh may be an image processing method that performs blur processing on a portion of an acquired image to make it appear as if only some of the objects in the image are in focus.

According to one embodiment, the at least one processor 120 may identify an improvement area based on the object and the depth map corresponding to the user input. The improvement area may include the object and the improvement object. The improvement area may be a portion of the image on which image processing is performed to increase resolution. Performing image processing on the improvement area is to output high-resolution images for other subjects near the subject corresponding to the object. Additionally, time efficiency can be secured by performing image processing to increase resolution on some areas rather than the entire image. This is because the at least one processor 120 may blur a portion of the image excluding the enhancement area for a bokeh effect. In other words, since image processing to increase resolution may not be performed on the portion of the image to be blurred, the at least one processor 120 performs image processing to increase resolution on the improvement area. Time efficiency can be secured. The at least one processor 120 may identify the distance of the object corresponding to the received user input based on the depth map. The at least one processor 120 may identify the improvement object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the improvement object to the electronic device 101 may be less than a second specified distance.

For example, the at least one processor 120 may identify the distance between the first object 303 corresponding to the user input and the electronic device 101 based on the depth map. The at least one processor 120 calculates the distance from the puppy doll corresponding to the first object 303 to the electronic device 101 and the distance from the subject corresponding to the improvement object to the electronic device 101. The improvement object whose difference is less than the second specified distance may be identified. The difference between the distance from the puppy doll corresponding to the first object 303 to the electronic device 101 and the distance from the USB port line corresponding to the second object 305 to the electronic device 101, It may be less than the second specified distance. The second object 305 may be the improvement object. The improvement area may include a first object 303 and a second object 305. The at least one processor 120 identifies the improvement area based on the first object 303 corresponding to the user input and the second designated distance to implement a natural bokeh image. You can.

According to one embodiment, the at least one processor 120 may determine whether blur information in the improvement area is greater than or equal to a reference value.

According to one embodiment, the at least one processor 120 may identify blur information based on edge characteristics of pixels included in the image. The blur information may include the degree of blur. An edge may be an indicator of how diverse the color change is compared to adjacent pixels. The value of the edge may be larger as the color change becomes more diverse compared to adjacent pixels. The edge characteristics may include the strength of the edge and the consistency of the edge direction. The at least one processor 120 may identify the intensity of the edge included in the pixel and the consistency of the edge direction. The intensity of the edge may be an indicator of how many edges with large changes between the corresponding pixel and adjacent pixels are included. The greater the change in color between adjacent pixels, the greater the intensity of the edge may be. The more pixels there are with varying colors, the larger the edges included in that pixel and adjacent pixels can be. Accordingly, the strength of the edge of the corresponding pixel may be determined according to the magnitude of gradient of the corresponding pixel. The strength of the edge may be identified based on the size of the gradient in a first direction of the pixel and adjacent pixels included in the image and the size of the gradient in a second direction perpendicular to the first direction. The larger the size of the gradient in the first direction and the size of the gradient in the second direction, the lower the degree of blur may be. The consistency of the edge direction may be an indicator of how consistent the edges included in the corresponding pixel and adjacent pixels are oriented. The edge direction may be orthogonal to the gradient. Therefore, if the direction of the gradient is consistent, the edge direction can also be consistent. The consistency of the edge direction may be identified according to the distribution of the gradient in the first direction of the pixel and the adjacent pixel and the distribution of the gradient in the second direction perpendicular to the first direction. The greater the consistency of the gradient in the first direction, the higher the consistency in the edge direction may be. The greater the consistency of the gradient in the second direction, the higher the consistency in the edge direction may be. The higher the consistency of the edge direction, the higher the degree of blur may be. The higher the consistency of the edge direction, the more likely it is that the blur was created by the movement of the subject.

According to one embodiment, the at least one processor 120 may identify blur information based on the distance of the object. When there is a subject corresponding to the object within the first reference distance range, the sharpness of the object having a distance within the first distance range may be greater than or equal to a specified value. When there is a subject corresponding to the object outside the first reference distance range, the sharpness of the object having a distance outside the first distance range may be less than a specified value. The at least one processor 120 identifies the blur information in the improvement area as being less than a reference value when the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range. You can. The at least one processor 120 may identify the blur information of the improvement area as a reference value or more when the distance of the object corresponding to the user input from the electronic device 101 is outside the first reference distance range. You can. The first reference distance range may be identified based on depth of field and the distance from the focused subject to the electronic device 101. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The first reference distance range may be identified based on the aperture value, the focal length of the camera used, and the distance between the focused subject and the electronic device 101. For example, as the amount of light that the aperture passes through increases, the first reference distance range may become narrower. In other words, if the f-number of the aperture is lowered, the first reference distance range can be narrowed. For example, the larger the focal length of the camera used, the narrower the first reference distance range may be. In other words, the higher the magnification of the camera, the narrower the first reference distance range may be. For example, as the distance between the focused subject and the electronic device 101 increases, the first reference distance range may become wider. In other words, the farther an object is focused from the electronic device, the wider the first reference distance range can be.

According to one embodiment, the at least one processor 120 may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The at least one processor 120 may perform the image processing to reduce the blur information through a neural network. Image processing to reduce the blur information for the improvement area may be performed through a central processing unit (CPU), a neural processing unit (NPU), or a graphics processing unit (GPU).

According to one embodiment, the at least one processor 120 may identify a focus area based on the object and the depth map corresponding to the user input. The focus area may include the object and the focus object. The at least one processor 120 may identify the distance of the object corresponding to the received user input based on the depth map. The at least one processor 120 may identify the focus object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the focus object to the electronic device 101 may be less than a first designated distance. For example, the at least one processor 120 may identify the distance between the first object 303 corresponding to the user input and the electronic device 101 based on the depth map. The at least one processor 120 calculates the distance from the puppy doll corresponding to the first object 303 to the electronic device 101 and the distance from the subject corresponding to the focus object to the electronic device 101. A focus object whose difference is less than the first specified distance may be identified. The difference between the distance from the puppy doll corresponding to the first object 303 to the electronic device 101 and the distance from the USB port line corresponding to the second object 305 to the electronic device 101, It may be less than the first designated distance. The second object 305 may be a focus object. The focus object included in the focus area may be identified as an object that will not be blurred among a plurality of objects included in the image 301. The focus object can be identified in digital bokeh as an object that appears to be in focus. The focus area may include a first object 303 and a second object 305. The at least one processor 120 identifies the focus area based on the first object 303 corresponding to the user input and the first designated distance to implement a natural bokeh image. You can.

According to one embodiment, the at least one processor 120 may perform blur processing on a portion of the image excluding the focus area. The at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area. For example, the at least one processor 120 may perform blur processing on the background area 307 excluding the focus area. The at least one processor 120 may generate a second image 309, which is an output image, by combining the focus area with the blurred background area 315. The focus area may include the first object 311 and the second object 313.

According to one embodiment, the at least one processor 120 may blur an object with a stronger intensity as the distance to the electronic device 101 increases. The intensity of the blur processing may be at the level of blur. The stronger the intensity of the blur processing, the blurrier the blurred image may be expressed. For example, an image blurred with a first intensity may be blurred compared to an image blurred with a second intensity that is less than the first intensity. For example, the at least one processor 120 may blur the first subject corresponding to the first image portion with a first intensity. The at least one processor 120 may blur the second subject corresponding to the second image portion with a second intensity that is smaller than the first intensity. The first subject may be separated from the electronic device 101 by a first distance. The second subject may be separated from the electronic device 101 by a second distance. The first distance may be greater than the second distance.

According to one embodiment, the at least one processor 120 may generate the output image 309 to which the bokeh effect is applied through software processing without changing the optical settings of the camera.

According to one embodiment, the at least one processor may display a notification on the display to indicate whether the digital bokeh effect can be applied. For example, the at least one processor 120 may generate the depth map including the depth data corresponding to the plurality of objects while acquiring the image through the at least one camera. The at least one processor 120 may identify whether the distance of the object is outside the second reference distance range. The at least one processor 120 may display a notification indicating difficulty in applying the bokeh effect based on identifying whether the distance of the object is outside the second reference distance range. The second reference distance range may be set to identify cases where it is difficult to increase the resolution beyond a specified value even if image processing to increase the resolution is performed. Even if image processing to increase resolution is performed on an object corresponding to a subject located outside the second reference distance range, the resolution may not be increased beyond a specified value. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

According to one embodiment, the at least one processor 120 may acquire the image through the at least one camera. The at least one processor 120 may identify the distance of the object in the image acquired through the at least one camera. The at least one processor 120 may display a notification based on identifying whether the distance of the object is outside the first reference distance range and within the second reference distance range. The notification may indicate that image processing is necessary to improve resolution when applying the bokeh effect. This is because, when the distance of the object is outside the first reference distance range, image processing is required to improve resolution. The image processing to improve the resolution may be performed to improve the performance of the digital bokeh effect. When the distance of the object is within the second reference distance range, if image processing to improve the resolution is performed, the resolution of the object may become less than a specified value. The at least one processor 120 may display a notification indicating difficulty in applying the bokeh effect based on identifying whether the distance of the object is outside the second reference distance range. This is because even if image processing to increase resolution is performed on an object corresponding to a subject located outside the second reference distance range, the resolution may not be increased beyond a specified value. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

Figure 4 shows an example of an output image generated without image processing to reduce blur information, according to embodiments.

Referring to FIG. 4, according to one embodiment, an electronic device (e.g., the electronic device 101 of FIG. 1) captures the first image 401 and the second image 407 with a camera (e.g., the camera module of FIG. 1). It can be obtained through (180)). The first image 401 may be obtained by focusing on and photographing the first object 403. The first image 401 may include a first object 403 and a second object 405. The first object 403 may be a part of the first image 401 corresponding to a dog doll. The second object 405 may be a part of the first image 401 corresponding to a human-shaped doll. The second image 407 may be an output image obtained through digital bokeh processing on the first image 401. The second image 407 may include a first object 409 and a second object 411. The first object 409 may be a part of the second image 407 corresponding to a dog doll. The second object 411 may be a part of the second image 407 corresponding to a human-shaped doll. The second image 407 may include the second object 411 in the focus area.

According to one embodiment, the electronic device 101 may acquire the third image 413 and the fourth image 419 through the camera 180. The third image 413 may be obtained by focusing on and photographing the second object 417. The third image 413 may include a first object 415 and a second object 417. The first object 415 may be a part of the third image 413 corresponding to a dog doll. The second object 417 may be a part of the third image 413 corresponding to a human-shaped doll. The fourth image 419 may be an output image obtained through digital bokeh processing on the third image 413. The fourth image 419 may include a first object 421 and a second object 423. The first object 421 may be a part of the fourth image 419 corresponding to a dog doll. The second object 423 may be a part of the fourth image 419 corresponding to a human-shaped doll. The fourth image 419 may include the first object 421 in the focus area.

According to one embodiment, the second object 405 may be photographed blurry due to optical reasons. For example, at least one processor (eg, processor 120 in FIG. 1) may acquire the first image 401 by focusing on and photographing the first object 403. The second object 405 may be photographed blurryly, beyond the depth of field. When the lens focuses on a specific subject, other subjects within a certain distance from the subject may also be focused. The depth of field may be a certain distance range in which the other subjects are also in focus. The distance between the electronic device 101 and the subject corresponding to the second object 405 may deviate from the depth of field. Because the subject corresponding to the second object 405 is located outside the depth of field, the second object 405 may be blurred.

According to one embodiment, the at least one processor 120 may generate a depth map including depth data corresponding to a plurality of objects. For example, the at least one processor 120 may obtain a depth map including depth data corresponding to the first object 403 and depth data corresponding to the second object 405. The depth data may be the distance from the subject corresponding to the object to the electronic device 101 including the camera 180. The at least one processor 120 may receive a user input for identifying an object that will not be blurred among a plurality of objects. For example, the user may touch an area on the display corresponding to the second object 405. The at least one processor 120 may identify a focus area based on the second object 405 and the depth map corresponding to the user input. The focus area may include the object and the focus object. For example, the focus area may include the second object 405 and an object corresponding to the partition wall where the second object 405 is located.

According to one embodiment, the at least one processor 120 may perform blur processing on portions of the image excluding the focus area. The at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area. For example, the focus area may include a second object 405. For example, the at least one processor 120 may perform blur processing on portions of the image excluding the focus area. The at least one processor 120 may generate a second image 407, which is an output image, by combining the image portion excluding the blurred focus area with the focus area. The focus area may include the second object 405. The output image may be the second image 407. However, even if blur processing for a digital bokeh effect is not performed on the second object 411, the second object 411 may be blurred. This may be because the second object 411 is outside the depth of field. In the second image 407, when the second object 411 is blurred, the digital bokeh effect may not be implemented. This is because the greater the difference between the sharpness of the focus area and the sharpness of the image portion excluding the focus area, the easier it is to implement the digital bokeh effect. If the degree of blur between the first object 409 and the second object 411 is similar, it may be difficult to implement the digital bokeh effect. Therefore, the at least one processor 120 may perform image processing around the improvement area to reduce the degree of blur by improving the resolution of the focus area.

According to one embodiment, the first object 415 may be photographed blurry due to optical reasons. For example, the at least one processor 120 may acquire the third image 413 by focusing on and photographing the second object 417 . The first object 415 may be photographed blurry, outside the depth of field. When a lens focuses on a specific subject, other subjects within a certain distance range from the subject may also be in focus. The depth of field may be a certain distance range in which the other subjects are also in focus. The distance between the electronic device 101 and the subject corresponding to the first object 415 may deviate from the depth of field. Because the subject corresponding to the first object 415 is located outside the depth of field, the first object 415 may be blurred.

According to one embodiment, the at least one processor 120 may generate a depth map including depth data corresponding to a plurality of objects. For example, the at least one processor 120 may obtain a depth map including depth data corresponding to the first object 415 and depth data corresponding to the second object 417. The depth data may be the distance from the subject corresponding to the object to the electronic device 101 including the camera 180. The at least one processor 120 may receive a user input for identifying an object that will not be blurred among a plurality of objects. For example, the user may touch an area on the display corresponding to the first object 415. The at least one processor 120 may identify a focus area based on the first object 415 and the depth map corresponding to the user input. The focus area may include the first object 415 and the focus object. For example, the focus area may include the first object 415 and an object corresponding to a USB port line next to the first object 415.

According to one embodiment, the at least one processor 120 may perform blur processing on portions of the image excluding the focus area. The at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area. For example, the focus area may include the first object 415. For example, the at least one processor 120 may perform blur processing on portions of the image excluding the focus area. The at least one processor 120 may generate a fourth image 419, which is an output image, by combining the image portion excluding the blurred focus area with the focus area. The focus area may include the first object 421. The image portion excluding the focus area may include a second object 423. The output image may be the fourth image 419. However, even if blur processing for a digital bokeh effect is not performed on the first object 421, the first object 421 may be blurred. This may be because the first object 421 is outside the depth of field. In the fourth image 419, when the first object 421 is blurred, the digital bokeh effect may not be implemented. This is because the greater the difference between the sharpness of the focus area and the sharpness of the image portion excluding the focus area, the easier it is to implement the digital bokeh effect. If the degree of blur between the first object 421 and the second object 423 is similar, it may be difficult to implement the digital bokeh effect. Therefore, the at least one processor 120 may perform image processing on the focus area to improve resolution and reduce the degree of blur.

5 shows an example of a depth map, according to embodiments.

Referring to Figure 5, according to one embodiment, image 501 may be acquired through a camera. The image 501 may include a first object 503, a second object 505, and a third object 507. The first object 503 may be a part of the image 501 corresponding to a dog doll. The second object 505 may be a part of the image 501 corresponding to a cat doll. The third object 507 may be a part of the image 501 corresponding to a human doll. The depth map 509 can intuitively express depth data corresponding to a plurality of objects included in the image 501. At least one processor (eg, processor 120 in FIG. 1) may obtain depth data of the first object 503 through the first area 511 of the depth map. The at least one processor 120 may obtain depth data of the second object 505 through the second area 513 of the depth map. The at least one processor 120 may obtain depth data of the third object 507 through the third area 515 of the depth map. In the depth map, an area corresponding to a subject closer to the electronic device (e.g., the electronic device 101 in FIG. 1) may be expressed with lower brightness compared to an area corresponding to a subject farther from the electronic device 101. .

According to one embodiment, the distance from the subject corresponding to the first object 503 to the electronic device 101 is the distance from the subject corresponding to the second object 505 to the electronic device 101. It may be closer. The distance from the subject corresponding to the second object 505 to the electronic device 101 may be shorter than the distance from the subject corresponding to the third object 507 to the electronic device 101. Therefore, the brightness of the first area 511 may be lower than that of the second area 513. The brightness of the second area 513 may be lower than that of the third area 515. The depth data may be the distance from the subject corresponding to the object to the electronic device 101 including the camera (eg, camera 180 in FIG. 1).

According to one embodiment, the depth data may be obtained through various methods. For example, the depth data may be obtained through a depth sensor. The at least one processor 120 may transmit a first signal to obtain depth data through a depth sensor. The at least one processor 120 may receive at least a portion of a second signal in which the transmitted first signal is reflected by one of the plurality of objects through the depth sensor. The depth sensor may be a time of flight (TOF) sensor. For example, the depth data may be obtained based on images acquired from a plurality of cameras. The at least one processor 120 may acquire a first image from a first camera. The at least one processor 120 may acquire a second image from a second camera. The at least one processor 120 may compare the first image and the second image to obtain depth data corresponding to a plurality of objects. The closer the subject is to the electronic device 101, the greater the difference between the positions of the object corresponding to the subject in the first image and the object corresponding to the subject in the second image compared to the subject farther from the electronic device 101. Because it is. For example, the depth data may be identified based on images acquired through a single camera. The at least one processor 120 may obtain depth data from the image through the convolution neural network (CNN). However, embodiments of the present disclosure are not limited thereto.

According to one embodiment, the at least one processor 120 may obtain information or identify an area through the depth data. For example, the at least one processor 120 may obtain blur information of the improvement area based on the depth data. The improvement area may correspond to another subject near the subject corresponding to the object. When the blur information in the improvement area is greater than or equal to a reference value, the at least one processor 120 may perform image processing to reduce the blur information in the improvement area. Image processing to reduce blur information may be performed through a central processing unit (CPU), a neural processing unit (NPU), or a graphics processing unit (GPU).

For example, the at least one processor 120 may identify an improvement area based on the depth data. The improvement area may include an object corresponding to a user input and an improvement object. The at least one processor 120 may identify the distance between the object corresponding to the user input and the electronic device 101, based on the depth map 509. The at least one processor 120 may identify the improvement object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the improvement object to the electronic device 101 may be less than a second specified distance.

For example, the at least one processor 120 may identify a focus area based on the depth data. The focus area may include an object corresponding to a user input and a focus object. The at least one processor 120 may identify the distance between the object corresponding to the user input and the electronic device 101 based on the depth map 509. The at least one processor 120 may identify the focus object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the focus object to the electronic device 101 may be less than a first designated distance.

Figure 6 shows an example of a reference distance range for identifying blur information in a focus area, according to embodiments.

Referring to Figure 6, according to one embodiment, in state 601, at least one processor (e.g., processor 120 of Figure 1) determines the f-number of the aperture 607, sensor Depending on the size of the CCD (charge coupled device) and the focal length of the lens, the distance range at which objects can be captured with high clarity can be identified. For example, the F-number of the aperture 607 may be 2.8. In state 603, the at least one processor 120 operates according to the f-number of the aperture 609, the size of the sensor (e.g., charge coupled device (CCD)), and the focal length of the lens. It is possible to identify the distance range where objects are photographed with high clarity. For example, the F-number of the aperture 609 may be 5.6. In state 605, the at least one processor 120 operates according to the f-number of the aperture 611, the size of the sensor (e.g., charge coupled device (CCD)), and the focal length of the lens. It is possible to identify the distance range where objects are photographed with high clarity. For example, the F-number of the aperture 611 may be 11.

According to one embodiment, the degree of blur caused by optical causes of an object included in an image may be determined by the distance between an electronic device (eg, the electronic device 101 of FIG. 1) and a subject corresponding to the object. The distance between the electronic device 101 and the subject corresponding to the object may hereinafter be referred to as the distance of the object. When a lens focuses on a specific subject, other subjects within a certain distance from the subject may also be in focus. A certain distance range in which the other subjects are also in focus may be referred to as depth of field. The distance between the electronic device 101 and the subject corresponding to the object may deviate from the depth of field. When the distance of the object exceeds the depth of field, the object may be expressed as blur.

According to one embodiment, the distance range of an object in which the object's sharpness is photographed above a specified value may be referred to as a first reference distance range. The first reference distance range may be identified by the depth of field. The depth of field may be identified based on the aperture value, the focal length of the camera used, and the distance between the focused subject and the electronic device 101. For example, the greater the amount of light that the aperture passes through, the narrower the depth of field may be. In other words, lowering the f-number of the aperture may narrow the depth of field. For example, the larger the focal length of the camera used, the narrower the depth of field may be. In other words, the higher the magnification of the camera, the narrower the depth of field may be. For example, the greater the distance between the focused subject and the electronic device 101, the wider the depth of field may be. In other words, the deeper an object is focused on that is farther away from the electronic device, the wider the depth of field can be.

According to one embodiment, in state 601, the distance from the electronic device 101 to the flower may be within the depth of field range. The distance from the electronic device 101 to the dog may be outside the depth of field range. The distance from the electronic device 101 to the tree may be outside the depth of field range. Therefore, the sharpness of the object corresponding to the flower in the acquired image may be greater than or equal to a specified value. The sharpness of the object corresponding to the dog may be less than a specified value. The sharpness of the object corresponding to the tree may be less than a specified value. In other words, the object corresponding to the flower in the acquired image can be acquired relatively clearly. Objects corresponding to dogs and objects corresponding to trees can be obtained blurrily. When the distance of an object is outside the depth of field range, the object may be blurred due to optical causes.

According to one embodiment, in state 603, the distance from the electronic device 101 to the flower may be within the depth of field range. The distance from the electronic device 101 to the dog may be within the depth of field range. The distance from the electronic device 101 to the tree may be outside the depth of field range. Therefore, the sharpness of the object corresponding to the flower in the acquired image may be greater than or equal to a specified value. The clarity of the object corresponding to the dog may be greater than or equal to a specified value. The sharpness of the object corresponding to the tree may be less than a specified value. In other words, objects corresponding to flowers and dogs in the acquired image may be acquired relatively clearly, and objects corresponding to trees may be obtained blurry.

According to one embodiment, in state 605, the distance from the electronic device 101 to the flower may be within the depth of field range. The distance from the electronic device 101 to the dog may be within the depth of field range. The distance from the electronic device 101 to the tree may be within the depth of field range. Therefore, the sharpness of the object corresponding to the flower in the acquired image may be greater than or equal to a specified value. The clarity of the object corresponding to the dog may be greater than or equal to a specified value. The clarity of the object corresponding to the tree may be greater than or equal to a specified value. In other words, objects corresponding to flowers, objects corresponding to dogs, and objects corresponding to trees in the acquired image can all be clearly acquired.

According to one embodiment, in state 601, the f-number of the aperture 607 may be 2.8. In state 603, the f-number of the aperture 609 may be 5.6. The depth of field identified by the F-number of the aperture 607 may be shallower than the depth of field identified by the F-number of the aperture 609. In state 603, the f-number of the aperture 609 may be 5.6. In state 605, the f-number of the aperture 611 may be 11. The depth of field identified by the F-number of the aperture 609 may be shallower than the depth of field identified by the F-number of the aperture 611. This is because the greater the amount of light that the aperture lets through, the narrower the depth of field can be.

According to one embodiment, the at least one processor 120 may identify whether the blur information of the object is greater than or equal to a specified value according to the distance of the object. Therefore, when the distance of the object is within the depth of field (e.g., a first reference distance range), the at least one processor 120 determines that the blur information is greater than a specified value and performs image processing to reduce the blur information. can be performed. Image processing to reduce the blur information may be performed to increase resolution. Image processing to reduce the blur information may be performed through a central processing unit (CPU), a neural processing unit (NPU), or a graphics processing unit (GPU).

FIG. 7 illustrates a flow of operations of an electronic device for generating an output image based on blur information of a focus area, according to embodiments.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 701 to 715 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1). there is.

Referring to Figure 7, according to one embodiment, in operation 701, at least one processor (e.g., processor 120 of Figure 1) may acquire an image and generate a depth map. According to one embodiment, the at least one processor 120 may generate a depth map including depth data corresponding to a plurality of objects. For example, the at least one processor 120 may store depth data corresponding to a first object (e.g., the first object 303 in FIG. 3) and a second object (e.g., the second object 305 in FIG. 3). )), and a depth map including depth data corresponding to a third object (eg, one of the objects in the background area 307 of FIG. 3). The at least one processor 120 may obtain depth data corresponding to a plurality of objects included in the image through the depth map. The depth data may be the distance from the subject corresponding to the object to an electronic device (e.g., the electronic device 101 of FIG. 1) including the camera (e.g., the camera module 180 of FIG. 1). The depth data may be obtained through a depth sensor. For example, the at least one processor 120 may transmit a first signal through a depth sensor to obtain depth data. The at least one processor 120 may receive at least a portion of a second signal in which the transmitted first signal is reflected by one of the plurality of objects through the depth sensor. The depth sensor may be a time of flight (TOF) sensor. For example, the depth data may be obtained based on images acquired from a plurality of cameras. The at least one processor 120 may acquire a first image from a first camera. The at least one processor 120 may acquire a second image from a second camera. The at least one processor 120 may compare the first image and the second image to obtain depth data corresponding to a plurality of objects. The closer the subject is to the electronic device 101, the greater the difference between the positions of the object corresponding to the subject in the first image and the object corresponding to the subject in the second image compared to the subject farther from the electronic device 101. Because it is. For example, the depth data may be identified based on images acquired through a single camera. The at least one processor 120 may obtain depth data from the image through the convolution neural network (CNN). However, embodiments of the present disclosure are not limited thereto.

According to one embodiment, in operation 703, the at least one processor 120 may receive a user input for identifying an object that will not be blurred. For example, the user may touch an area on the display corresponding to a first object (eg, the first object 303 in FIG. 3). The first object 303 may be designated as an object that will not be blurred among a plurality of objects included in the image. The first object 303 may be designated as an object that appears to be in focus in digital bokeh. The digital bokeh may be an image processing method that performs blur processing on a portion of an acquired image to make it appear as if only some of the objects in the image are in focus.

According to one embodiment, in operation 705, the at least one processor 120 may identify an enhancement area and a focus area based on the object and the depth map. According to one embodiment, the at least one processor 120 may identify an improvement area based on the object and the depth map corresponding to the user input. The improvement area may include the object and the improvement object. The improvement area may be a portion of the image on which image processing is performed to increase resolution. Performing image processing on the improvement area is to output high-resolution images for other subjects near the subject corresponding to the object. Additionally, time efficiency can be secured by performing image processing to increase resolution on some areas rather than the entire image. This is because image processing to increase resolution may not be performed on the portion of the image to be blurred. The at least one processor 120 may identify the distance of the object corresponding to the received user input based on the depth map. The at least one processor 120 may identify the improvement object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the improvement object to the electronic device 101 may be less than a second designated distance.

According to one embodiment, the at least one processor 120 may identify a focus area based on the object and the depth map corresponding to the user input. The focus area may include the object and the focus object. The at least one processor 120 may identify the distance of the object corresponding to the received user input based on the depth map. The at least one processor 120 may identify the focus object. The difference between the distance from the subject corresponding to the object to the electronic device 101 and the distance from the subject corresponding to the focus object to the electronic device 101 may be less than a first designated distance.

According to one embodiment, in operation 707, the at least one processor 120 may identify blur information of the enhancement area. According to one embodiment, the at least one processor 120 may identify blur information based on edge characteristics of pixels included in the image. The blur information may include the degree of blur. The edge may be an indicator indicating how diverse the color change is compared to adjacent pixels. The edge value may be larger as the color change becomes more diverse compared to adjacent pixels. The edge characteristics may include the strength of the edge and the consistency of the edge direction. The intensity of the edge may be an indicator of how many edges with large changes between the corresponding pixel and adjacent pixels are included. The greater the color change between adjacent pixels, the greater the edge strength. Accordingly, the strength of the edge of the corresponding pixel may be determined according to the magnitude of gradient of the corresponding pixel. The strength of the edge may be identified based on the size of the gradient in a first direction of the pixel and the adjacent pixel included in the image and the size of the gradient in a second direction perpendicular to the first direction. The larger the size of the gradient in the first direction and the size of the gradient in the second direction, the lower the degree of blur may be. The consistency of the edge direction may be an indicator of how consistent the edges included in the corresponding pixel and adjacent pixels are oriented. The edge direction may be orthogonal to the gradient. Therefore, if the direction of the gradient is consistent, the edge direction can also be consistent. The consistency of the edge direction may be identified according to the distribution of the gradient in the first direction of the pixel and the adjacent pixel and the distribution of the gradient in the second direction perpendicular to the first direction. The greater the consistency of the gradient in the first direction, the higher the consistency in the edge direction may be. The greater the consistency of the gradient in the second direction, the higher the consistency in the edge direction may be. The higher the consistency of the edge direction, the higher the degree of blur may be. The higher the consistency of the edge direction, the more likely it is that the blur was created by the movement of the subject.

According to one embodiment, the at least one processor 120 may identify blur information based on the distance of the object. When there is a subject corresponding to the object within the first reference distance range, the sharpness of the object having a distance within the first distance range may be greater than or equal to a specified value. When there is a subject corresponding to the object outside the first reference distance range, the sharpness of the object having a distance outside the first distance range may be less than a specified value. The at least one processor 120 identifies the blur information in the improvement area as being less than a reference value when the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range. You can. The at least one processor 120 may identify the blur information of the improvement area as a reference value or more when the distance of the object corresponding to the user input from the electronic device 101 is outside the first reference distance range. You can. The first reference distance range may be identified based on depth of field and the distance from the focused subject to the electronic device 101. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The first reference distance range may be identified based on the aperture value, the focal length of the camera used, and the distance between the focused subject and the electronic device 101.

According to one embodiment, in operation 709, the at least one processor 120 may identify whether blur information in the improvement area is greater than or equal to a reference value. The at least one processor 120 may perform operation 713 when the blur information in the improvement area is greater than or equal to a reference value. The at least one processor 120 may perform operation 711 when the blur information of the improvement area is less than a reference value. This is because the operations performed vary depending on whether the improvement area requires image processing to improve resolution.

According to one embodiment, in operation 711, the at least one processor 120 may perform blur processing on a portion of the image excluding the focus area. According to one embodiment, the at least one processor 120 may blur an object with a stronger intensity as the distance to the electronic device 101 increases. The intensity of the blur processing may be at the level of blur. The stronger the intensity of the blur processing, the blurrier the blurred image may be expressed. For example, an image blurred with a first intensity may be blurred compared to an image blurred with a second intensity that is less than the first intensity. For example, the at least one processor 120 may blur the first subject corresponding to the first image portion with a first intensity. The at least one processor 120 may blur the second subject corresponding to the second image portion with a second intensity that is smaller than the first intensity. The first subject may be separated from the electronic device 101 by a first distance. The second subject may be separated from the electronic device 101 by a second distance. The first distance may be greater than the second distance.

According to one embodiment, in operation 713, the at least one processor 120 may perform image processing on the enhancement area. According to one embodiment, the at least one processor 120 may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The at least one processor 120 may perform the image processing to reduce the blur information through a neural network. Image processing to reduce the blur information for the improvement area may be performed through a central processing unit (CPU), a neural processing unit (NPU), or a graphics processing unit (GPU).

According to one embodiment, in operation 715, the at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area. The at least one processor 120 may generate an output image in which the sharpness of the focus area is higher than that of the non-focus area. The at least one processor 120 may perform image processing to increase resolution on an improvement area around the object designated by the user to generate an output image. The at least one processor 120 may obtain an output image with relatively high clarity for the focus area around the object designated by the user.

FIG. 8 illustrates a flow of operations of an electronic device for identifying blur information when acquiring an image, according to embodiments.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 801 to 805 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1). there is.

Referring to FIG. 8, according to one embodiment, in operation 801, at least one processor (eg, processor 120 of FIG. 1) may acquire depth data corresponding to a plurality of objects. For example, the depth data may be obtained by a depth sensor. As another example, the depth data may be identified by images acquired from a plurality of cameras. The depth data can be obtained by inputting an image acquired from a single camera into a neural network.

According to one embodiment, in operation 803, the at least one processor 120 may identify a first reference distance range. The sharpness of the object corresponding to the subject placed in the first reference distance range may be greater than or equal to a specified value. The first reference distance range may be identified based on an aperture value, a focal length of the at least one camera used, and a distance between a focused subject and the electronic device. For example, the first reference distance range may become wider as the f-number of the aperture increases. For another example, the first reference distance range may become narrower as the focal length of the camera increases. In other words, the higher the magnification of the camera, the narrower the depth of field may be. For another example, the greater the distance between a focused subject and the electronic device, the wider the depth of field may be.

According to one embodiment, in operation 805, the at least one processor 120 may store depth data and first reference distance information. According to one embodiment, the at least one processor 120 may obtain information or identify an area based on the stored depth data and first reference distance information. For example, the at least one processor 120 may obtain blur information of the improvement area based on the depth data. For example, the at least one processor 120 may identify an improvement area based on the depth data. For example, the at least one processor 120 may identify a focus area based on the depth data.

9 illustrates a flow of operations of an electronic device for performing digital bokeh while acquiring an image, according to embodiments.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 901 to 907 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1). there is.

Referring to FIG. 9 , according to one embodiment, in operation 901, at least one processor (eg, processor 120 of FIG. 1) may receive an image. The at least one processor 120 may receive an image based on a camera (eg, the camera module 180 of FIG. 1).

According to one embodiment, in operation 903, the at least one processor 120 may generate a depth map. The depth map may include depth data corresponding to a plurality of objects in the image. The depth data may be the distance from the subject corresponding to the object to the electronic device 101 including the camera 180. According to one embodiment, the depth data may be acquired through a depth sensor. The depth sensor may be a time of flight (TOF) sensor. According to one embodiment, the depth data may be obtained based on images acquired from a plurality of cameras. The at least one processor 120 may acquire a first image from a first camera. The at least one processor 120 may acquire a second image from a second camera. The at least one processor 120 may compare the first image and the second image to obtain depth data corresponding to a plurality of objects. The closer the subject is to the electronic device 101, the greater the difference between the positions of the object corresponding to the subject in the first image and the object corresponding to the subject in the second image compared to the subject farther from the electronic device 101. Because it is. For example, the depth data may be identified based on images acquired through a single camera. The at least one processor 120 may obtain depth data from the image through the convolution neural network (CNN). However, embodiments of the present disclosure are not limited thereto.

According to one embodiment, in operation 905, the at least one processor 120 may apply a bokeh effect based on the acquired image. According to one embodiment, the at least one processor 120 may receive a user input for identifying an object that will not be blurred among a plurality of objects. According to one embodiment, the at least one processor 120 may identify an improvement area based on the object and the depth map corresponding to the user input. According to one embodiment, the at least one processor 120 may identify a focus area based on the object and the depth map corresponding to the user input. The improvement area may be a portion of the image on which image processing is performed to increase resolution. The improvement area may be the same as the focus area. However, it is not limited to this. According to one embodiment, the at least one processor 120 may identify a focus area based on the object and the depth map corresponding to the user input. According to one embodiment, the at least one processor 120 may determine whether blur information in the improvement area is greater than or equal to a reference value. For example, the at least one processor 120 may identify blur information based on edge characteristics of pixels included in the image. For example, the at least one processor 120 may identify blur information based on the distance of the object. The at least one processor 120 identifies the blur information in the improvement area as being less than a reference value when the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range. You can. When the distance of the object corresponding to the user input from the electronic device 101 is outside the first reference distance range, the at least one processor 120 identifies the blur information in the improvement area as greater than or equal to a reference value. You can. According to one embodiment, the at least one processor 120 may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. According to one embodiment, the at least one processor 120 may perform blur processing on a portion of the image excluding the focus area. The at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area.

According to one embodiment, in operation 907, the at least one processor 120 may output an output image and store the image. The at least one processor 120 may output the output image to a display. The digital bokeh effect can also be applied to frames included in an image. In the case of video, the at least one processor 120 may store the depth of the focused subject and the depth of field for each frame.

FIG. 10 illustrates a flow of operations of an electronic device for performing digital bokeh after acquiring an image, according to embodiments.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 1001 to 1007 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1). there is.

Referring to FIG. 10, according to one embodiment, in operation 1001, the at least one processor 120 may acquire an image. The at least one processor 120 may acquire an image based on a camera (eg, the camera module 180 of FIG. 1).

According to one embodiment, in operation 1003, the at least one processor 120 may identify depth data based on a depth map. The depth map may include depth data corresponding to a plurality of objects in the image. The depth data may be the distance from the subject corresponding to the object to the electronic device 101 including the camera 180. The at least one processor 120 may identify depth data of a subject corresponding to an object based on the stored depth map.

According to one embodiment, in operation 1005, the at least one processor 120 may apply a bokeh effect based on the acquired image.

According to one embodiment, the at least one processor 120 may receive a user input for identifying an object that will not be blurred among a plurality of objects. According to one embodiment, the at least one processor 120 may identify an improvement area based on the object and the depth map corresponding to the user input. According to one embodiment, the at least one processor 120 may identify a focus area based on the object and the depth map corresponding to the user input. According to one embodiment, the at least one processor 120 may determine whether blur information in the improvement area is greater than or equal to a reference value. According to one embodiment, the at least one processor 120 may perform image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. According to one embodiment, the at least one processor 120 may perform blur processing on a portion of the image excluding the focus area. The at least one processor 120 may generate an output image by combining the focus area with a portion of the image excluding the focus area.

According to one embodiment, in operation 1007, the at least one processor 120 may output an output image and store the image. The at least one processor 120 may output an output image and store the image. The at least one processor 120 may output the output image to a display. The digital bokeh effect can also be applied to frames included in an image.

According to one embodiment, operation 905 of FIG. 9 for applying a digital bokeh effect to an image may be performed during image acquisition before the image is stored in memory. According to another embodiment, operation 1005 of FIG. 10 for applying a digital bokeh effect to an image may be executed after the image is stored in memory. Accordingly, embodiments according to the present disclosure can be applied both before and after an image is acquired.

As described above, according to one embodiment, the electronic device 101 may include at least one processor 120 and at least one camera 180. The at least one processor 120 may acquire images 301, 401, 413, and 501 through the at least one camera 180. The at least one processor 120 may generate a depth map 509 including depth data corresponding to a plurality of objects. The at least one processor 120 may receive a user input for identifying an object that will not be blurred among the plurality of objects. The at least one processor 120 may identify an improvement area based on the object and the depth map 509 corresponding to the user input. The at least one processor 120 may identify a focus area based on the object and the depth map 509 corresponding to the user input. The at least one processor 120 may identify whether blur information in the improvement area is greater than or equal to a reference value. The at least one processor 120 may perform image processing to reduce the blur information in the improvement area based on the blur information that is greater than or equal to the reference value. The at least one processor 120 may perform blur processing on portions of the image excluding the focus area. The at least one processor 120 may generate an output image 309 based on a synthesis of the focus area and a portion of the image excluding the focus area.

According to one embodiment, in order to perform the image processing to reduce the blur information for the improvement area, the image processing may be performed to increase resolution for the improvement area through a neural processing unit (NPU). there is.

According to one embodiment, in order to identify whether the blur information in the enhancement area is greater than or equal to the reference value, the at least one processor 120, based on edge characteristics of pixels included in the image, Blur information can be identified. The edge characteristics may include the strength of the edge and the consistency of the edge direction. The strength of the edge may be identified based on the size of the gradient in a first direction of the pixel and surrounding pixels included in the image and the size of the gradient in a second direction perpendicular to the first direction. The consistency in the edge direction may be identified according to the distribution of the gradient in the first direction of the pixel and the surrounding pixels and the distribution of the gradient in the second direction perpendicular to the first direction.

According to one embodiment, in order to identify whether the blur information in the improvement area is greater than or equal to the reference value, the at least one processor 120 selects the object corresponding to the user input from the electronic device 101. It is possible to identify whether the distance is within the first reference distance range. The at least one processor 120 determines whether the distance of the object corresponding to the user input from the electronic device is the first reference distance in order to identify whether the blur information of the improvement area is greater than or equal to the reference value. If it is within the range, the blur information in the improvement area can be identified as being less than the reference value. The at least one processor 120 determines whether the distance of the object corresponding to the user input from the electronic device is within the first reference distance range in order to identify whether the blur information of the improvement area is greater than or equal to the reference value. In other cases, the blur information in the improvement area can be identified as being equal to or higher than the reference value. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The first reference distance range may be identified based on the aperture value, the focal length of the at least one camera 180 used, and the distance between the focused subject and the electronic device.

According to one embodiment, to identify the focus area based on the received user input and the depth map 509, the at least one processor 120 selects the object corresponding to the received user input. distance can be identified. The at least one processor 120 may identify the focus object where the difference between the distance of the focus object and the object is less than a first designated distance. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The distance of the focus object may be the distance from the subject corresponding to the focus object to the electronic device 101. The focus area may include the object and the focus object.

According to one embodiment, in order to generate the depth map 509 including the depth data corresponding to the plurality of objects, the at least one processor 120 uses a first signal transmitted through a depth sensor to A second signal reflected by one of the plurality of objects may be received through the depth sensor. The depth data corresponding to one object among the plurality of objects may be obtained based on a time interval between transmission of the first signal and reception of the second signal.

According to one embodiment, in order to generate the depth map 509 including the depth data corresponding to the plurality of objects, the at least one processor 120 uses the first camera 180 to Images can be obtained. The at least one processor 120 may acquire a second image through the second camera 180. The at least one processor 120 may obtain the depth data corresponding to the plurality of objects by comparing the first image and the second image.

According to one embodiment, in order to perform the blur processing on the portion of the image excluding the focus area, the at least one processor 120 blurs the first subject corresponding to the first image portion with a first intensity. It can be handled. The at least one processor 120 may blur the second subject corresponding to the second image portion at a second intensity. The first subject may be separated from the electronic device by a first distance. The second subject may be separated from the electronic device by a second distance. The first distance may be greater than the second distance. The first intensity may be greater than the second intensity.

According to one embodiment, the at least one processor 120, while acquiring the image through the at least one camera 180, the depth map including the depth data corresponding to the plurality of objects ( 509) can be additionally created. The at least one processor 120 may additionally identify whether the distance of the object is outside the second reference distance range. The at least one processor 120 may additionally display a notification indicating difficulty in applying the bokeh effect based on identifying whether the distance of the object is outside the second reference distance range. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

According to one embodiment, the at least one processor 120 may additionally acquire the image through the at least one camera 180. The at least one processor 120 may additionally identify the distance of the object in the image acquired through the at least one camera 180. The at least one processor 120 may additionally display a notification based on identifying whether the distance of the object is outside the first reference distance range and within the second reference distance range. The notification may indicate that image processing is necessary to improve resolution when applying the digital bokeh effect. The at least one processor 120 may additionally display a notification indicating difficulty in applying the digital bokeh effect based on identifying whether the distance of the object is outside the second reference distance range. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

As described above, according to one embodiment, a method performed by an electronic device may include an operation of acquiring an image through at least one camera 180. The method may include generating a depth map 509 including depth data corresponding to a plurality of objects. The method may include receiving a user input to identify an object that will not be blurred among the plurality of objects. The method may include identifying an improvement area based on the depth map 509 and the object corresponding to a user input. The method may include identifying a focus area based on the object and the depth map 509 corresponding to the user input. The method may include an operation of identifying whether blur information in the improvement area is greater than or equal to a reference value. The method may include performing image processing to reduce the blur information for the improvement area based on the blur information that is greater than or equal to the reference value. The method may include performing blur processing on a portion of the image excluding the focus area. The method may include generating an output image 309 based on a synthesis of the focus area and a portion of the image excluding the focus area.

According to one embodiment, the operation of performing the image processing to reduce the blur information with respect to the improvement area includes performing the image processing to increase resolution with respect to the improvement area through a neural processing unit (NPU). Can include actions.

According to one embodiment, the operation of identifying whether the blur information in the improvement area is greater than or equal to the reference value may include identifying the blur information based on edge characteristics of pixels included in the image. You can. The edge characteristics may include the strength of the edge and the consistency of the edge direction. The strength of the edge may be identified based on the size of the gradient in a first direction of the pixel and surrounding pixels included in the image and the size of the gradient in a second direction perpendicular to the first direction. The consistency of the edge direction may be identified according to the distribution of the gradient of the pixel and the surrounding pixels in the first direction and the distribution of the gradient in the second direction perpendicular to the first direction.

According to one embodiment, the operation of identifying whether the blur information in the improvement area is greater than or equal to the reference value is performed when the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range. It may include an operation to identify whether it is recognized or not. The operation of identifying whether the blur information of the improvement area is greater than or equal to the reference value includes, when the distance of the object corresponding to the user input from the electronic device is within the first reference distance range, the improvement area of the improvement area. An operation of identifying the blur information as less than a reference value may be included. The operation of identifying whether the blur information of the improvement area is greater than or equal to the reference value includes, when the distance of the object corresponding to the user input from the electronic device is outside the first reference distance range, the blur information of the improvement area It may include an operation to identify as being greater than or equal to a reference value. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The first reference distance range may be identified based on the aperture value, the focal length of the at least one camera 180 used, and the distance between the focused subject and the electronic device.

According to one embodiment, the operation of identifying the focus area based on the received user input and the depth map 509 may include identifying the distance of the object corresponding to the received user input. You can. The operation of identifying the focus area based on the received user input and the depth map 509 includes identifying the focus object where the difference between the distance of the focus object and the distance of the object is less than a first specified distance. can do. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The distance of the focus object may be the distance from the subject corresponding to the focus object to the electronic device 101. The focus area may include the object and the focus object.

According to one embodiment, the operation of generating the depth map 509 including the depth data corresponding to the plurality of objects is performed when the first signal transmitted through the depth sensor is transmitted to one of the plurality of objects. It may include receiving a second signal reflected by the depth sensor. The depth data corresponding to one object among the plurality of objects may be obtained based on a time interval between transmission of the first signal and reception of the second signal.

According to one embodiment, the operation of generating the depth map 509 including the depth data corresponding to the plurality of objects may include acquiring a first image through the first camera 180. there is. The operation of generating the depth map 509 including the depth data corresponding to the plurality of objects may include acquiring a second image through the second camera 180. The operation of generating the depth map 509 including the depth data corresponding to the plurality of objects includes comparing the first image and the second image to generate the depth data corresponding to the plurality of objects. It may include acquisition operations.

According to one embodiment, the operation of performing the blur processing on a portion of the image excluding the focus area may include blurring a first subject corresponding to the first image portion with a first intensity. The operation of performing blur processing on a portion of the image excluding the focus area may include blurring a second subject corresponding to the second image portion at a second intensity. The first subject may be separated from the electronic device by a first distance. The second subject may be separated from the electronic device by a second distance. The first distance may be greater than the second distance. The first intensity may be greater than the second intensity.

According to one embodiment, the method includes generating the depth map 509 including the depth data corresponding to the plurality of objects while acquiring the image through the at least one camera 180. Additional information may be included. The method may additionally include an operation of identifying whether the distance of the object is outside the second reference distance range. The method may additionally include displaying a notification to indicate difficulty in applying the bokeh effect, based on identifying whether the distance of the object is outside the second reference distance range. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

According to one embodiment, the method may additionally include an operation of acquiring the image through the at least one camera 180. The method may additionally include an operation of identifying the distance of the object in the image acquired through the at least one camera 180. The method may additionally include displaying a notification based on identifying whether the distance of the object is outside the first reference distance range and within the second reference distance range. The notification may indicate that image processing is necessary to improve resolution when applying the bokeh effect. The method may additionally include a notification indicating difficulty in applying the bokeh effect based on identifying whether the distance of the object is outside the second reference distance range. The distance of the object may be the distance from the subject corresponding to the object to the electronic device 101. The second reference distance range may include the first reference distance range.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be "coupled" or "connected" to another (e.g. second) component, with or without the terms "functionally" or "communicatively". Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between cases where it is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. or one or more other operations may be added.

Claims (20)

In the electronic device 101,
at least one processor 120,
Comprising at least one camera 180,
The at least one processor 120,
Acquire images (301; 401; 413; 501) through the at least one camera (180),
Generate a depth map 509 containing depth data corresponding to a plurality of objects,
Receiving user input to identify an object that will not be blurred among the plurality of objects,
identify areas of improvement based on the object and the depth map (509) corresponding to the user input;
Identifying a focus area based on the object and the depth map (509) corresponding to the user input,
Identify whether blur information in the improvement area is greater than or equal to a reference value,
Based on the blur information greater than or equal to the reference value, image processing is performed to reduce the blur information for the improvement area,
Perform blur processing on portions of the image excluding the focus area,
configured to generate an output image (309) based on a synthesis of the focus area and a portion of the image excluding the focus area,
Electronic devices.
In claim 1,
To perform the image processing to reduce the blur information for the enhancement area, the at least one processor 120:
Configured to perform the image processing to increase resolution for the improvement area through a neural processing unit (NPU),
Electronic devices.
In claim 1,
In order to identify whether the blur information in the improvement area is greater than or equal to the reference value, the at least one processor 120,
Configured to identify the blur information based on edge characteristics of pixels included in the image,
The edge characteristics include the strength of the edge and the consistency of edge direction,
The strength of the edge is identified based on the magnitude of the gradient in a first direction and the magnitude of the gradient in a second direction perpendicular to the first direction of the pixel and surrounding pixels included in the image,
The consistency in the edge direction is identified according to the distribution of the gradient in the first direction of the pixel and the surrounding pixels and the distribution of the gradient in the second direction perpendicular to the first direction.
Electronic devices.
In claim 1,
In order to identify whether the blur information in the improvement area is greater than or equal to the reference value, the at least one processor 120,
configured to identify whether the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range,
When the distance of the object corresponding to the user input from the electronic device is within the first reference distance range, identify the blur information in the improvement area as less than a reference value,
configured to identify blur information of the improvement area as greater than or equal to a reference value when the distance of the object corresponding to the user input from the electronic device is outside the first reference distance range,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The first reference distance range is identified based on the aperture value, the focal length of the at least one camera 180 used, and the distance between the focused subject and the electronic device,
Electronic devices.
In claim 1,
To identify the focus area based on the received user input and the depth map 509, the at least one processor 120:
Identifying the distance of the object corresponding to the received user input,
configured to identify the focus object where the difference between the distance of the focus object and the distance of the object is less than a first specified distance,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The distance of the focus object is the distance from the subject corresponding to the focus object to the electronic device 101,
The focus area includes the object and the focus object,
Electronic devices.
In claim 1,
To generate the depth map 509 including the depth data corresponding to the plurality of objects, the at least one processor 120:
A first signal transmitted through a depth sensor receives a second signal reflected by one of the plurality of objects through the depth sensor,
The depth data corresponding to one object among the plurality of objects is obtained based on a time interval between transmission of the first signal and reception of the second signal,
Electronic devices.
In claim 1,
To generate the depth map 509 including the depth data corresponding to the plurality of objects, the at least one processor 120:
Obtaining a first image through the first camera 180,
Obtaining a second image through the second camera 180,
configured to obtain the depth data corresponding to the plurality of objects by comparing the first image and the second image,
Electronic devices.
In claim 1,
In order to perform the blur processing on a portion of the image excluding the focus area, the at least one processor 120,
The first subject corresponding to the first image portion is blurred with the first intensity,
A second subject corresponding to the second image portion is configured to be blurred with a second intensity,
The first subject is separated from the electronic device by a first distance,
The second subject is a second distance away from the electronic device,
The first distance is greater than the second distance,
wherein the first century is greater than the second century,
Electronic devices.
In claim 4,
The at least one processor 120,
While acquiring the image through the at least one camera (180), generate the depth map (509) including the depth data corresponding to the plurality of objects,
Identify whether the distance of the object is outside the second reference distance range,
Based on identifying whether the distance of the object is outside the second reference distance range, it is further configured to display a notification to indicate difficulty in applying the bokeh effect,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The second reference distance range includes the first reference distance range,
Electronic devices.
In claim 4,
The at least one processor 120,
Obtaining the image through the at least one camera 180,
Identifying the distance of the object in the image acquired through the at least one camera 180,
Additional configured to display a notification based on identifying whether the distance of the object is outside the first reference distance range and within the second reference distance range,
The above notification indicates that image processing is necessary to improve resolution when applying the digital bokeh effect,
Based on identifying whether the distance of the object is outside the second reference distance range, it is further configured to display a notification to indicate difficulty in applying the digital bokeh effect,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The second reference distance range includes the first reference distance range,
Electronic devices.
In a method performed by an electronic device,
An operation of acquiring an image through at least one camera 180,
An operation of generating a depth map 509 including depth data corresponding to a plurality of objects;
An operation of receiving a user input to identify an object that will not be blurred among the plurality of objects;
identifying an improvement area based on the object and the depth map (509) corresponding to the user input;
Identifying a focus area based on the object and the depth map 509 corresponding to the user input;
An operation of identifying whether blur information of the improvement area is greater than or equal to a reference value;
An operation of performing image processing to reduce the blur information for the improvement area based on the blur information greater than or equal to the reference value;
An operation of performing blur processing on a portion of the image excluding the focus area;
Comprising an operation of generating an output image (309) based on a synthesis of the focus area and a portion of the image excluding the focus area,
method.
In claim 11,
The operation of performing the image processing to reduce the blur information for the improvement area includes:
Comprising the operation of performing the image processing to increase resolution for the improvement area through a neural processing unit (NPU),
method.
In claim 11,
The operation of identifying whether the blur information in the improvement area is greater than or equal to the reference value includes:
An operation of identifying the blur information based on edge characteristics of pixels included in the image;
The edge characteristics include the strength of the edge and the consistency of edge direction,
The strength of the edge is identified based on the size of the gradient in a first direction and the size of the gradient in a second direction perpendicular to the first direction of the pixel and surrounding pixels included in the image,
The consistency of the edge direction is identified according to the distribution of the gradient in the first direction of the pixel and the surrounding pixels and the distribution of the gradient in the second direction perpendicular to the first direction,
method.
In claim 11,
The operation of identifying whether the blur information in the improvement area is greater than or equal to the reference value includes:
An operation of identifying whether the distance of the object corresponding to the user input from the electronic device 101 is within a first reference distance range;
When the distance of the object corresponding to the user input from the electronic device is within the first reference distance range, identifying the blur information in the improvement area as less than a reference value;
When the distance of an object corresponding to the user input from the electronic device is outside the first reference distance range, identifying blur information of the improvement area as greater than or equal to a reference value;
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The first reference distance range is identified based on the aperture value, the focal length of the at least one camera 180 used, and the distance between the focused subject and the electronic device,
method.
In claim 11,
The operation of identifying the focus area based on the received user input and the depth map 509 includes:
Identifying the distance of the object corresponding to the received user input;
Identifying the focus object where the difference between the distance of the focus object and the object is less than a first specified distance,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The distance of the focus object is the distance from the subject corresponding to the focus object to the electronic device 101,
The focus area includes the object and the focus object,
method.
In claim 11,
The operation of generating the depth map 509 including the depth data corresponding to the plurality of objects includes,
Comprising an operation of receiving, through the depth sensor, a second signal in which a first signal transmitted through the depth sensor is reflected by one of the plurality of objects,
The depth data corresponding to one object among the plurality of objects is obtained based on a time interval between transmission of the first signal and reception of the second signal,
method.
In claim 11,
The operation of generating the depth map 509 including the depth data corresponding to the plurality of objects includes,
An operation of acquiring a first image through the first camera 180,
An operation of acquiring a second image through the second camera 180,
Comprising the operation of obtaining the depth data corresponding to the plurality of objects by comparing the first image and the second image,
method.
In claim 11,
The operation of performing the blur processing on a portion of the image excluding the focus area includes:
An operation of blurring the first subject corresponding to the first image portion with a first intensity;
comprising blurring a second subject corresponding to the second image portion at a second intensity;
The first subject is separated from the electronic device by a first distance,
The second subject is a second distance away from the electronic device,
The first distance is greater than the second distance,
wherein the first century is greater than the second century,
method.
In claim 14,
Generating the depth map 509 including the depth data corresponding to the plurality of objects while acquiring the image through the at least one camera 180;
An operation of identifying whether the distance of the object is outside a second reference distance range;
Based on identifying whether the distance of the object is outside the second reference distance range, additionally comprising displaying a notification to indicate difficulty in applying the bokeh effect,
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The second reference distance range includes the first reference distance range,
method.
In claim 14,
An operation of acquiring the image through the at least one camera 180;
Identifying the distance of the object in the image acquired through the at least one camera 180;
Additionally comprising the operation of displaying a notification based on identifying whether the distance of the object is outside the first reference distance range and within the second reference distance range,
The above notification indicates that image processing is necessary to improve resolution when applying the bokeh effect,
Based on identifying whether the distance of the object is outside the second reference distance range, it additionally includes an operation of displaying a notification to indicate difficulty in applying the bokeh effect;
The distance of the object is the distance from the subject corresponding to the object to the electronic device 101,
The second reference distance range includes the first reference distance range,
method.

Images