WO2023058874A1

WO2023058874A1 - Method for providing image and electronic device supporting same

Info

Publication number: WO2023058874A1
Application number: PCT/KR2022/011594
Authority: WO
Inventors: 이석근
Original assignee: 삼성전자 주식회사
Priority date: 2021-10-08
Filing date: 2022-08-05
Publication date: 2023-04-13
Also published as: KR20230050617A

Abstract

According to various embodiments, an electronic device comprises a camera and a processor, wherein the processor may be configured to: obtain, by using the camera, a first image with respect to a first subject while in an optimal exposure state; obtain, by using the camera, a second image with respect to the first subject while in a short exposure state; identify a loss region of the first image on the basis of the first image and the second image, the loss region including a first boundary region and a first peripheral region located inside the first boundary region; modify the first boundary region on the basis of data of the second image; and modify the first peripheral region on the basis of data of the first image.

Description

Method for providing an image and electronic device supporting the same

Various embodiments relate to a method for providing an image and an electronic device supporting the same.

An electronic device such as a smart phone or a tablet PC may include a camera (or a camera module or camera device) and take pictures or videos.

There is an increasing user demand for an electronic device capable of providing a wider screen and a beautiful appearance while being miniaturized. In order to satisfy these user needs, a notch, a U-shaped hole, a V-shaped hole, or an O-shaped hole is formed in a part of a housing (or a part of a display), and the formed notch or hole An electronic device including a camera module that is exposed to the outside has been released.

Recently, in order to implement a full screen, an under display camera (UDC) technology in which a camera is disposed under a display is being implemented in an electronic device.

When the electronic device includes an under display camera (UDC), image quality may deteriorate due to the pattern characteristics of the display panel. For example, diffraction or scattering of light may occur due to a pattern of a display panel, and thus, some image loss may occur in a boundary area of a subject. In addition, when the electronic device captures the light source using an under display camera (UDC) or when the light source is photographed using a camera device other than the UDC, flare of light splitting may be lost. can

A method for providing an image and an electronic device supporting the same according to various embodiments provide loss information when the boundary between light and dark is saturated due to a light smearing phenomenon caused by diffraction or scattering of light in a backlight condition, resulting in loss of original information. A method for compensating for can be provided.

According to various embodiments, an electronic device may include a camera; and a processor, wherein the processor obtains a first image for a first subject in a proper exposure state using the camera, and acquires a first image for the first subject in a short exposure state using the camera. Acquiring a second image, and based on the first image and the second image, identifying a lost region of the first image, wherein the lost region is located inside a first boundary region and the first boundary region Including a first peripheral area that is configured to modify the first border area based on data of the second image, and modify the first peripheral area based on data of the first image. there is.

According to various embodiments, a method of operating an electronic device may include obtaining a first image of a first subject in an appropriate exposure state using a camera of the electronic device; obtaining a second image of the first subject in a single exposure state using the camera; An operation of identifying a loss area of the first image based on the first image and the second image—the loss area includes a first boundary area and a first peripheral area located inside the first boundary area. contains-; modifying the first boundary area based on data of the second image; and modifying the first peripheral area based on data of the first image.

By providing a method for providing an image and an electronic device supporting the same according to various embodiments, it is possible to provide a method for improving image quality by compensating for a loss region of an image using a plurality of images having different exposure values.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.

2 is a block diagram illustrating a camera module, in accordance with various embodiments.

3 is a diagram for describing an electronic device according to various embodiments.

4 is a cross-sectional view of a display and camera module according to various embodiments.

5 is a flowchart illustrating an operation of an electronic device according to various embodiments.

6 is a diagram for describing an operation of an electronic device according to various embodiments.

7 is a flowchart illustrating an operation of an electronic device according to various embodiments.

8 is a diagram for explaining an operation of an electronic device according to various embodiments.

9 is a diagram for explaining an operation of an electronic device according to various embodiments.

10 is a flowchart illustrating an operation of an electronic device according to various embodiments.

11 is a diagram for describing an operation of an electronic device according to various embodiments.

Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize Peak data rate (eg, 20 Gbps or more) for realizing 1eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram illustrating a camera module, in accordance with various embodiments. Referring to FIG. 2 , the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (eg, a buffer memory), or an image signal processor. (260). The lens assembly 210 may collect light emitted from a subject that is an image capturing target. The lens assembly 210 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 210 . In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may have the same lens properties as other lens assemblies. may have one or more lens properties different from the lens properties of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to one embodiment, the image sensor 230 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 240 moves at least one lens or image sensor 230 included in the lens assembly 210 in a specific direction in response to movement of the camera module 180 or the electronic device 101 including the same. Operation characteristics of the image sensor 230 may be controlled (eg, read-out timing is adjusted, etc.). This makes it possible to compensate at least part of the negative effect of the movement on the image being taken. According to an embodiment, the image stabilizer 240 may include a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180. Such a movement of the camera module 180 or the electronic device 101 may be detected using . According to one embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer. The memory 250 may at least temporarily store at least a portion of an image acquired through the image sensor 230 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 250 and , a copy image (eg, a low resolution image) corresponding thereto may be previewed through the display device display module 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 250 may be obtained and processed by the image signal processor 260 , for example. According to one embodiment, the memory 250 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 260 may perform one or more image processes on an image obtained through the image sensor 230 or an image stored in the memory 250 . The one or more image processes, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 260 may include at least one of the components included in the camera module 180 (eg, an image sensor). 230) may be controlled (eg, exposure time control, read-out timing control, etc.) The image processed by the image signal processor 260 is stored again in the memory 250 for further processing. or may be provided as an external component of the camera module 180 (eg, the memory 130, the display module 160, the electronic device 102, the electronic device 104, or the server 108). According to an example, the image signal processor 260 may be configured as at least a part of the processor 120 or may be configured as a separate processor that operates independently of the processor 120. The image signal processor 260 may be configured as a processor 120 When configured as a separate processor, at least one image processed by the image signal processor 260 may be displayed through the display module 160 as it is or after additional image processing by the processor 120 .

According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.

Referring to FIG. 3 , according to various embodiments, an electronic device 301 (eg, the electronic device 101 of FIG. 1 ) includes a main body 310 and a display 320 (eg, the electronic device 101 of FIG. 1 ). The display module 160) and the camera module 330 (eg, the camera module 180 of FIG. 1 or 2) may be included. In FIG. 3, a case including one camera module is shown as an example, but is not limited thereto.

According to various embodiments, the body part (or housing) 310 may include various components necessary for the operation of the electronic device 301 . For example, the body portion 310 may include a substrate (e.g., a printed circuit board (PCB), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB)) therein, a processor (e.g., the processor 120 of FIG. 1) , memory (eg, memory 130 of FIG. 1) or communication module (eg, communication module 190 of FIG. 1).

According to various embodiments, the display 320 may be disposed on a first surface (eg, the front surface) of the body unit 310, and the camera module 330 may be disposed facing the first surface. For example, the camera module 330 may not be visually exposed and may be an under display camera (UDC).

In FIG. 3, a form in which the camera module 330 is disposed facing the first surface (eg, the front surface, where the display 320 is mainly disposed) of the main body unit 310 is shown as an example, but is limited thereto. it is not going to be For example, when the display 320 extends to the second surface (eg, rear) of the main body 310, the camera module 330 faces the second surface (eg, rear) of the main body 310. can be arranged to do so.

According to various embodiments, the display 320 may display various contents such as text or images. The display 320 may be composed of a plurality of layers. For example, the display 320 may have a structure in which a window layer, a touch screen panel, a display panel, and/or a protective layer are sequentially stacked (see FIG. 4 ).

According to various embodiments, the display 320 may pass external light through at least a partial region where the camera module 330 is disposed. For example, the display 320 may pass external light through an empty space between pixels included in the display 320 . The camera module 330 may capture an image using external light introduced through the display 320 .

According to various embodiments, the camera module 330 may be mounted in an area where at least some layers included in the display 320 are removed. For example, a layer (eg, a shielding layer) through which external light cannot pass may be removed, and a lens unit (eg, the lens unit 331 of FIG. 4 ) of the camera module 330 is disposed in the removed area. It can be.

According to various embodiments, a metal layer having a pattern (hereinafter, a pattern layer) may be disposed on the front surface of the lens unit (eg, the lens unit 331 of FIG. 4 ) of the camera module 330 . For example, the pattern layer (eg, the pattern layer 410 of FIG. 4 ) may be one layer constituting the display 320 and may be disposed between a pixel of the display panel and a lens unit of the camera module.

4 is a cross-sectional view of a display and camera module according to various embodiments. The camera module of FIG. 4 may be the camera module 330 of FIG. 3 . The display of FIG. 4 may be the display 320 of FIG. 3 . Figure 4 is illustrative and not limited thereto.

Referring to FIG. 4 , a display (eg, the display 320 of FIG. 3 ) may include a window layer 401, a display panel 403, and a protective layer (or shielding layer, back cover) 480. .

According to various embodiments, the window layer (eg, ultra thin glass (UTG)) 401 may include a polymer. In this case, the window layer 401 may include polyethylene terephthalate (PET) or polyimide (PI). In various embodiments, a plurality of window layers 401 may be disposed.

Although not shown in FIG. 4 , the display 320 may further include a touch panel (touch sensor) between the window layer 401 and the display panel 403 .

According to various embodiments, the display 320 may include a control circuit (not shown). For example, the control circuit (not shown) may include a display driver IC (DDI) and/or a touch display driver IC (TDDI) disposed in a chip on panel (COP) or chip on film (COF) method.

According to various embodiments, the electronic device 301 may include a plurality of displays (eg, first and second displays), and at least one of the plurality of displays may include a flexible characteristic. there is. For example, in the electronic device 301, the first display (eg, display 320) may include an on cell touch AMOLED (OCTA) display, and the second display (eg, flexible display) may include a UB (eg, flexible display). It may include an unbreakable) type active matrix organic light-emitting diode (OLED) display.

According to various embodiments, the display panel 403 includes a base layer 405, a pattern layer 410, a wiring layer 420, a light emitting layer (or organic layer) 430, and an encapsulation layer (or protective layer) 440. can include Although not shown in FIG. 2 , the display panel 403 may further include a polarizer (eg, a polarizer film), an adhesive layer, and a touch panel. For example, the adhesive layer, as an adhesive member (eg, OCA (optical clear adhesive), PSA (pressure sensitive adhesive)), may be disposed between the respective layers.

According to various embodiments, a base layer 405 may be formed in a downward direction of the pattern layer 410 (eg, a direction toward the camera module 330). For example, the wiring layer 420 and the light emitting layer 430 may be stacked on the base layer 405 . According to various embodiments, the base layer 405 may include a transparent insulating substrate (eg, a substrate). For example, the base layer 405 may be formed of a glass substrate, a quartz substrate, or a transparent resin substrate. For example, the transparent resin substrate may be a polyimide-based resin, an acryl-based resin, a polyacrylate-based resin, a polycarbonate-based resin, or a polyether-based resin. -based) resin, sulfonic acid-based resin, and/or polyethyleneterephthalate-based resin may be included.

According to various embodiments, the pattern layer (or bottom metal layer (BML)) 410 is formed in an area where the protection layer (or shielding layer, back cover) 480 is at least partially removed for the arrangement of the camera module 330. patterns can be formed. The pattern layer 410 may include a blocking portion (or blocking area) 415 and an opening (or open area) 416 . The blocking portion 415 may be a region corresponding at least partially to the pixels 431 of the light emitting layer (organic material layer) 430, and the opening 416 is a panel opening between the pixels 431 of the light emitting layer (organic material layer) 430. It may be at least a part of the corresponding region. According to an embodiment, the pattern layer 410 may be a metal material and may be formed below the wiring layer 420 by deposition and/or patterning. The pattern layer 410 may protect the pixels 431 of the light emitting layer (organic layer) 430 and block light generated from the pixels 431 . According to various embodiments, the pattern layer 410 may include a designated pattern (black matrix) for reducing diffraction of light introduced into the camera module 330 or an opaque metal layer (eg, a buffer layer) including designated patterns. can

According to various embodiments, external light passing through the opening 416 may be introduced into the lens unit 331 . For example, light may be diffracted or scattered depending on the shape or size of the opening 416 .

According to various embodiments, the wiring layer 420 and the light emitting layer 430 may be formed by depositing a light emitting element (eg, organic electro luminescence (EL)) on a thin film transistor (TFT) substrate. According to various embodiments, the light emitting layer 430 may include a plurality of pixels 431 in which a plurality of subpixels (eg, red, green, and blue) constitute one pixel. According to various embodiments, the display panel 403 may include an active area (eg, a display area) and an inactive area (eg, a non-display area). For example, the active area may be an area corresponding to the area where the plurality of pixels 431 are disposed, and the inactive area is disposed outside the active area and corresponds to the bezel area of the display panel 403. can be an area.

According to various embodiments, the wiring layer 420 may include a TFT element, metal wiring, or an insulating film for driving each pixel of the active region. According to various embodiments, the wiring layer 420 may include liquid crystal polymer (LCP) or low temperature polycrystalline silicon (LTPS) glass, and the plurality of pixels may include thin film transistors (TFTs) formed on the LTPS glass. can include

According to various embodiments, the light emitting layer 430 may include a light emitting element (eg, organic electro luminescence (EL)). Organic EL can generate light when holes and electrons are injected from the cathode and anode.

According to various embodiments, when viewing the display panel 403 from the first side (eg, the front side), at least one component included in the electronic device 301 (eg, the camera module 330 or the sensor module) (eg, the sensor module 176 of FIG. 1 ) and at least a portion of the overlapping area may not have a plurality of pixels, or may include a plurality of pixels having a lower arrangement density than the non-overlapping area.

According to various embodiments, the encapsulation layer 440 (eg, TFE (thin film encapsulation)) may be a layer that protects the light emitting element from oxygen or moisture by alternately covering organic and inorganic layers on the light emitting layer 430. For example, The encapsulation layer 440 may be a pixel protection layer for protecting the plurality of pixels 431. For example, the encapsulation layer 440 may include encapsulation glass.

According to various embodiments, the protective layer (or shielding layer) 480 may support and protect the display panel 403 . The protective layer 480 may block light or electromagnetic waves introduced from the display panel 403 from being introduced into the electronic device 301 . The protective layer 480 may include a black film and a metal (eg, copper) plate. For example, the protective layer 480 is disposed under the display panel 403 to provide a dark background for ensuring visibility of the display panel 403 and to act as a buffering member (eg, a cushion) for a buffering effect. can be formed For example, the protective layer 480 is opaque to remove air bubbles that may occur between the display panel 403 and lower attachments thereof and to block light generated from the display panel 403 or light incident from the outside. A metal layer (eg, a black layer including a rough pattern) and/or a buffer layer (eg, a sponge layer) disposed to mitigate impact may be included.

According to various embodiments, the protective layer 480 may include a heat dissipation member (eg, a graphite sheet) and/or a conductive member (eg, a metal plate) for heat dissipation. For example, the conductive member may help to reinforce the rigidity of the electronic device 301, shield surrounding noise, and may be used to dissipate heat emitted from surrounding heat dissipating components.

According to various embodiments, at least a portion of the protective layer 480 may be open, and the lens unit 331 may be disposed in the open area. A pattern of the pattern layer 410 may be formed in a region where the protective layer 480 is removed.

Referring to FIG. 4 , according to various embodiments, light emitted from a light source and passing through a portion (eg, a first portion 499) of a display 320 (eg, a display panel 403) may reach the lens unit 331 and be used for image capture. For example, the first part 499 of the display 320 (eg, the display panel 403) is a camera module 330 (eg, the lens unit 331) in the display panel 403. ) may mean a region corresponding to the disposed region. For example, the electronic device 301 may acquire an image using light emitted from a light source and introduced through the first portion 499 of the display panel 403 .

According to various embodiments, the electronic device 301 (eg, the processor 120 of the electronic device 101 of FIG. 1 or the image signal processor 260 of FIG. 2 ) is part of the display 320 (eg, the first portion 499) may be controlled. The electronic device 301 controlling one part (eg, the first part 499) of the display 320 means that one part (eg, the first part 499) of the display panel 403 can mean controlling. For example, controlling the first part 499 of the display 320 by the electronic device 301 means that the wiring layer 420 of the display panel 403 included in the first part 499, and/or This may mean controlling at least a portion of the light emitting layer 430 . For example, controlling the first part 499 of the display 320 by the electronic device 301 means that among a plurality of pixels (eg, pixels 431) of the electronic device 301, the first part This may mean controlling at least one pixel (eg, pixel 431) included in 499. For example, the electronic device 301 may display a screen on the first portion 499 of the display 320 by controlling the first portion 499 of the display 320. A state in which the screen is displayed on part 1 499 may be referred to as an active state (or an under display camera-on (UDC-On) state). For example, the electronic device 301 acquires an image using the camera module 330 in a state in which the first part 499 of the display 320 is activated (eg, UDC-On state). can For another example, the electronic device 301 may control the first portion 499 of the display 320 so that the screen is not displayed on the first portion 499 of the display 320, and the display 320 A state in which the screen is not displayed on the first part 499 of may be referred to as an inactive state (or an under display camera-off (UDC-Off) state). For example, the electronic device 301 acquires an image using the camera module 330 in a state in which the first part 499 of the display 320 is inactive (eg, UDC-Off state). can

5 is a flowchart illustrating an operation of an electronic device according to various embodiments. 5 will be described with reference to FIG. 6 . 6 is a diagram for describing an operation of an electronic device according to various embodiments.

Referring to FIG. 5 , in operation 501, an electronic device 101 (eg, the electronic device 301) (eg, the processor 120 of FIG. 1 or the processor 120 of FIG. 2 ) according to various embodiments. The image signal processor 260 uses a camera module (eg, 180) to generate a first image (eg, 633 of FIG. 6(a) ) as a target. 610 of FIG. 6 (a)) can be obtained. In operation 503, the electronic device 101 according to various embodiments uses a camera module (eg, 180) to obtain a first subject (eg, the same subject as the subject of operation 501). A second image (eg, 620 in (b) of FIG. 6 ) may be obtained for 643 in (b) of FIG. 6 ). For example, the electronic device 101 displays a plurality of images (eg, two images) targeting the same subject (eg, the same person) at predetermined time intervals (eg, consecutively). ), and the time interval at which the electronic device 101 acquires a plurality of images is not limited. For example, the electronic device 101 may target a first subject (eg, 633) at a first time point in a first exposure state (eg, an appropriate exposure state) to obtain a first image (eg, 633). 610) may be acquired, and a second exposure state (eg, short exposure state) is obtained for a first subject (eg, 643) that is the same subject at a second time point after a first time interval from the first time point. A second image (eg, 620) may be obtained from . The exposure state may refer to a state of adjusting an amount of light (eg, an exposure amount or an exposure value) used to acquire an image. For example, the exposure state may be determined according to the aperture of the aperture of the electronic device 101, the shutter speed, or the ISO sensitivity, and the factors determining the exposure state are not limited. In addition, in the electronic device 101, there is no limitation on a method of adjusting an exposure state for obtaining an image. A proper exposure state or an appropriate exposure image may refer to a state in which a bright area, a dark area, and an area of medium brightness of an image are harmoniously expressed by calculating an appropriate exposure amount (or exposure value) to obtain an image, or the image. . An exposure value of appropriate exposure may be set according to various implementation examples, and is not limited thereto. The short-exposure image or short-exposure state may refer to an image obtained with an exposure value smaller than an appropriate exposure value (eg, a short exposure time) or a state thereof. For example, a short exposure image may be expressed darker than a properly exposed image. A long-exposure image or a long-exposure state may refer to an image obtained with an exposure value greater than an appropriate exposure value (eg, a long-time exposure) or a state thereof. For example, a long exposure image may be displayed brighter than a properly exposed image. Exposure values of the appropriate exposure image, short exposure image, and long exposure image may be relative, the exposure values may be determined in various ways, and the range of exposure values is not limited. Hereinafter, for convenience of description, a method of improving the quality of an image using a properly exposed image and a short exposure image will be described, but this is exemplary, and the technique described below includes a plurality of images obtained by varying the exposure value. It can be understood that the dog image can be used to improve the image quality.

In operation 505, according to various embodiments, the electronic device 101 displays a first image (eg, 610 of FIG. 6 ) in a first exposure state (eg, an appropriate exposure state) and a second exposure state. The loss region (eg, FIG. 6 ) of the first image (eg, 610 of FIG. 6 ) based on the second image (eg, 620 of FIG. 6 ) of (eg, short exposure state) It is possible to check the first loss region 612 or the second loss region 615 of . The loss region may refer to a region in which a subject, which is an image target, is distorted by a light source. For example, when an image is acquired in a backlight state (eg, a light source (eg, a first light source 631 in FIG. 6 or a second light source) around a subject (eg, 633 in FIG. 6 ) 632) is present, when the light source and the subject are simultaneously acquired as one image (eg, 610 in FIG. 6), the boundary area between the subject and the background in the image by diffraction or scattering of light Due to the occurrence of light smearing or light splitting, the original information of the subject may be lost. For example, in the case of the electronic device 301 of FIG. 3 , a loss area may be formed based on at least light passing through the display 320 . For example, in FIG. 4 , light passing through the display panel 403 is diffracted or scattered according to the shape or size of the opening 416 of the pattern layer 410 , thereby forming a loss region. For another example, even when an under display camera (UDC) is not included as shown in FIG. 3 , a loss region may be formed by diffraction or scattering of light. For example, (a) and (b) of FIG. 6 show two

images

610 and 620 obtained for the same subject (eg, the same person) (eg, 633 and 643). . For example, the first image 610 in (a) of FIG. 6 means an image obtained in a proper exposure state, and the second image 620 in (b) in FIG. 6 is obtained in a short exposure state. image can mean. For example, in (a) of FIG. 6 , the first image 610 is a properly exposed image obtained in a state in which the first light source 631 and the second light source 632 exist around the subject 633. In (b) of FIG. 6 , the second image 620 is a single exposure obtained in a state in which the same first light source 641 and the same second light source 642 exist around the same subject 643. It can be an image. For example, referring to FIG. 6 , the electronic device 101 selects a first image (which is a properly exposed image) based on a first image 610 that is a properly exposed image and a second image 620 that is a short exposure image. 610) loss areas (eg, 612 and 615) can be identified. According to an embodiment, the loss region (eg, the first loss region 612 or the second loss region 615 of FIG. 6 ) is a boundary region (eg, the first boundary region of FIG. 6 ( 613), or the second boundary area 616), and/or a peripheral area (eg, the first peripheral area 614 or the second peripheral area 617 of FIG. 6). A boundary region (eg, the first boundary region 613 in FIG. 6 or the second boundary region) in the loss region (eg, the first loss region 612 or the second loss region 615 in FIG. 6 ) The region 616) may refer to a region forming a boundary between the subject and the background among the loss regions. The peripheral area (eg, the first peripheral area 614 in FIG. 6 or the second peripheral area) in the loss area (eg, the first loss area 612 or the second loss area 615 in FIG. 6 ) The area 617) may refer to an area located inside the boundary area (eg, in the direction of the person in the boundary area between the background and the subject) of the loss area. An embodiment in which the electronic device 101 checks the lost areas (eg, 612 and 615 of FIG. 6 ) of the image (eg, 610 of FIG. 6 ) will be described later with reference to FIG. 7 .

In operation 507, the electronic device 101, according to various embodiments, displays a first image (eg, 610 of FIG. 6 ) based on the data of the second image (eg, 620 of FIG. 6 ). Boundary regions (eg, 613 and 616 of FIG. 6 ) of the loss regions (eg, 612 and 615 of FIG. 6 ) of may be corrected. For example, the electronic device 101 may select a first image (eg, 620 of FIG. 6 ) in a boundary area (eg, 621 of FIG. 6 ) of a second image (eg, 620 of FIG. 610), the boundary regions (eg, 623 and 626 of FIG. 6) corresponding to the boundary regions (eg, 613 and 616 of FIG. 6) among the loss regions (eg, 612 and 615 of FIG. 6). Based on the data for , the boundary areas (eg, 613 and 616 of FIG. 6 ) of the first image (eg, 610 of FIG. 6 ) may be corrected. An embodiment of operation 507 will be described later with reference to FIG. 8 .

In operation 509, the electronic device 101, according to various embodiments, displays a first image (eg, 610 of FIG. 6 ) based on data of the first image (eg, 610 of FIG. 6 ). It is possible to correct peripheral areas (eg, 614 and 617 of FIG. 6 ) among loss areas (eg, 612 and 615 of FIG. 6 ) of . An embodiment of operation 509 will be described later with reference to FIGS. 8 and 9 .

According to another embodiment, the electronic device 101 may, based on data of images different from the first image (eg, 610 of FIG. 6 ) and the second image (eg, 620 of FIG. 6 ), It is possible to correct peripheral areas (eg, 614 and 617 in FIG. 6 ) of loss areas (eg, 612 and 615 in FIG. 6 ) of one image (eg, 610 in FIG. 6 ). , which will be described later in FIG. 11 .

7 is a flowchart illustrating an operation of an electronic device according to various embodiments. 7 will be described with reference to FIG. 6 .

According to an embodiment, the electronic device 101 generates a first image (eg, 620 of FIG. 6 ) based on location information of a boundary area (eg, 621 of FIG. 6 ) of the second image (eg, 620 of FIG. 6 ). For example, loss regions (eg, 612 and 615 of FIG. 6 ) of 610 of FIG. 6 may be identified.

Referring to FIG. 7 , in operation 701, an electronic device 101 (eg, the electronic device 301) (eg, the processor 120 of FIG. 1 or the processor 120 of FIG. 2 ) according to various embodiments. The image signal processor 260 includes a boundary area (eg, 621 of FIG. 6 ) corresponding to the first subject (eg, 643 of FIG. 6 ) of the second image (eg, 620 of FIG. 6 ). ) location information can be checked. For example, in (b) of FIG. 6 , the electronic device 101 displays a border area 621 (eg, an outline of the subject 643) between the subject 643 and the background in the second image 620. , a line connecting the head and shoulders of the target person in FIG. 6 (b)) can be confirmed. For example, the electronic device 101 may check location information (eg, coordinate information) of the boundary area 621 of the subject 643 in the second image 620 .

In operation 703, the electronic device 101, according to various embodiments, based on the location information of the boundary area (eg, 621 of FIG. 6 ) of the second image (eg, 620 of FIG. 6 ) , The boundary area (eg, 610 of FIG. 6 ) of the first image (eg, 610 of FIG. 6 ) corresponding to the boundary area (eg, 621 of FIG. 6 ) of the second image (eg, 620 of FIG. 6 ). , 611 of FIG. 6) (for example, as an outline of the subject 633 of FIG. 6, a line connecting the head and shoulders of the target person) may be checked. For example, the electronic device 101 may check the location information of the boundary area (eg, 611 in FIG. 6 ).

In operation 705 , the electronic device 101 according to various embodiments is a predetermined distance (eg, 611 of FIG. 6 ) from the boundary area (eg, 610 of FIG. 6 ) of the first image (eg, 610 of FIG. 6 ). For example, it is possible to check whether the region within the first distance) is saturated. Checking whether an area within a certain distance (eg, a first distance) from the border area (eg, 611 in FIG. 6 ) is saturated indicates location information of the border area (eg, 611 in FIG. 6 ). This may mean checking whether regions (eg, pixels) located around the boundary region (eg, 611 of FIG. 6 ) are saturated based on . Saturation may mean a state in which a pixel value of a corresponding pixel is out of a certain range because the boundary area between the subject (eg, 633) and the background is distorted by the light source. For example, the electronic device 101 may determine whether the corresponding pixel is saturated based on the pixel values (eg, R, G, and B values) of the corresponding pixel, and determine whether the corresponding pixel is saturated. There is no limit to the criteria and degree of saturation.

In operation 707, the electronic device 101, according to various embodiments, displays a loss area (eg, the first loss area 612 or the second loss area 612 of the first image (eg, 610 of FIG. 6 )). loss region 615). For example, in (a) of FIG. 6 , the electronic device 101 checks whether the boundary area 611 of the first image 610 is saturated, and thus the first light source 631 and the second light source 632 ), it is possible to check an area where the subject is distorted (eg, the first loss area 612 or the second loss area 615). For example, in the first image 610, the electronic device 101 includes a first loss region 612 including a first boundary region 613 and a first peripheral region 614, and a second boundary region. 616 and the second loss region 615 including the second peripheral region 617 can be identified. For example, in (a) of FIG. 6 , the first boundary area 613 and the second boundary area 616 are part of the boundary area 611, and a light source (eg, the first light source 631) and a region saturated by the second light source 632. The first peripheral area 614 is an area located inside the first boundary area 613 and may mean an area saturated by the first light source 631 . The second peripheral area 617 is an area located inside the second boundary area 616 and may mean an area saturated by the second light source 632 .

8 is a diagram for explaining an operation of an electronic device according to various embodiments. 8 will be described with reference to FIG. 9 .

Referring to FIG. 8 , according to various embodiments, an electronic device 101 (eg, the electronic device 301) (eg, the processor 120 of FIG. 1 or the image signal processor of FIG. 2 ( 260) combines the first image 810 in the first exposure state (eg, proper exposure state) and the second image 820 in the second exposure state (eg, short exposure state) (890). )can do. For example, the first image 810 may be an image acquired in a first exposure state (eg, a proper exposure state) of a first subject (eg, 812) at a first viewpoint, The second image 820 is displayed in a second exposure state (eg, short exposure state) for a first subject (eg, 822) that is the same subject at a second time point after a first time interval from the first time point. ) may be an image acquired from. For example, the electronic device 101 continuously obtains the first image 810 and the second image 820, and then synthesizes the first image 810 and the second image 820 (890). can For example, synthesizing 890 the first image 810 and the second image 820 by the electronic device 101 means that the first image 810 is modified using data of the second image 820. can mean doing For example, the electronic device 101 uses the data of the boundary region 821 of the second image 820 to determine the boundary region (eg, the boundary region 811) among the lost regions of the first image 810. saturated region) can be corrected. For example, as described above with reference to FIGS. 5 to 7 , the electronic device 101 detects a loss area of the first image 810 based on location information of the boundary area 821 of the second image 820. can confirm. Thereafter, the electronic device 101 may correct the area identified as the loss area among the boundary areas 811 of the first image 810 using data of the border area 821 of the second image 820. . For example, in FIG. 8 , a third image 830 in which the first image 810 and the second image 820 are synthesized (eg, a portion of the first image 810 is the second image 820). When the first area 840 of the image modified based on the data of ) is enlarged, the boundary area 841 between the subject 848 and the background (eg, the sky background) becomes the boundary of the second image 821. It can be confirmed that the area 821 has been modified based on the data. For example, the electronic device 101 uses data of the boundary area 821 of the second image 820, information on the area identified as the loss area among the boundary area 811 of the first image 810. However, this is only an example, and the electronic device 101 converts the area identified as the loss area among the boundary area 811 of the first image 810 to the border area 821 of the second image 820. There is no limit to the method of modifying using the data of .

Referring to FIG. 8 , the electronic device 101, according to various embodiments, based on data of the first image 810, displays a peripheral area (eg, 843 ) can be modified. For example, in the third image 830, a first portion (eg, a saturated region of the boundary region 811) of the first image 810 is modified based on data of the second image 820. and the second part of the first image 810 (for example, the boundary area 811 or the peripheral area 843 that is a saturated area located inside the 841) is based on the data of the first image 810. As the modified image, an image including the data for the remaining part of the first image 810 as it is may be an image. For example, the electronic device 101 provides data of a saturated area (eg, 842) of the background of the first image 810 and data of a non-saturated area (eg, 844) of the subject 848. Based on , the peripheral area 843 of the lost area may be corrected. The saturated region in the background may refer to a region (eg, a sky background region) saturated by a light source in the background included in the image. The non-saturated area of the subject may refer to an area of the subject included in the image that is not saturated by the light source, and the unsaturated area of the subject may be referred to as a “center area”. For example, the electronic device 101 is located outside the loss area of the subject (eg 812) of the image (eg 810) (eg, in the direction of the background in the boundary area between the background and the subject). One area of the background can be identified as a saturated area. For example, the electronic device 101 is located inside the loss area of the subject (eg 812) of the image (eg 810) (eg, in the direction of the subject in the boundary area between the background and the subject). An area of the subject (eg, an unsaturated area located around the peripheral area (eg, 843) identified as the loss area) may be identified as a central area of the subject. For example, the electronic device 101 may determine whether each area is saturated based on pixel values of pixels in the background area and the subject area. There are no limitations on the method, criteria, and location of determining the position of the saturated region in the background and the position of the central region in the subject.

For example, referring to FIG. 9 , the electronic device 101 may check a boundary area 904 between a subject and a background in an image 900, and correct an area identified as a loss area among the boundary areas 904. and has been described above. According to an embodiment, the electronic device 101 may check the peripheral area 903, which is a saturated area, among areas located inside the boundary area 904. According to an embodiment, the electronic device 101 may compensate for the peripheral area 903 of the loss area based on the data of the saturated area 901 of the background and the data of the central area 902 of the subject. For example, when the pixel value of the peripheral region 903 is equal to the pixel value of the saturated region 901, the electronic device 101 may compensate the peripheral region 903 with the pixel value of the central region 902. there is. For example, when the pixel value of the peripheral region 903 is the same as the pixel value of the central region 902, the electronic device 101 may not compensate (or correct) the peripheral region 903. . For example, the electronic device 101 determines the peripheral area 903 based on the relative relationship between the pixel values of the peripheral area 903, the pixel values of the saturated area 901, and the pixel values of the central area 902. The degree of compensation can be determined. For example, the electronic device 101 determines a difference between a first pixel value of the saturated region 901 and a second pixel value of the center region 902 (eg, the first difference value), and a saturated region ( 901) and a third pixel value of the peripheral area 903 (eg, a second difference value) may be checked, and based on the ratio of the first difference value and the second difference value , the degree of compensation for the peripheral area 903 can be determined. For example, if the first pixel value of the saturated area 901 is A, the second pixel value of the central area 902 is B, and the third pixel value of the peripheral area 903 is C, the electronic device 101 , (1-(A-C)/(A-B)), a compensation value for the peripheral area 903 may be determined.

10 is a flowchart illustrating an operation of an electronic device according to various embodiments. 10 will be described with reference to FIGS. 11 and 8 . 11 is a diagram for describing an operation of an electronic device according to various embodiments.

Referring to FIG. 10 , in operation 1001, an electronic device 101 (eg, the electronic device 301) (eg, the processor 120 of FIG. 1 or the processor 120 of FIG. 2 ) according to various embodiments. The image signal processor 260 may continuously capture a properly exposed image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) and a short exposure image (eg, 820 in FIG. 8 ). For example, a properly exposed image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) and a short exposure image (eg, 820 in FIG. 8 ) are the same subject (eg, 812 in FIG. 8 , 822 or 1112 of FIG. 11) may be an acquired image.

In operation 1003, the electronic device 101, according to various embodiments, displays a subject (eg, 812 of FIG. 8 or 810 of FIG. It is possible to check whether the boundary area (eg, 811 in FIG. 8 or 1111 in FIG. 11) of 1112 in FIG. 11 is saturated, and accordingly, an appropriate exposure image (eg, 810 in FIG. 8 or 11 1110) of the loss region can be confirmed. The embodiment in which the electronic device 101 checks the loss area has been described above with reference to FIG. 7 .

In operation 1005, the electronic device 101 according to various embodiments determines a properly exposed image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) based on the loss area. (eg, 810 of FIG. 8 or 1110 of FIG. 11 ) and a single exposure image (eg, 820 of FIG. 8 ) may be synthesized. For example, the electronic device 101 may, based on the data of the boundary area 821 of the short-exposure image (eg, 820 of FIG. A boundary region (eg, a saturated region of the boundary region 811 or 1111) of the loss region of 1110 of FIG. 11 may be corrected. An embodiment in which the electronic device 101 corrects the boundary area of the loss area has been described above with reference to FIG. 8 .

In operation 1007, the electronic device 101 according to various embodiments displays a properly exposed image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) and a short exposure image (eg, 820 in FIG. 8 ). Another image (eg, 1120 of FIG. 11) including the same subject (eg, 1122 of FIG. 11) as the subject (eg, 812, 822 of FIG. 8, or 1112 of FIG. ) can be checked for existence. For example, the electronic device 101 includes a memory 130 of the electronic device 101, a storage (or buffer) of a gallery application of the electronic device 101, and a server accessible by the electronic device 101 (eg For example, among the images stored in the server 108 of FIG. 1 , a proper exposure image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) and a short exposure image (eg, 820 in FIG. 8 ) Another image (eg, 1120 in FIG. 11 ) including the same subject (eg, 1122 in FIG. 11 ) as the included subject (eg, 812 , 822 in FIG. 8 , or 1112 in FIG. 11 ) You can check whether it exists or not. For example, the third image (1120 in FIG. 11 ) is included in a properly exposed image (eg, 810 in FIG. 8 or 1110 in FIG. 11 ) and a short exposure image (eg, 820 in FIG. 8 ). including the same subject (eg, 1122 in FIG. 11) as the subject (eg, 812, 822 in FIG. 8, or 1112 in FIG. 11), but with a properly exposed image (eg, 810 in FIG. 8, Alternatively, it may be an image obtained from a background different from the background of 1110 in FIG. 11 and the short-exposure image (eg, 820 in FIG. 8 ). Alternatively, for example, although not shown, the third image may be an image obtained in a third exposure state for the same subject as the subject included in the proper exposure image and the short exposure image. According to an embodiment, the electronic device 101 determines that another image (eg, 1120 of FIG. 11 ) including the same subject (eg, 1122 of FIG. 11 ) exists, based on 1009 Operation can be performed. According to an embodiment, the electronic device 101 determines that another image (eg, 1120 of FIG. 11 ) including the same subject (eg, 1122 of FIG. 11 ) does not exist based on , 1011 operation can be performed.

In operation 1009, according to various embodiments, the electronic device 101 generates a properly exposed image 1110 based on a third image (eg, 1120) including the same subject (eg, 1122). The loss area of can be compensated (1190). For example, referring to FIG. 11 , the electronic device 101 checks location information of a peripheral area 1143 among the lost areas of a properly exposed image 1110, and obtains a third image (corresponding to the checked location information). Data of the peripheral area 1123 of the image 1120 may be checked, and the peripheral area 1143 of the loss area of the properly exposed image 1110 may be corrected using the data of the peripheral area 1123 of the third image 1120. . For example, in FIG. 11 , when the first area 1140 of the compensated image 1130 is enlarged, the compensated image 1130, the boundary area 1141 of the subject 1148 and the background is a short exposure image ( For example, the image compensated based on the second image 820 of FIG. 8) and the peripheral region 1143, which is a saturated region inside the boundary region 1141, is compensated based on the third image 1120. can be

In operation 1011, according to various embodiments, the electronic device 101 compensates for a loss region of the properly exposed image 1110 by using information on images located around the loss region in the properly exposed image 1110. It can be done, and the embodiment of operation 1011 has been described above with reference to FIGS. 8 and 9 .

Those skilled in the art can understand that the various embodiments described in this specification can be organically applied to each other within the applicable range.

According to various embodiments, an electronic device (eg, the electronic device 101 or the electronic device 301) may include a camera (eg, the camera module 180 or the camera module 330); and a processor (eg, the processor 120), wherein the processor obtains a first image of a first subject in an appropriate exposure state using the camera, and using the camera, Obtaining a second image of the first subject in an exposure state, determining a loss area of the first image based on the first image and the second image—the loss area is a first boundary area and a first peripheral area located inside the first boundary area, based on data of the second image, modifying the first border area, and based on data of the first image, It can be set to correct the first peripheral area.

According to various embodiments, the processor determines location information of a border area corresponding to the first subject of the second image, and based on the location information of the border area of the second image, the first subject It can be set to identify the missing area of the image.

According to various embodiments, the processor determines a boundary area of the first image corresponding to the boundary area of the second image based on the location information of the boundary area of the second image, and The loss area may be determined by checking whether an area within a first distance from the boundary area of 1 image is saturated.

According to various embodiments, the processor may be configured to identify an area having a pixel value equal to or greater than a reference value among the areas within the first distance from the boundary area in the first image as the loss area.

According to various embodiments, the processor may be configured to correct the first boundary region by using data of a second boundary region of the second image corresponding to the first boundary region of the first image. .

According to various embodiments, the processor may obtain data of a first saturated area located outside the first boundary area in the first image and a first center located around the first peripheral area in the first image. It may be set to modify the first peripheral area using data of the area.

According to various embodiments, the processor may determine a difference between a first pixel value of the first saturated region and a second pixel value of the first central region, and a difference between the first pixel value of the first saturated region and the first pixel value of the first central region. The first peripheral area may be corrected based on a ratio of difference between third pixel values of the peripheral area.

According to various embodiments, the processor may check whether a third image including a subject corresponding to the first subject and different from the first image and the second image exists, and checking the third image. Based on this, the first peripheral area of the first image may be further set to be corrected using data of a third peripheral area of the third image corresponding to the first peripheral area of the first image. there is.

According to various embodiments of the present disclosure, the processor may determine whether an image different from the first image and the second image including a subject corresponding to the first subject does not exist, in the first image. Setting the first peripheral area to be corrected using data of a first saturated area located outside the first boundary area and data of a first central area located in the periphery of the first peripheral area in the first image It can be.

According to various embodiments, the electronic device may further include a display (eg, the display module 160 or the display 320), and the camera may be disposed under the display. The loss area may be formed based on at least light passing through the display.

According to various embodiments, a method of operating an electronic device (eg, the electronic device 101 or the electronic device 301) includes a camera (eg, a camera module 180 or a camera module (eg, the electronic device 101 or the electronic device 301)). 330)) to acquire a first image of a first subject in an appropriate exposure state; obtaining a second image of the first subject in a single exposure state using the camera; An operation of identifying a loss area of the first image based on the first image and the second image—the loss area includes a first boundary area and a first peripheral area located inside the first boundary area. contains-; modifying the first boundary area based on data of the second image; and modifying the first peripheral area based on data of the first image.

According to various embodiments, the method further includes an operation of checking location information of a boundary area corresponding to the first subject of the second image, and the operation of checking the loss area of the first image includes: and identifying the lost area of the first image based on the location information of the boundary area of the second image.

According to various embodiments, the method further includes an operation of identifying a boundary region of the first image corresponding to the boundary region of the second image, based on the location information of the boundary region of the second image. And, the operation of checking the loss area of the first image may include an operation of checking the loss area by checking whether an area within a first distance from the boundary area of the first image is saturated. there is.

According to various embodiments, the operation of identifying the loss area of the first image may include defining an area having a pixel value equal to or greater than a reference value among the areas within the first distance from the boundary area in the first image as the loss area. It may include a confirmation action.

According to various embodiments of the present disclosure, the modifying of the first boundary area may include the correction of the first border area by using data of a second border area of the second image corresponding to the first border area of the first image. It may include an operation to modify.

According to various embodiments, the modifying of the first peripheral area may include data of a first saturated area located outside the first boundary area in the first image and data of the first peripheral area in the first image. An operation of modifying the first peripheral area using data of the first central area located in the periphery may be included.

According to various embodiments, the modifying of the first peripheral area may include a difference between a first pixel value of the first saturated area and a second pixel value of the first central area, and the first pixel value of the first saturated area. An operation of modifying the first peripheral area based on a ratio of a difference between a pixel value of 1 and a third pixel value of the first peripheral area.

According to various embodiments, the method may include an operation of determining whether a third image including a subject corresponding to the first subject and different from the first image and the second image exists; and based on the identification of the third image, using data of a third peripheral area of the third image corresponding to the first peripheral area of the first image, the first peripheral area of the first image An operation of modifying the region may be further included.

According to various embodiments, the modifying of the first peripheral area based on the data of the first image may include an image different from the first image including the subject corresponding to the first subject and the second image. Based on confirming that it does not exist, data of a first saturated area located outside the first boundary area in the first image and a first center located around the first peripheral area in the first image An operation of modifying the first peripheral area using data of the area may be included.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store ^TM ) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a storage medium readable by a device such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims

In electronic devices,

camera; and

It includes a processor, the processor comprising:

Obtaining a first image of a first subject in an appropriate exposure state using the camera;

Obtaining a second image of the first subject in a single exposure state using the camera;

Based on the first image and the second image, a loss area of the first image is identified, and the loss area includes a first border area and a first peripheral area located inside the first border area. box-,

Modifying the first boundary area based on the data of the second image;

Based on the data of the first image, set to correct the first peripheral area,

electronic device.
According to claim 1,

the processor,

Checking location information of a boundary area corresponding to the first subject of the second image;

Based on the location information of the boundary area of the second image, it is set to identify the loss area of the first image,

electronic device.
According to claim 2,

the processor,

Based on the location information of the border area of the second image, identifying a border area of the first image corresponding to the border area of the second image;

Set to check the loss area by checking whether an area within a first distance from the boundary area of the first image is saturated,

electronic device.
According to claim 3,

the processor,

Among the areas within the first distance from the boundary area in the first image, an area having a pixel value equal to or greater than a reference value is set to be identified as the loss area.

electronic device.
According to claim 1,

the processor,

Using data of a second boundary area of the second image corresponding to the first boundary area of the first image, the first boundary area is set to be corrected.

electronic device.
According to claim 1,

the processor,

Using data of a first saturated region located outside the first boundary region in the first image and data of a first central region located in the periphery of the first peripheral region in the first image, the first which is set to modify the surrounding area,

electronic device.
According to claim 6,

the processor,

A difference between the first pixel value of the first saturated area and the second pixel value of the first central area, and the difference between the first pixel value of the first saturated area and the third pixel value of the first peripheral area Based on the ratio, set to modify the first peripheral area,

electronic device.
According to claim 1,

the processor,

Checking whether a third image including a subject corresponding to the first subject and different from the first image and the second image exists;

Based on the identification of the third image, the first peripheral area of the first image is determined using data of a third peripheral area of the third image corresponding to the first peripheral area of the first image. which is further set to modify

electronic device.
According to claim 8,

the processor,

Based on confirming that there are no images other than the first image and the second image including a subject corresponding to the first subject, a first image positioned outside the first boundary area in the first image Using data of 1 saturated area and data of a first central area located in the periphery of the first peripheral area in the first image, the first peripheral area is set to be corrected.

electronic device.
In the first,

The electronic device further includes a display, and the camera is disposed under the display,

The loss area is formed based at least on light passing through the display,

electronic device.
In the operating method of the electronic device,

obtaining a first image of a first subject in an appropriate exposure state using a camera of the electronic device;

obtaining a second image of the first subject in a single exposure state using the camera;

An operation of identifying a loss area of the first image based on the first image and the second image—the loss area includes a first boundary area and a first peripheral area located inside the first boundary area. contains-;

modifying the first boundary area based on data of the second image; and

Based on the data of the first image, modifying the first peripheral area,

method.
According to claim 11,

Further comprising an operation of checking location information of a boundary area corresponding to the first subject of the second image,

The operation of identifying the loss area of the first image includes an operation of identifying the loss area of the first image based on the location information of the boundary area of the second image.

method.
According to claim 12,

Further comprising an operation of identifying a boundary region of the first image corresponding to the boundary region of the second image based on the location information of the boundary region of the second image;

The operation of checking the loss area of the first image includes an operation of checking the loss area by checking whether an area within a first distance from the boundary area of the first image is saturated,

method.
According to claim 13,

The operation of identifying the loss area of the first image includes an operation of identifying, as the loss area, a region having a pixel value equal to or greater than a reference value among the regions within the first distance from the boundary region in the first image. ,

method.
According to claim 11,

The modifying of the first boundary region includes modifying the first boundary region using data of a second boundary region of the second image corresponding to the first boundary region of the first image. doing,

method.