KR20220005283A - Electronic device for image improvement and camera operation method of the electronic device - Google Patents

Abstract

An electronic device according to one embodiment of the present document includes a display, a first camera module including a first camera having a first viewing angle, a second camera module including a second camera having a second viewing angle different from the first viewing angle, and a processor operatively connected to the first camera module and the second camera module. The processor displays, through the display, a first preview screen which is acquired using the first camera module and displays a first image including a subject image, determines whether the first image satisfies a condition for determining whether image improvement is necessary, acquires a second image including the subject image by using the second camera module, matches a first object coordinate of the first image and a second object coordinate of the second image when determined that the condition is satisfied, and displays a second preview screen displaying a composite image acquired by synthesizing the first image and the second image through the display. According to various embodiments of the present disclosure, image improvement of a shadow generated when an image of a subject is captured and image improvement of a light reflection by a subject occurring when an image is taken may be performed using the camera of the electronic device. Also, by performing image improvement on a subject photographed by the electronic device, a user experience may be improved.

Description

Electronic device for image improvement and camera operation method of the electronic device

Various embodiments of the present disclosure relate to an electronic device that generates an improved image when capturing an image using a camera of the electronic device, and a method of operating the electronic device.

When an image is captured using an electronic device, it may be necessary to improve an image to be captured according to a property included in the configuration of the electronic device or a surrounding environment. For example, the electronic device may need to improve an angle of view and improve shading according to positions of light sources and objects.

According to the prior art, when capturing an image using an electronic device, capturing may be performed based on a plurality of attributes. For example, the electronic device may be equipped with a wide camera, an ultra-wide camera, a tele photo camera, a time of flight (TOF) camera, and the like. The electronic device may capture an image based on various angles of view or distance information by mounting the aforementioned cameras.

According to the prior art, when a shadow occurs when capturing an image, the electronic device may estimate or determine the shadow based on distance information about the object, and may perform image correction on the determined shadow. Also, the electronic device may determine whether an event in the image has occurred by using shooting information such as auto focusing.

When photographing using a camera of an electronic device, a problem arises that a shadow is formed on a subject, light reflection occurs due to a reflective property (eg, glass, metal) of the subject, or a shadow generated by a photographer is reflected on the subject. For example, due to the location of the subject and the photographer, shadows caused by forward light, backlight, metering, upper light, lower light, rear light, transfer light, etc. occur. In this case, the user has to compose a composition so as not to generate a shadow when taking an image, and while trying various compositions, the user cannot take a picture in a position suitable for taking an image while no shadow is generated.

In addition, when photos of glass, jewelry, metal products, or old films are taken using the camera of the electronic device, light by various chemicals such as a developer and a photoresist may be reflected, and an image desired by the user may not be captured.

An electronic device according to an embodiment of the present document includes a display, a first camera module including a first camera having a first viewing angle, and a second camera including a second camera having a second viewing angle different from the first viewing angle a module, and a processor operatively connected to the first camera module and the second camera module, wherein the processor is configured to display a first image obtained using the first camera module and including a subject image. 1 A preview screen is displayed through the display, it is determined whether the first image satisfies a condition for determining whether image improvement is required, and a second image including the subject image is captured using the second camera module. obtained, and when it is determined that the condition is satisfied, a composite image obtained by matching the first object coordinates of the first image and the second object coordinates of the second image, and synthesizing the first image and the second image and an electronic device that displays the displayed second preview screen through the display.

The method of operating an electronic device according to an embodiment of the present document includes: displaying a first preview screen through the display, which is acquired using the first camera module and displays a first image including a subject image; Determining whether the first image satisfies a condition for determining whether image improvement is necessary, obtaining a second image including the subject image by using the second camera module, and determining whether the condition is satisfied In this case, a second preview screen for displaying a composite image in which the first object coordinates of the first image and the second object coordinates of the second image are matched, and the first image and the second image are combined; It may include an operation of displaying through the display.

According to various embodiments of the present disclosure, image improvement may be performed on a shadow generated when an image of a subject is captured by using the camera of the electronic device.

According to various embodiments of the present disclosure, image improvement may be performed with respect to light reflection by a subject that occurs when an image is captured by using a camera of an electronic device.

According to various embodiments of the present disclosure, the user experience may be improved by improving the image of the subject photographed by the electronic device.

Effects that can be obtained based on various embodiments are not limited to the effects mentioned above, and other effects not mentioned are clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. it could be

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;
2 is a block diagram illustrating a camera module according to various embodiments of the present disclosure;
3 is a diagram illustrating a configuration of a part of an electronic device and timing for explaining an example in which the electronic device operates according to various embodiments of the present disclosure;
4 is a diagram illustrating an image before and after image enhancement is performed in an electronic device according to an exemplary embodiment.
5 is a diagram illustrating a configuration of an electronic device according to an embodiment.
6 is a flowchart illustrating a flow of performing image enhancement in an electronic device according to an exemplary embodiment.
7 is a flowchart illustrating a part of a detailed flow of performing image enhancement in an electronic device according to an exemplary embodiment.
8 is a diagram illustrating an embodiment of dividing a subject image area of a preview screen while image enhancement is being performed in an electronic device according to an exemplary embodiment.
9 is a diagram illustrating an embodiment of determining whether an electronic device satisfies a threshold value condition for determining whether image improvement is necessary while performing image improvement, according to an embodiment.
10 is a diagram illustrating an embodiment of obtaining distance values between a subject and a light source using a TOF camera while image enhancement is being performed in an electronic device, according to an embodiment.
11 is a diagram illustrating an embodiment of generating object coordinates and switching to a second camera while performing image enhancement in an electronic device, according to an embodiment.
12 is a flowchart illustrating a part of a detailed flow of performing image enhancement in an electronic device according to an exemplary embodiment.
13 is a diagram illustrating an embodiment in which image synthesis is performed by matching object coordinates in an electronic device according to an embodiment.
14 is a diagram illustrating an embodiment of generating a guide based on a subject for image improvement in an electronic device according to an embodiment.
15 is a diagram illustrating an embodiment of generating a shadow area for a subject image in an electronic device according to an embodiment.
16 is a diagram illustrating an example of generating a light reflection area for a subject image in an electronic device according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments of the present disclosure.

Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input device 150 , a sound output device 155 , a display device 160 , an audio module 170 , and a sensor module ( 176 , interface 177 , haptic module 179 , camera module 180 , power management module 188 , battery 189 , communication module 190 , subscriber identification module 196 , or antenna module 197 . ) may be included. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, the program 140 ) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 120 stores a command or data received from another component (eg, the sensor module 176 or the communication module 190 ) into the volatile memory 132 . may be loaded into the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the resulting data in the non-volatile memory 134 . According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and an auxiliary processor 123 (eg, a graphic processing unit or an image signal processor) that can be operated independently or together with the main processor 121 . , a sensor hub processor, or a communication processor). Additionally or alternatively, the auxiliary processor 123 may be configured to use less power than the main processor 121 or to be specialized for a specified function. The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as a part of another functionally related component (eg, camera module 180 or communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output a sound signal to the outside of the electronic device 101 . The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display device 160 may include a touch circuitry configured to sense a touch or a sensor circuit (eg, a pressure sensor) configured to measure the intensity of a force generated by the touch. have.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input device 150 , or an external electronic device (eg, a sound output device 155 ) connected directly or wirelessly with the electronic device 101 . The sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 388 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (eg, a cellular network, the Internet, or It can communicate with an external electronic device through a computer network (eg, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified and authenticated.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as a part of the antenna module 197 .

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the electronic devices 102 and 104 may be the same or a different type of device from the electronic device 101 . According to an embodiment, all or some of the operations performed by the electronic device 101 may be executed by one or more of the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

2 is a block diagram 200 illustrating a camera module 180, according to various embodiments.

Referring to FIG. 2 , the camera module 180 includes a lens assembly 210 , a flash 220 , an image sensor 230 , an image stabilizer 240 , a memory 250 (eg, a buffer memory), or an image signal processor. (260). The lens assembly 210 may collect light emitted from a subject, which is an image to be captured. The lens assembly 210 may include one or more lenses. According to an embodiment, the camera module 180 may include a plurality of lens assemblies 210 . In this case, the camera module 180 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly. It may have one or more lens properties different from the lens properties of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from the subject. According to an embodiment, the flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to an embodiment, the image sensor 230 may include, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

In response to the movement of the camera module 180 or the electronic device 101 including the same, the image stabilizer 240 moves at least one lens or the image sensor 230 included in the lens assembly 210 in a specific direction or Operation characteristics of the image sensor 230 may be controlled (eg, read-out timing may be adjusted, etc.). This makes it possible to compensate for at least some of the negative effects of the movement on the image being taken. According to an embodiment, the image stabilizer 240 is, according to an embodiment, the image stabilizer 240 is a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 . can be used to detect such a movement of the camera module 180 or the electronic device 101 . According to an embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer. The memory 250 may temporarily store at least a portion of the image acquired through the image sensor 230 for the next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high-resolution image) is stored in the memory 250 and , a copy image corresponding thereto (eg, a low-resolution image) may be previewed through the display device 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 250 may be obtained and processed by, for example, the image signal processor 260 . According to an embodiment, the memory 250 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 260 may perform one or more image processing on an image acquired through the image sensor 230 or an image stored in the memory 250 . The one or more image processes may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 260 may include at least one of the components included in the camera module 180 (eg, an image sensor). 230), for example, exposure time control, readout timing control, etc. The image processed by the image signal processor 260 is stored back in the memory 250 for further processing. or may be provided as an external component of the camera module 180 (eg, the memory 130, the display device 160, the electronic device 102, the electronic device 104, or the server 108). According to an example, the image signal processor 260 may be configured as at least a part of the processor 120 or as a separate processor operated independently of the processor 120. The image signal processor 260 may be configured as the processor 120 and a separate processor, the at least one image processed by the image signal processor 260 may be displayed through the display device 160 as it is by the processor 120 or after additional image processing.

According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least the other may be a rear camera.

3 is a diagram illustrating a timing for explaining a configuration 300 of a portion of the electronic device 301 and an example 390 in which the electronic device 301 operates according to various embodiments of the present disclosure.

In various embodiments, the electronic device 301 may correspond to the electronic device 101 of FIG. 1 .

Referring to FIG. 3 , the electronic device 301 may include a processor 310 , a memory 330 , a camera module 340 , a TOF camera 350 , and a display 360 .

According to various embodiments, the camera module 340 may capture still images and moving images. According to an embodiment, the camera module 340 may include one or more lenses, image sensors, image signal processors, or flashes. The camera module 340 may include at least a first camera module 342 and a second camera module 344 . In addition, the first camera module 342 and the second camera module 344 may perform the same or similar functions to the functions performed by the camera module 340 .

According to various embodiments, the TOF camera 350 may acquire distance information (or depth information) between the electronic device 301 and the object (eg, subject) 302 . The TOF camera 350 may include a time of flight (ToF) module. According to an embodiment, the TOF camera 350 may include a light emitting module 352 and a receiving module 354 , or a combination thereof. For example, the light emitting module 352 may include one or more vertical-cavity surface emitting lasers (VCSELs), light emitting diodes (eg, red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or It may include a xenon lamp. Also, the reception module 354 may be configured as an RGB sensor, a black and white (BW) sensor, an infrared sensor, an ultraviolet sensor, or a combination thereof. For example, the image sensor included in the reception module 354 may be implemented using a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or a combination thereof.

t)만큼의 차이가 존재할 수 있다.According to an embodiment, the TOF camera 350 transmits the irradiated light 371 of a specified frequency to the object 302 (eg, a subject, an external object) using the light emitting module 352 as shown in FIG. 3 . etc.) can be irradiated during the irradiation time Tp of the specified period Tc. For example, the TOF camera 350 may not irradiate the irradiation light 371 during the non-irradiation time Tnp of the specified period Tc. Also, the TOF camera 350 may receive the reflected light 375 from the object 302 for an interval corresponding to the irradiation time Tp, using the receiving module 354 . According to an embodiment, the irradiation time of the irradiation light 371 and the reception time of the reflected light 375 are time (

t) may exist.

According to an embodiment, the reception module 354 may include at least two capacitors T1 and T2. For example, the first capacitor T1 may be switched from the off state to the on state during the irradiation time Tp of the specified period Tc, and may be switched from the on state to the off state during the non-irradiation time Tnp. Also, the second capacitor T2 may be switched from the on state to the off state during the irradiation time Tp of the specified period Tc, and may be switched from the off state to the on state during the non-irradiation time Tnp. According to various embodiments, the at least two capacitors T1 and T2 may accumulate charge amounts Q1 and Q2 corresponding to the light amount of the incident reflected light 375 while they are in the on state. According to an embodiment, the charges accumulated in the two capacitors T1 and T2 may be charges generated in response to light received by the reception module.

t)를 나타내는 정보를 전달할 수 있다. 다양한 실시 예에 따르면, TOF 카메라(350)는 카메라 모듈(340)의 일부로 구성될 수 있다. 일 실시 예에 따르면, 카메라 모듈(340)과 TOF 카메라(350)는 하나의 센서, 예를 들어, 이미지 센서로 구성될 수 있다. 그러나, 이는 예시적일 뿐, 본 발명의 실시 예가 이에 한정되는 것이 아니다.According to an embodiment, the TOF camera 350 transmits information indicating the distance D between the electronic device 301 and the object 302 to the processor 310 or indicates the distance D from the processor 310 . time difference (

t) can be transmitted. According to various embodiments, the TOF camera 350 may be configured as a part of the camera module 340 . According to an embodiment, the camera module 340 and the TOF camera 350 may be configured as one sensor, for example, an image sensor. However, this is only an example, and the embodiment of the present invention is not limited thereto.

According to various embodiments, the processor 310 includes at least one other component (eg, the memory 330 , the camera module 340 , the TOF camera 350 , or the display of the electronic device 301 connected to the processor 310 ). (360)) can be controlled. Also, the processor 310 may correspond to the processor 120 of FIG. 1 .

According to an embodiment, the processor 310 activates the TOF camera 350 in response to an input to measure the distance to the object 302 , and communicates with the electronic device 301 using the activated TOF camera 350 . Distance information between the objects 302 may be acquired. For example, an input for measuring a distance to the object 302 may be received through a screen provided by an arbitrary application. For example, an input for measuring a distance to the object 302 may be an input for executing an arbitrary application (eg, a camera application). However, this is only an example, and the embodiment of the present invention is not limited thereto. For example, an input for measuring a distance to the object 302 may be received through at least one of a voice command, a button input, and a gesture input.

According to an embodiment, the processor 310 may calculate the distance D between the electronic device 301 and the object 302 based on Equation 1 below.

t는 조사광(371)의 조사 시점과 반사광(375)의 수신 시점 간의 시간 차이를 나타내고, Tp는 조사 시간이고, Q1은 조사 시간(Tp) 동안 온 상태로 전환된 커패시터(T1)가 축적한 전하량을 나타내고, Q2는 비조사 시간(Tnp) 동안 온 상태로 전환된 커패시터(T2)가 축적한 전하량을 나타낼 수 있다. In [Equation 1], c is a constant representing the speed of light (c = 3*108 m/s),

t represents the time difference between the irradiation time of the irradiation light 371 and the reception time of the reflected light 375, Tp is the irradiation time, and Q1 is the accumulation of the capacitor T1 switched on during the irradiation time Tp. represents the amount of charge, and Q2 may represent the amount of charge accumulated by the capacitor T2 switched to the on state during the non-irradiation time Tnp.

According to an embodiment, the processor 310 may provide a service based on distance information measured using the TOF camera 350 . For example, the service may be at least one function capable of distinguishing a background from an object in an image captured by a camera. For example, the processor 310 may distinguish at least a part (eg, a background) included in the image from another part based on the measured distance information, and a different effect (eg, blur effect) on the divided part and the other part. We can provide services that apply In addition, the processor 310 may classify at least one object included in the image based on the measured distance information, the distance between the electronic device and the object 302 , the size of the object 302 , and the object 302 . ) may provide a service for measuring at least one of information such as thickness (eg, AR Ruler service, map service, Mixed Reality service, etc.). However, this is only an example, and the embodiment of the present invention is not limited thereto. For example, various information based on distance information may be provided as part of a service.

According to an embodiment, in response to determining that the distance measured using the TOF camera 350 deviates from the threshold distance, the processor 310 uses the camera module 340 to connect the electronic device 301 and the object ( 302), it is possible to obtain distance information between them. The threshold distance may be a distance between the electronic device 301 and the object 302 at which an error in a distance measured using the TOF camera 350 may be guaranteed to be less than or equal to a certain level. For example, when the measured distance deviates from the threshold distance, the processor 310 may stop the operation of the TOF camera 350 and acquire distance information using the camera module 340 . For example, the processor 310 may stop the operation of the TOF camera 350 to prevent unnecessary power consumption, and use the camera module 340 that consumes relatively less power than the TOF camera 350 . A time point at which the operation of 350 is resumed (eg, the object 302 that exists within a threshold distance is checked) may be checked. However, this is only an example, and the embodiment of the present invention is not limited thereto. For example, the processor 310 may stop the operation of the TOF camera 350 for a specified time and temporarily operate after the specified time to confirm the timing of resuming the operation of the TOF camera 350 . According to an embodiment, the processor 310 may stop providing a service based on the distance information while acquiring the distance information using the camera module 340 . For example, the processor 310 may notify a message indicating that the operation of the TOF camera 350 is stopped.

According to an embodiment, when the distance measured using the camera module 340 is included in the threshold distance, the processor 310 may process the stopped operation of the TOF camera 350 to resume. For example, the processor 310 may measure distance information using the TOF camera 350 and may provide a service based on the measured distance information.

According to an embodiment, the processor 310 may change the measurement parameter while the TOF camera 350 is operating. The measurement parameters may include an irradiation frequency (eg, frequency) of the light emitting module 352 , an irradiation period (eg, exposure time), a frame rate, and the like.

For example, the processor 310 may change the measurement parameter based on the position of the object 302 . For example, the processor 310 may decrease the irradiation period and increase the irradiation frequency as the distance of the object 302 approaches the electronic device 301 . Also, the processor 310 may increase the irradiation period and decrease the irradiation frequency as the distance of the object 302 increases from the electronic device 301 . In addition, the processor 310 may reduce the frame rate to reduce power consumption while the TOF camera 350 is operating. As another example, the processor 310 may change the measurement parameter based on the photographing environment. The photographing environment may be related to the type of the object 302 . For example, when a person with relatively little movement is the object 302 , the processor 310 may obtain distance information from the object 302 using a preset reference parameter. Also, when the object 302 having relatively many movements is used, the processor 310 may change a measurement parameter to enlarge a distance measurement range, for example, a threshold distance. For example, the processor 310 may expand the range of the distance measurement by increasing the irradiation period and decreasing the irradiation frequency.

According to various embodiments, the memory 330 may include at least one data required for a distance from the object 302 . According to an embodiment, the memory may include data related to a measurement parameter and a threshold distance. As described above, the measurement parameter may include an irradiation frequency (eg, frequency) of the light emitting module, an irradiation period (eg, exposure time), a frame rate, and the like. However, this is only an example, and the embodiment of the present invention is not limited thereto. For example, the measurement parameter may include various parameters such as irradiation intensity of the light emitting module. For example, the measurement parameter may be defined according to a measurement environment. For example, a measurement parameter suitable for the object 302 having relatively few movements and a measurement parameter suitable for the object 302 having relatively many movements may be distinguished and stored. As another example, the measurement parameter may be defined according to a distance from the object 302 . In addition, the measurement parameters may include parameters that must be changed in order to enlarge the threshold distance.

According to various embodiments of the present disclosure, an electronic device (eg, electronic device 301 ) includes a camera module (eg, camera module 340 ) that acquires an image including at least one object, and a TOF camera (eg, TOF camera 350) that generates depth data for It may include at least one memory (eg, the memory 330 ) and a display (eg, the display 360 ).

According to an embodiment, the memory, when executed, causes the processor to, while the image is acquired using the camera module, based on the depth data generated using the TOF camera, the at least one A first distance between the object and the electronic device is calculated, and if the calculated first distance is greater than a specified distance, the TOF camera is stopped, and the TOF camera is used while the TOF camera is stopped. Instructions for calculating a second distance between at least one object and the electronic device and resuming the operation of the TOF camera when the calculated second distance is within the specified distance may be stored.

According to various embodiments, the instruction may calculate the second distance based on an auto-focus function of the camera module.

According to various embodiments, the instruction may provide notification information indicating that the calculated first distance is greater than a specified distance. According to an embodiment, the instruction may provide notification information including information guiding that the first distance is included within a specified distance.

According to various embodiments, when receiving an input instructing the operation of the TOF camera while the calculated first distance is greater than the specified distance, the instruction enlarges the specified distance and exists within the enlarged specified distance It is possible to generate depth data for an object to perform a task, and to provide a service based on the generated depth data.

According to various embodiments, the instructions include at least one of an irradiation frequency (eg, frequency), an irradiation period (eg, exposure time), and a frame rate of the light emitting module (eg, the light emitting module 352) related to the TOF camera. It can be changed to enlarge the specified distance.

According to various embodiments, when the second distance cannot be calculated using the camera module, the instruction may calculate the second distance using the TOF camera.

According to various embodiments, the instruction may terminate a function related to the TOF camera when the second distance cannot be calculated using the TOF camera.

According to various embodiments, the instruction may generate depth data while changing a measurement parameter of the resumed TOF camera, and provide a service based on the generated depth data. According to an embodiment, the measurement parameter may include at least one of an irradiation frequency, an irradiation period, and a frame rate of the light emitting module related to the TOF camera.

According to various embodiments, the instruction may change the measurement parameter based on at least one of a shooting mode and environment information. According to an embodiment, the shooting mode may include at least one of a moving image mode and a still image mode. Also, the environment information may include at least one of a movement of the object, the number of the objects, and the type of the object.

According to various embodiments, the TOF camera may generate the depth data using a time when light arrives after being reflected by an object.

4 is a diagram illustrating an image before and after image enhancement is performed in an electronic device according to an exemplary embodiment.

In an embodiment, the electronic device 301 may acquire the image 400 using a camera module (eg, the camera module 340 ), and preview the acquired image 400 through the display 360 . It can be displayed on a screen (eg, the first preview screen). The image 400 acquired by the electronic device 301 may include regions requiring image improvement. For example, as shown in FIG. 4 , there may be an area where light is reflected by a fluorescent lamp, an area where a shadow is generated by a subject, and an area where a shadow is generated by a photographer.

In an embodiment, the electronic device 301 may acquire the image 410 by performing an image enhancement operation. The image 410 on which the electronic device 301 has performed image improvement is a region in the image 400 requiring image improvement (eg, a region where light is reflected by a fluorescent lamp, a region where a shadow is generated by a subject, and a shadow by a photographer). It may be an image in which the area in which the ) has been removed. Also, the electronic device 301 may display the improved image 410 on a preview screen (eg, a second preview screen) through the display 360 .

An image improvement operation of the electronic device 301 according to various embodiments will be described later with reference to FIGS. 6 to 16 .

5 is a diagram illustrating a configuration of an electronic device according to an embodiment.

Referring to FIG. 5 , an electronic device 500 according to an embodiment may be configured as follows. The illustrated configurations are examples, and at least some of the illustrated configurations may be changed according to a platform included in the electronic device 500 .

The electronic device 500 according to an embodiment may correspond to the electronic device 101 of FIG. 1 and the electronic device 301 of FIG. 3 .

According to an embodiment, the electronic device 500 includes an application layer 510 , a framework layer 520 , a hardware abstraction layer (HAL) 530 , and a kernel driver layer. It may include a kernel driver layer 540 and a hardware layer 550 .

The application layer 510 according to an embodiment includes at least one application 511 stored in a memory (eg, the memory 130 of FIG. 1 ) and executable by a processor (eg, the processor 120 of FIG. 1 ); A system UI 515 may be included.

In an embodiment, the type of the application 511 may not be limited, such as an Internet browser, a video application, or a game.

In an embodiment, the system UI 515 includes various GUI (graphics) implemented on a system of an electronic device (eg, the electronic device 101 of FIG. 1 ) such as a status bar and a quick view. user interface) may mean an application constituting the screen. Also, the system UI 515 may be an application for displaying a screen related to the system.

The framework layer 520 according to an embodiment provides various functions so that a function or information provided from one or more resources of the electronic device (eg, the electronic device 101 of FIG. 1 ) can be used by the application 511 . It may be provided as an application 511 .

The framework layer 520 according to an embodiment includes an activity manager 512 , a window manager 522 , a view system 523 , and a power manager 524 . ), an input manager 525 , a display manager 526 , and a sensor manager 527 .

The activity manager 512 according to an embodiment may control a life cycle and an activity stack of the application 511 .

The window manager 522 according to an embodiment may manage one or more GUI resources used on a screen of a display (eg, the display 160 of FIG. 1 ).

The view system 523 according to an embodiment may be a set of extensible views used to generate a user interface for the application 511 .

The power manager 524 according to an embodiment may manage capacity, temperature, or power of a battery (eg, the battery 189 of FIG. 1 ). Also, the power manager 524 may determine or provide related information necessary for the operation of the electronic device (eg, the electronic device 101 of FIG. 1 ) by using the information acquired during the management.

The input manager 525 according to an embodiment may provide information on the input device provided by the electronic device (eg, the electronic device 101 of FIG. 1 ).

The display manager 526 according to an embodiment may manage a life cycle (eg, connection/property change/removal) of a display (eg, the display 160 of FIG. 1 ). In addition, the display manager 526 may manage the output of GUI elements on the screen of the display (eg, the display 160 of FIG. 1 ), and the display (eg, the display 160 of FIG. 1 ) to be output according to an event such as a change in the folded state. The display 160 of 1 may be changed.

The sensor manager 527 according to an embodiment may control a sensor (eg, the sensor module 176 of FIG. 1 ) based on the usability of the application 511 .

In various embodiments, components included in the framework layer 520 may be components included in an intelligent shadow perspective prediction manager (ISPPM).

The hardware abstraction layer 530 according to an embodiment is an abstraction layer between a plurality of hardware components included in the hardware layer 550 and software components of an electronic device (eg, the electronic device 101 of FIG. 1 ). can mean The hardware abstraction layer 530 according to an embodiment may include an input dispatcher 531 , an event hub 532 , and a surface flinger 533 .

The input dispatcher 531 according to an embodiment may determine whether to transmit the input event to the input target window, process, or application 511 , and may transmit the input event according to the determination.

The event hub 532 according to an embodiment may standardize an input event generated by a sensor (eg, the sensor module 176 of FIG. 1 ).

The surface flinger 533 according to an embodiment may perform a function of providing an execution screen to be displayed on a display (eg, the display 160 of FIG. 1 ) among execution screens generated by the application 511 . Also, when the configuration of the display (eg, the display 160 of FIG. 1 ) is changed, the surface flinger 533 may request the application 511 to process the change in resolution and density according to the changed display configuration.

The kernel driver layer 540 according to an embodiment may include a plurality of drivers to control hardware included in the electronic device (eg, the electronic device 101 of FIG. 1 ). Also, the kernel driver layer 540 according to an embodiment may include a touch driver 541 , a sensor driver 542 , a general display 543 , and a DDI controller 544 .

In an embodiment, the touch driver 541 may include an interface for controlling the touch controller 551 and may control the touch controller 551 .

In an embodiment, the sensor driver 542 may include an interface for controlling the sensor controller 552 and may control the sensor controller 552 .

In an embodiment, the DDI controller 544 may drive the general display 543 .

The hardware layer 550 according to an embodiment may include a touch controller 551 , a sensor controller 552 , a display controller 553 , and an LCD panel 554 .

According to various embodiments, components included in the electronic device 500 (eg, the application layer 510 , the framework layer 520 , the hardware abstraction layer 530 , the kernel driver layer 540 , and the hardware layer 750 ) ) may be controlled by a processor (eg, the processor 120 of FIG. 1 or the processor 310 of FIG. 3 ).

6 is a flowchart illustrating a flow of performing image enhancement in an electronic device according to an exemplary embodiment.

According to various embodiments, operations 610 to 650 may be controlled by the processor 310 of the electronic device 301 , and data processed by the processor 310 may be at least temporarily stored in the memory 330 . have.

According to an embodiment, in operation 610 , the electronic device 301 displays a first preview screen displaying a first image including a subject image acquired using the first camera module 342 through the display 360 . can be displayed

In an embodiment, the camera module 340 of the electronic device 301 may include a plurality of camera modules (eg, a first camera module 342 and a second camera module 344 ). For example, the first camera module 342 may be a camera module including a wide-angle camera, and the second camera module 344 may be a camera module including a telephoto camera.

In an embodiment, the electronic device 301 may acquire images using a plurality of camera modules (eg, the first camera module 342 and the second camera module 344 ) under the control of the processor 310 . have. For example, the electronic device 301 may acquire the first image including the subject image by photographing the subject using the wide-angle camera included in the first camera module 342 under the control of the processor 310 . .

In an embodiment, the electronic device 301 displays a preview screen (eg, a first preview) that displays an image (eg, a first image) acquired by using a camera module (eg, the first camera module 342 ). screen) may be displayed through a display (eg, the display 360 ) under the control of the processor 310 . For example, referring to FIG. 4 , the electronic device 301 uses a first camera module 342 including a wide-angle camera to obtain a first image (eg, an image 400 ) under the control of the processor 310 . may be displayed through the display 360 . In this case, the displayed screen of the display 360 may be the first preview screen. As another example, the first image (eg, the image 400 ) displayed on the first preview screen of the electronic device 301 is a region requiring image improvement (eg, a region reflected by a fluorescent lamp, It may be an image including an area in which a shadow is generated by the photographer, an area in which a shadow is generated by the photographer).

According to an embodiment, in operation 620, the electronic device 301 may determine whether the first image satisfies a condition for determining whether image improvement is required.

In an embodiment, the electronic device 301 may determine whether the first image displayed on the first preview screen needs image improvement under the control of the processor 310 based on a condition. For example, in the electronic device 301, a condition (eg, a threshold condition) related to image improvement may be set. As a specific example, in the first image, there may be a case in which a shadow area generated by a subject or a photographer or a light reflection area generated by the subject exists. The electronic device 301 may determine that the value of the shadow region or the light reflection region is equal to or greater than a threshold value under the control of the processor 310 . In this case, the electronic device 301 may determine that the first image needs image improvement under the control of the processor 310 .

A detailed description of operation 620 according to an embodiment will be described with reference to FIG. 7 .

According to an embodiment, in operation 630 , the electronic device 301 may acquire a second image including a subject image using the second camera module 344 .

In an embodiment, the second camera module 344 of the electronic device 301 may be a camera module including a telephoto camera. For example, when the electronic device 301 uses the telephoto camera included in the second camera module 344, it may be easier to photograph a distant subject than when the wide-angle camera included in the first camera module 342 is used. can

In an embodiment, the electronic device 301 may photograph a subject using a telephoto camera included in the second camera module 344 under the control of the processor 310 . In this case, the electronic device 301 may acquire the second image including the subject image by photographing it using a telephoto camera.

In an embodiment, the distance value between the subject and the electronic device 301 is higher than when the electronic device 301 acquires the first image using the wide-angle camera included in the first camera module 342 , the second The case of acquiring the second image by using the telephoto camera included in the camera module 344 may be greater.

According to an embodiment, when it is determined that the condition is satisfied in operation 640 , the electronic device 301 may match the first object coordinates of the first image and the second object coordinates of the second image.

In an embodiment, the electronic device 301 may generate object coordinates (eg, first object coordinates) while acquiring the first image including the object image under the control of the processor 310 . For example, referring to FIG. 4 , the electronic device 301 obtains object coordinates (eg, first object coordinates) for a subject (eg, a watch) included in the image 400 while acquiring the image 400 . may have created

In an embodiment, the electronic device 301 may generate object coordinates (eg, second object coordinates) while acquiring the second image including the object image under the control of the processor 310 . Also, for example, the value of the distance between the subject and the electronic device 301 is higher than when the electronic device 301 acquires the first image by using the first camera module 342 , rather than using the second camera module 344 . Since the case where the second image is obtained by using the second image may be greater, the second object coordinates may be different from the first object coordinates.

In an embodiment, the electronic device 301 may match the first object coordinates of the first image with the second object coordinates of the second image under the control of the processor 310 .

According to an embodiment, in operation 650 , the electronic device 301 may display a second preview screen displaying a composite image obtained by synthesizing the first image and the second image through the display.

In an embodiment, the electronic device 301 controls images (eg, first image, second image) corresponding to object coordinates (eg, first object coordinates, second object coordinates) under the control of the processor 310 . ) can be synthesized (or registered) to create a composite image. For example, referring to FIG. 4 , the electronic device 301 may generate a composite image (eg, an image 410 ) under the control of the processor 310 . Also, in this case, the image 410 may be an image in which regions requiring image improvement (eg, a region reflected by a fluorescent lamp, a region in which a shadow is generated by a subject, an region in which a shadow is generated by a photographer) are removed.

7 is a flowchart illustrating a part of a detailed flow of performing image enhancement in an electronic device according to an exemplary embodiment. Operations 710 to 750 of FIG. 7 are specific flows related to operation 620 of FIG. 6 .

According to various embodiments, operations 710 to 750 of FIG. 7 may be controlled by the processor 310 of the electronic device 301 , and data processed by the processor 310 is at least temporarily stored in the memory 330 . can be saved.

Operations 710 to 750 according to various embodiments may be described with reference to FIGS. 8 to 11 , and various embodiments may not be limited as described below.

According to an embodiment, in operation 710, the electronic device 301 may classify the subject image area of the first preview screen.

Operation 710 according to an embodiment will be described with reference to FIG. 8 showing an embodiment of dividing a subject image area of a preview screen while image enhancement is being performed in the electronic device according to an embodiment.

In an embodiment, the electronic device 301 may display the subject image area and the image area excluding the subject image on the first preview screen under the control of the processor 310 . Referring to FIG. 8 , the electronic device 301 may display a subject image 800 including text or a subject image 810 having a shape on the first preview screen under the control of the processor 310 . have.

In an embodiment, the electronic device 301 divides the subject image area (eg, the subject image area 802 including text and the subject image area 812 having a shape characteristic) under the control of the processor 310 . can For example, referring to FIG. 8 , the electronic device 301 displays a subject 801 including text or a subject 811 having a shape through a first preview screen under the control of the processor 310 . It may be displayed as an image (eg, the first image). The electronic device 301 may generate a guide 802 for classifying the subject 801 including text under the control of the processor 310 , and a guide 812 for classifying the subject 811 having a shape characteristic. ) can be created. In this case, the electronic device 301 generates the guides 802 and 812 under the control of the processor 310 , thereby generating a subject image area (eg, a subject image area 802 including text, a subject image having a shape). region 812) can be divided.

According to an embodiment, in operation 720, the electronic device 301 may determine whether a threshold condition for determining whether image improvement is required is satisfied.

Operation 720 according to an embodiment will be described with reference to FIG. 9 showing an embodiment of determining whether the electronic device satisfies the threshold value condition for determining whether image improvement is necessary while performing image improvement, according to an embodiment.

In an embodiment, the electronic device 301 may determine whether image improvement is necessary based on the first image area 910 . For example, as shown in FIG. 9 , the first image area 910 may include a subject and a shadow of the subject. The electronic device 301 may determine whether image improvement is required under the control of the processor 310 based on the first image area 910 including the subject and the subject's shadow.

In an embodiment, the electronic device 301 may distinguish the background area 920 from the first image area 910 under the control of the processor 310 . For example, the electronic device 301 may classify the remaining areas excluding the subject from the first image area 910 including the subject and the subject's shadow under the control of the processor 310 .

In an embodiment, the electronic device 301 may obtain a color difference value for a color difference (eg, a color difference between the subject and an area excluding the subject) of the subject boundary in the first image 910 under the control of the processor 310 . have.

In an embodiment, the electronic device 301 may determine whether the color difference value is equal to or greater than a threshold value (eg, determine whether a threshold value condition is satisfied) under the control of the processor 310 . For example, when the color difference value is not equal to or greater than a threshold value under the control of the processor 310 , the electronic device 301 may determine that the threshold value condition is not satisfied. Also, when the color difference value is equal to or greater than a threshold value under the control of the processor 310 , the electronic device 301 may determine that the threshold value condition is satisfied.

According to an embodiment, in operation 730, when the threshold condition is satisfied, the electronic device 301 may obtain a distance value between the subject and the light source by using the TOF camera.

Operation 730 according to an embodiment will be described with reference to FIG. 10 showing an embodiment of obtaining distance values between a subject and a light source using a TOF camera while image enhancement is being performed in an electronic device, according to an embodiment.

Referring to FIG. 10 , the relationship between the length of the shadow 1000 according to the light source and the incident angle and the reflection angle 1010 according to the surface of the object can be seen.

In an embodiment, the electronic device 301 may obtain a distance value between the subject and the light source using the TOF camera 350 .

According to an embodiment, in operation 740 , the electronic device 301 may extract key points of the subject and generate first object coordinates based on the extracted key points.

Operation 740 according to an embodiment will be described with reference to FIG. 11 showing an embodiment of generating object coordinates and switching to a second camera while image enhancement is being performed in the electronic device, according to an embodiment.

In an embodiment, the electronic device 301 may extract a feature point of the subject under the control of the processor 310 using the distance values between the subject and the light source obtained in operation 730 . For example, the electronic device 301 may acquire a first image of the subject based on the first camera module 1100 (eg, corresponding to the first camera module 342 of FIG. 3 ). In this case, the electronic device 301 may have a distance value of h1 from the subject. The electronic device 301 may extract a feature point of the subject by using the obtained distance value of the subject and the distance value of the light source.

In an embodiment, the electronic device 301 may generate object coordinates (eg, first object coordinates) under the control of the processor 310 based on the extracted feature points of the object. For example, the electronic device 301 may generate object coordinates (eg, first object coordinates) under the control of the processor 310 based on the feature points of the object extracted while acquiring the image (eg, the first image). and the object coordinates (eg, first object coordinates) may be 3D spatial coordinates (eg, first object coordinates (X, Y, Z)).

According to an embodiment, in operation 750 , the electronic device 301 may switch to the second camera module 1110 . The second camera module 1110 may include a telephoto camera and may correspond to the second camera module 344 of FIG. 3 .

In an embodiment, the electronic device 301 may switch from the first camera module 1100 to the second camera module 1110 under the control of the processor 310 . For example, when it is determined that the image (eg, the first image) needs to be improved, the electronic device 301 switches from the first camera module 1100 to the second camera module 1110 under the control of the processor 310 . can be switched

In an embodiment, when detecting that the distance value between the electronic device 301 and the subject increases, the electronic device 301 may switch to the second camera module under the control of the processor 310 . For example, when the distance value between the electronic device 301 and the subject increases from h1 to h2, the electronic device 301 controls the processor 310 to capture the image with the telephoto camera by using the second camera module 1110 . ) can be converted to

12 is a flowchart illustrating a part of a detailed flow of performing image enhancement in an electronic device according to an exemplary embodiment.

Operations 630, 1210 to 1220, and 640 according to various embodiments will be described with reference to FIG. 13 showing an embodiment in which image synthesis is performed by matching object coordinates in an electronic device.

According to an embodiment, in operation 630 , the electronic device 301 may acquire a second image including a subject image using the second camera module 344 . The electronic device 301 may photograph a subject using a telephoto camera included in the second camera module 344 under the control of the processor 310 . In this case, the electronic device 301 may acquire the second image including the subject image by photographing it using a telephoto camera. The distance value between the subject and the electronic device 301 is higher than when the electronic device 301 acquires the first image using the wide-angle camera included in the first camera module 342 , the second camera module 344 A case in which the second image is acquired by using the telephoto camera included in .

According to an embodiment, in operation 1210 , the electronic device 301 may obtain a distance value between the subject and the light source using the TOF camera 350 . For example, the electronic device 301 may obtain a distance value to the subject included in the second image using the TOF camera 350 under the control of the processor 310 . Also, the electronic device 301 may acquire the distance value of the light source under the control of the processor 310 using the TOF camera 350 in the same or similar manner as described with reference to FIG. 10 .

According to an embodiment, in operation 1220 , the electronic device 301 may extract key points of the subject and generate second object coordinates based on the extracted key points.

In an embodiment, the electronic device 301 may extract a feature point of the subject under the control of the processor 310 using the obtained distance values of the subject and the light source. For example, the electronic device 301 may acquire a second image of the subject based on the second camera module 344 . The electronic device 301 may extract a feature point of the subject by using the obtained distance value of the subject and the distance value of the light source.

In an embodiment, the electronic device 301 may generate object coordinates (eg, second object coordinates) under the control of the processor 310 based on the extracted feature points of the object. For example, the electronic device 301 may generate object coordinates (eg, second object coordinates) based on feature points of a subject extracted while acquiring an image (eg, a second image), and may generate object coordinates (eg, second image). 2 object coordinates) may be 3D spatial coordinates (eg, second object coordinates (X, Y, Z)).

According to an embodiment, in operation 640 , the electronic device 301 may match the first object coordinates of the first image and the second object coordinates of the second image.

Referring to FIG. 13 , the electronic device 301 may have acquired the first object coordinates from the first image. For example, the electronic device 301 may have obtained (X1, Y1, Z), (X2, Y2, Z), (X3, Y3, Z), (X4, Y4, Z) as the first object coordinates. have. In addition, the electronic device 301 provides the second object coordinates as (X1, Y1, Z+H), (X2, Y2, Z+H), (X3, Y3, Z+H), (X4, Y4, Z+). H) may have been obtained.

In an embodiment, as shown in FIG. 13 , the electronic device 301 configures the first object coordinates ((X1, Y1, Z), (X2, Y2, Z), (X3, Y3, Z) of the first image. , (X4, Y4, Z)) to the second object coordinates of the second image ((X1, Y1, Z+H), (X2, Y2, Z+H), (X3, Y3, Z+H), ( X4, Y4, Z+H)) can be matched.

In operation 650 according to an embodiment, the electronic device 301 may display a second preview screen displaying a composite image obtained by synthesizing the first image and the second image through the display.

In an embodiment, the electronic device 301 controls the first object coordinates ((X1, Y1, Z), (X2, Y2, Z), (X3, Y3, Z), ( X4, Y4, Z)), and the second object coordinates ((X1, Y1, Z+H), (X2, Y2, Z+H), (X3, Y3, Z+H), (X4, Y4, Z +H))) corresponding images (eg, the first image and the second image) may be synthesized (or registered) to generate a composite image. For example, referring to FIG. 13 , the electronic device 301 may generate a synthesized image under the control of the processor 310 . Also, in this case, the synthesized image may be an image in which regions requiring image improvement (eg, a region reflected by a fluorescent lamp, a region in which a shadow is generated by a subject, an region in which a shadow is generated by a photographer) are removed.

According to various embodiments, for convenience of description, first object coordinates ((X1, Y1, Z), (X2, Y2, Z), (X3, Y3, Z), (X4, Y4, Z)), and The second object coordinates ((X1, Y1, Z+H), (X2, Y2, Z+H), (X3, Y3, Z+H), (X4, Y4, Z+H)) were explained as an example. , may not be limited to the above-described embodiment. For example, the X and Y coordinates may be different, and H may also be different. In addition, depending on the case, X, Y, Z, and H of the above-described four coordinates may be different, respectively.

14 is a diagram illustrating an embodiment of generating a guide based on a subject for image improvement in an electronic device according to an embodiment.

In an embodiment, the electronic device 301 may acquire an image (eg, a first image) including a subject and a shadow of the subject by using the first camera module 1410 . Also, the electronic device 301 may display a subject and an image (eg, a first image) including a shadow of the subject on the screen of the display 360 (eg, displayed on the first preview screen). In this case, the distance value between the electronic device 301 and the subject may be h1.

In an embodiment, the electronic device 301 may display the guide 1420 on the screen of the display 360 (eg, the first preview screen) to improve the image. The electronic device 301 displays the guide 1420 on the screen of the display 360 (eg, the first preview screen) to allow the user to perform an operation regarding the distance value between the electronic device 301 and the subject. can For example, the user may move the electronic device 301 further away from the subject according to the guide 1420 . Referring to FIG. 14 , in this case, the electronic device 301 may have a distance value of h2 from the subject.

In an embodiment, the distance (eg, h2 ) between the electronic device 301 and the subject according to the guide 1420 may be a distance value capable of photographing without a shadow. The electronic device 301 may obtain a distance value capable of photographing without a shadow based on at least the TOF camera 350 .

In an embodiment, the electronic device 301 may switch the first camera module 1410 to the second camera module 1430 while the distance value h1 is changed to the distance value h2 . The electronic device 301 may automatically perform the conversion, or may perform the conversion manually in response to obtaining a user's input.

In an embodiment, the electronic device 301 may acquire an image that does not require image improvement (eg, a second image) in a state in which it is switched to the second camera module 1430 . Also, the electronic device 301 may display an image (eg, a second image) that does not require image improvement on a screen (eg, a second preview screen) of the display 360 .

15 is a diagram illustrating an embodiment of generating a shadow area for a subject image in an electronic device according to an embodiment.

For example, referring to FIG. 15 , when a user desires to create a shadow, the electronic device 301 may generate a shadow based on an image 1510 on which image enhancement has been performed (eg, an image from which the shadow has been removed). have.

As a specific example, by synthesizing a shadow image with the image 1510 on which image enhancement has been performed using image data (eg, image data including shadow) stored in the memory 330 during the image enhancement process, the image 1520 is ) can be created.

16 is a diagram illustrating an example of generating a light reflection area for a subject image in an electronic device according to an exemplary embodiment.

For example, referring to FIG. 16 , when a user desires to generate a light reflection effect, the electronic device 301 transmits light based on an image 1610 on which image enhancement is performed (eg, an image from which the light reflection effect is removed). You can create a reflective effect.

As a specific example, an image including a light reflection effect in the image 1610 on which image enhancement is performed using image data (eg, image data including a light reflection effect) stored in the memory 330 during the image enhancement process By synthesizing , an image 1620 may be generated.

The electronic device according to various embodiments disclosed in this document may be a device of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B," "A, B or C," "at least one of A, B and C," and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the element from other elements in question, and may refer to elements in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 101 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to an embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices (eg, It can be distributed (eg downloaded or uploaded) directly or online between smartphones (eg: smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repetitively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Claims (20)

In an electronic device,
display;
a first camera module including a first camera having a first viewing angle;
a second camera module including a second camera having a second viewing angle different from the first viewing angle; and
a processor operatively connected to the first camera module and the second camera module;
The processor is
A first preview screen obtained by using the first camera module and displaying a first image including a subject image is displayed through the display,
Determining whether the first image satisfies the condition for determining whether image improvement is necessary,
Obtaining a second image including the subject image by using the second camera module,
When it is determined that the condition is satisfied, the coordinates of the first object of the first image and the coordinates of the second object of the second image are matched;
and displaying a second preview screen displaying a composite image obtained by synthesizing the first image and the second image through the display. The method according to claim 1,
The processor is
and distinguishing the subject image area on the first preview screen. The method according to claim 1,
The condition for determining whether the image improvement is necessary includes at least a condition regarding a threshold value. The method according to claim 1,
further comprising a TOF camera,
The processor is
Acquire a distance value for at least one of a subject and a light source based at least on the TOF camera. 5. The method according to claim 4,
The processor is
extracting a feature point of the subject based at least on the TOF camera,
and generating the first object coordinates with respect to a first image based on the extracted feature points. 5. The method according to claim 4,
The processor is
Switching to a second camera module including a second camera based on a distance value for at least one of a subject and a light source acquired based on the TOF camera at least. 5. The method according to claim 4,
The processor is
extracting a feature point of the subject based at least on the TOF camera,
and generating the second object coordinates with respect to a second image based on the extracted feature points. 5. The method according to claim 4,
The processor is
and displaying a guide on the first preview screen based on a distance value for at least one of a subject and a light source acquired based on the TOF camera at least. 9. The method of claim 8,
The processor is
and continuously displaying the guide on the second preview screen. The method according to claim 1,
The image improvement is at least one of shadow removal, shadow generation, light reflection removal, and light reflection generation, the electronic device. A method of operating an electronic device, comprising:
displaying, through the display, a first preview screen obtained by using the first camera module and displaying a first image including a subject image;
determining whether the first image satisfies a condition for determining whether image improvement is necessary;
obtaining a second image including the subject image by using the second camera module;
matching the first object coordinates of the first image and the second object coordinates of the second image when it is determined that the condition is satisfied; and
and displaying, through the display, a second preview screen displaying a composite image obtained by synthesizing the first image and the second image. 12. The method of claim 11,
and classifying the subject image area on the first preview screen. 12. The method of claim 11,
wherein the condition for determining whether the image enhancement is necessary is a condition for at least a threshold value. 12. The method of claim 11,
and obtaining a distance value for at least one of a subject and a light source based at least on a TOF camera. 15. The method of claim 14,
extracting the feature points of the subject based at least on the TOF camera; and
and generating the first object coordinates with respect to a first image based on the extracted feature points. 15. The method of claim 14,
and switching to a second camera module including a second camera based on a distance value for at least one of a subject and a light source acquired based on at least the TOF camera. 15. The method of claim 14,
extracting the feature points of the subject based at least on the TOF camera; and
and generating the second object coordinates with respect to a second image based on the extracted feature points. 15. The method of claim 14,
and displaying a guide on the first preview screen based on a distance value with respect to at least one of a subject and a light source acquired based on at least the TOF camera. 19. The method of claim 18,
and continuously displaying the guide on the second preview screen. 12. The method of claim 11,
The image improvement is at least one of shadow removal, shadow generation, light reflection removal, and light reflection generation.

Images