KR20220156335A - Electronic device and method for processing image based on depth information using the same - Google Patents

Abstract

According to various embodiments of the present disclosure, an electronic device comprises a camera, a sensor circuit, and a processor operatively connected to the camera and the sensor circuit. The processor can be configured to: obtain a plurality of frames including a plurality of subjects through the camera; identify depth information of the plurality of subjects included in the plurality of frames through the sensor circuit; remove, from the plurality of frames, a first subject having first depth information from among the plurality of subjects, on the basis of the depth information of the plurality of subjects; and generate an image by combining the plurality of frames from which the first subject is removed. Various other embodiments are also possible in addition to embodiments disclosed in the present document. The present invention can easily perform an operation of processing an image.

Description

Electronic device and image processing method based on depth information using the same {ELECTRONIC DEVICE AND METHOD FOR PROCESSING IMAGE BASED ON DEPTH INFORMATION USING THE SAME}

Various embodiments of the present disclosure relate to an electronic device and a method of processing an image based on depth information using the same.

Electronic devices provide a camera function to generate and store images or videos by capturing a subject. When an obstacle exists when capturing an image through a camera, the electronic device may obtain an image of a main subject to be photographed by removing the obstacle from an image acquired through the camera. Alternatively, the electronic device may obtain an image of a main subject to be photographed by removing an obstacle detected based on deep learning in an image obtained through a camera.

However, it may take a lot of time to determine whether an obstacle exists in an image obtained through a camera, and it is inconvenient to perform pre-learning to optimize a deep learning model each time to detect an obstacle.

An electronic device according to various embodiments of the present disclosure may acquire a plurality of images including a plurality of subjects at different photographing points of time through a camera. The electronic device removes a subject located at a short distance among the plurality of subjects from an image based on depth information on a plurality of subjects confirmed through a sensor, and removes a subject located at a short distance from the image, and detects a long distance covered by the subject located at a short distance. A partial area of the main subject located at may be restored.

An electronic device according to various embodiments of the present disclosure includes a camera, a sensor circuit, and a processor operatively connected to the camera and the sensor circuit, wherein the processor includes a plurality of subjects including a plurality of subjects through the camera. Obtain frames of, check depth information on the plurality of subjects included in the plurality of frames through the sensor circuit, and based on the depth information on the plurality of subjects, the plurality of frames It may be set to generate an image by removing a first subject having first depth information from among the plurality of subjects and combining a plurality of frames from which the first subject is removed.

An image processing method based on depth information of an electronic device according to various embodiments of the present disclosure includes an operation of acquiring a plurality of frames including a plurality of subjects through a camera, and included in the plurality of frames through a sensor circuit. Checking depth information of the plurality of subjects, removing a first subject having first depth information among the plurality of subjects from the plurality of frames based on the depth information of the plurality of subjects and generating an image by synthesizing a plurality of frames from which the first subject is removed.

In an electronic device according to various embodiments of the present disclosure, an image obtained through a camera of a subject located at a short distance corresponding to an obstacle among a plurality of subjects using depth information on a plurality of subjects identified through a sensor , and an image processing operation may be easily performed by restoring a partial region of a main subject located at a distance based on a plurality of images acquired at different times through a camera.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram illustrating a camera module, in accordance with various embodiments.
3 is a block diagram illustrating an electronic device, according to various embodiments.
4A is a flowchart illustrating a method of processing an image based on depth information, according to various embodiments.
4B is a flowchart illustrating a method of processing an image based on depth information, according to various embodiments.
5A and 5B are diagrams for explaining a method of acquiring a plurality of frames including a plurality of subjects at different viewpoints in an obstruction removal mode in panoramic photographing according to various embodiments.
5C is a diagram for explaining a method of generating a depth map for a plurality of frames including a plurality of subjects in an obstruction removal mode in panoramic photography, according to various embodiments.
6 is a graph for explaining a method of determining whether an object located at a short distance from an electronic device is an obstacle based on the number of pixels for each distance of a plurality of objects, according to various embodiments.
7A is a method of removing a first subject having first depth information from a plurality of frames in an obstruction removal mode in a panoramic photographing and interpolating a region of a frame from which the first subject is removed, according to various embodiments; It is a drawing for explaining.
FIG. 7B is a diagram for explaining a method of generating an image by synthesizing interpolated frames in an obstruction removal mode in panoramic photography, according to various embodiments.
8A and 8B are diagrams for describing a method of obtaining a plurality of frames including a plurality of subjects at different viewpoints in an obstruction removal mode in normal photographing, according to various embodiments.
FIG. 8C is a diagram for explaining a method of generating a depth map for a plurality of frames including a plurality of subjects in an obstruction removal mode in normal shooting, according to various embodiments.
9A is a method of removing a first subject having first depth information from a plurality of frames in an obstruction removal mode in normal shooting and interpolating a region of a frame from which the first subject is removed, according to various embodiments; It is a drawing for explaining.
9B is a diagram for explaining a method of generating an image by synthesizing interpolated frames in an obstruction removal mode in normal shooting, according to various embodiments.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments.

Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram 200 illustrating a camera module 180, in accordance with various embodiments.

Referring to FIG. 2 , the camera module 180 includes a lens assembly 210, a flash 220, an image sensor 230, an image stabilizer 240, a memory 250 (eg, a buffer memory), or an image signal processor. (260). The lens assembly 210 may collect light emitted from a subject that is an image capturing target. The lens assembly 210 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 210 . In this case, the camera module 180 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may have the same lens properties as other lens assemblies. may have one or more lens properties different from the lens properties of . The lens assembly 210 may include, for example, a wide-angle lens or a telephoto lens.

The flash 220 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 220 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp. The image sensor 230 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 210 into an electrical signal. According to an embodiment, the image sensor 230 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 240 moves at least one lens or image sensor 230 included in the lens assembly 210 in a specific direction in response to movement of the camera module 180 or the electronic device 101 including the same. Operation characteristics of the image sensor 230 may be controlled (eg, read-out timing is adjusted, etc.). This makes it possible to compensate at least part of the negative effect of the movement on the image being taken. According to an embodiment, the image stabilizer 240 uses a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180 to control the camera module 180 or an electronic device ( 101) can detect such a movement. According to one embodiment, the image stabilizer 240 may be implemented as, for example, an optical image stabilizer. The memory 250 may at least temporarily store at least a portion of an image acquired through the image sensor 230 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 250 and , a copy image (eg, a low resolution image) corresponding thereto may be previewed through the display module 160 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 250 may be obtained and processed by the image signal processor 260 , for example. According to one embodiment, the memory 250 may be configured as at least a part of the memory 130 or as a separate memory operated independently of the memory 130 .

The image signal processor 260 may perform one or more image processes on an image acquired through the image sensor 230 or an image stored in the memory 250 . The one or more image processes, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening. Additionally or alternatively, the image signal processor 260 may include at least one of the components included in the camera module 180 (eg, an image sensor). 230) may be controlled (eg, exposure time control, read-out timing control, etc.) The image processed by the image signal processor 260 is stored again in the memory 250 for further processing. or may be provided as an external component of the camera module 180 (eg, the memory 130, the display module 160, the electronic device 102, the electronic device 104, or the server 108). According to an example, the image signal processor 260 may be configured as at least a part of the processor 120 or may be configured as a separate processor that operates independently of the processor 120. The image signal processor 260 may be configured as a processor 120 When configured as a separate processor, at least one image processed by the image signal processor 260 may be displayed through the display module 160 as it is or after additional image processing by the processor 120 .

According to an embodiment, the electronic device 101 may include a plurality of camera modules 180 each having different properties or functions. In this case, for example, at least one of the plurality of camera modules 180 may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules 180 may be a front camera, and at least another one may be a rear camera.

3 is a block diagram 300 illustrating an electronic device 301 according to various embodiments.

Referring to FIG. 3, an electronic device 301 (eg, the electronic device 101 of FIG. 1) includes a communication circuit 310 (eg, the communication module 190 of FIG. 1), a memory 320 (eg, the FIG. 1 memory 130), camera 330 (eg, camera module 180 of FIG. 1), sensor circuit 340 (eg, sensor module 176 of FIG. 1), touch screen display 350 ( Example: the display module 160 of FIG. 1), and/or a processor 360 (eg, the processor 120 of FIG. 1 and/or the image signal processor 260 of FIG. 2).

According to various embodiments of the present disclosure, the communication circuit 310 (eg, the communication module 190 of FIG. 1 ) is an external electronic device (eg, the electronic devices 102 and 104 of FIG. 1 , the server 108 ) It is possible to establish a communication channel with and to support transmission and reception of various data with an external electronic device.

According to various embodiments of the present disclosure, when the memory 320 (eg, the memory 130 of FIG. 1 ) is set to the obstruction removal mode, a first depth from a plurality of frames obtained through the camera 330 Instructions for identifying a first subject having information may be stored. The memory 320 may store instructions for interpolating an area of a frame from which the first subject is removed from a plurality of frames by using at least one other frame among the plurality of frames. The memory 320 may store instructions for synthesizing interpolated frames in an area from which the first subject is removed.

According to various embodiments of the present disclosure, the camera 330 (eg, the camera module 180 of FIG. 1 ) transfers the collected image to the display 351 as a preview screen so that the user can view it through the camera 330. You can check the losing video. In response to a photographing request input, the camera 330 may generate image data by capturing an image collected at the time when the photographing request input occurs. In one embodiment, one or more cameras 330 may be provided.

According to various embodiments of the present disclosure, the sensor circuit 340 (eg, the sensor module 176 of FIG. 1 ) may include a depth sensor 341 and/or an inertial sensor 343 .

In one embodiment, the depth sensor 341 may obtain depth information on a subject. The depth sensor 341 may measure distance information between the electronic device 301 and a plurality of subjects. The depth sensor 341 may generate a depth map for a plurality of frames including a plurality of subjects acquired through the camera 330 .

In one embodiment, the inertial sensor 343 may detect the inertial force of motion and provide various navigation-related information such as acceleration, speed, direction, and/or distance of the camera 330 , which is an inertial measurement target. For example, the inertial sensor 343 may include an acceleration sensor, a gyro sensor, and/or a geomagnetic sensor.

In one embodiment, the acceleration sensor includes a sensor that measures the movement of the electronic device 301 in space, and detects, for example, a change in acceleration and/or a change in angular acceleration that occurs when the user moves the electronic device 301. It may include a sensor that The acceleration sensor may detect acceleration in three axis directions. Also, the acceleration sensor may detect the inclination of the electronic device 301 .

In one embodiment, the geomagnetic sensor includes a sensor for measuring an azimuth, and may include a sensor for measuring an azimuth by detecting a magnetic field formed in a north-south direction of the earth. The geomagnetic sensor may detect geomagnetism in three axial directions.

In one embodiment, the gyro sensor may include an inertial sensor that measures the rotational angular velocity of the electronic device 301 and may include a sensor that detects a current direction using an inertial force of a rotating object. The gyro sensor can measure rotational angular velocity in two axis directions.

According to various embodiments of the present disclosure, the touch screen display 350 (eg, the display module 160 of FIG. 1 ) may be integrally configured including the display 351 and the touch panel 353 .

In one embodiment, the touch screen display 350 displays an image under the control of the processor 360, and includes a liquid crystal display (LCD), a light-emitting diode (LED) display, and an organic light emitting diode (OLED). It may be implemented as any one of an organic light-emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. However, it is not limited thereto. For example, the touch screen display 350 may include a rollable, slidable, stretchable, or foldable display.

In one embodiment, the touchscreen display 350 may display a user interface related to a camera function based on executing a camera application under the control of the processor 360 . The touch screen display 350 may display a preview image including a plurality of subjects obtained from the camera 330 under the control of the processor 360 . When panoramic photography is performed under the control of the processor 360, the touch screen display 350 may display a panorama view path guiding movement in a specific direction (eg, a horizontal direction and/or a vertical direction).

According to various embodiments of the present disclosure, the processor 360 (eg, the processor 120 of FIG. 1 and/or the image signal processor 260 of FIG. 2 ) is, for example, a micro controller unit (MCU). unit), and may control a plurality of hardware components connected to the processor 360 by driving an operating system (OS) or an embedded software program. For example, the processor 360 may control a plurality of hardware components according to instructions stored in the memory 320 (eg, the program 140 of FIG. 1 ).

In various embodiments, the processor 360 may control overall operation of the electronic device 301 and signal flow between internal components of the electronic device 301 and may process data.

The processor 360 according to various embodiments may acquire a plurality of frames including a plurality of subjects through the camera 330 . The plurality of subjects may include a first subject positioned at a first distance (eg, short distance) from the camera 330 and a second subject positioned at a second distance (eg, far distance) greater than the first distance. The processor 360 may check depth information on a plurality of subjects included in a plurality of frames through the depth sensor 341 . For example, the depth information of the plurality of subjects may include distance information between the camera 330 and each of the plurality of subjects and/or histogram information according to the distance information (eg, distance-by-distance to the plurality of subjects included in each frame). number of pixels).

In an embodiment, the processor 360 may remove a first subject having first depth information from among a plurality of subjects from a plurality of frames, based on depth information of the plurality of subjects. For example, the first subject having the first depth information may be a subject located at a short distance from the camera 330 among a plurality of subjects and/or a subject whose number of pixels is less than a specified number among the plurality of subjects included in each frame (or , a subject having a small number of pixels among a plurality of subjects included in each frame).

In an embodiment, the processor 360 checks coordinate values of an area from which the first subject is removed in each of the plurality of frames, and based on the coordinate values of the area from which the first subject is removed, the processor 360 determines the RGB values of the plurality of frames. At least one other frame including a pixel having a value may be identified. The processor 360 may interpolate an area from which the first subject is removed in each frame based on the identified RGB values of at least one other frame. The processor 360 may synthesize a plurality of interpolated frames to generate an image and display the image on the display 351 .

An electronic device 301 according to various embodiments includes a camera 330, a sensor circuit 340, and a processor 360 operatively connected to the camera 330 and the sensor circuit 340, The processor 360 acquires a plurality of frames including a plurality of subjects through the camera 330, and obtains information about the plurality of subjects included in the plurality of frames through the sensor circuit 340. Checking depth information, removing a first subject having first depth information among the plurality of subjects from the plurality of frames based on the depth information of the plurality of subjects, and It may be set to generate an image by synthesizing a plurality of removed frames.

In various embodiments, the plurality of frames obtained through the camera may include a plurality of frames captured at different points of time according to the movement of the electronic device 301 .

In various embodiments, the depth information of the plurality of subjects may include distance information between each of the plurality of subjects from the camera 330 and/or histogram information according to the distance information.

In various embodiments, the histogram information may include the number of pixels for each distance to a plurality of subjects included in each of the plurality of frames.

The processor 360 according to various embodiments checks distance information between the camera 330 and each of the plurality of subjects and/or the number of pixels for each distance to the plurality of subjects according to the distance information, and , and based on the distance information and/or the number of pixels per distance to the plurality of subjects, the first subject having the first depth information among the plurality of subjects may be determined.

The processor 360 according to various embodiments selects a subject located at a short distance from the camera 330 from among the plurality of subjects and/or a subject having a small number of pixels among the plurality of subjects, the first It may be set to determine a first subject having depth information.

The processor 360 according to various embodiments may set coordinate values on 3D space coordinates for the plurality of frames through the sensor circuit 340 .

The processor 360 according to various embodiments may be configured to map coordinate values on 3D spatial coordinates of each of the plurality of frames to pixel coordinates and RGB values of each frame.

The processor 360 according to various embodiments includes a pixel having an RGB value among the plurality of frames based on coordinate values of an area from which the first subject having the first depth information is removed. At least one other frame may be identified, and an area from which the first subject is removed in each frame may be interpolated based on the identified RGB values of the at least one other frame.

The processor 360 according to various embodiments may be set to generate the image by synthesizing a plurality of frames obtained by interpolating an area from which the first subject is removed.

In various embodiments, the generated image may include a second subject having second depth information among the plurality of subjects. FIG. 4A illustrates an image based on depth information according to various embodiments. It is a flowchart 400 for explaining a method of processing.

Referring to FIG. 4A , a processor (eg, the processor 360 of FIG. 3 ) of an electronic device (eg, the electronic device 301 of FIG. 3 ), in operation 401 , a camera (eg, the camera 330 of FIG. 3 ) It is possible to obtain a plurality of frames including a plurality of subjects through. The plurality of subjects may include a first subject positioned at a first distance (eg, short distance) from the camera 330 and a second subject positioned at a second distance (eg, far distance) greater than the first distance. However, it is not limited thereto.

In various embodiments, when an image is captured through the camera 330, a first subject located at a first distance from the camera 330 among a plurality of subjects is located at a second distance greater than the first distance. The second subject (eg, the target subject) may be covered. The processor 360 removes the first subject located at the first distance from the plurality of frames that covers the second subject located at the second distance, and only uses the second subject (eg, the target subject) located at the second distance. Operations 403 to 407 described below may be performed to acquire the included image.

In an embodiment, the processor 360 checks depth information on a plurality of subjects included in a plurality of frames through a sensor circuit (eg, the sensor circuit 340 of FIG. 3 ) in operation 403 . can In operation 405, the processor 360 may remove a first subject having first depth information among a plurality of subjects from a plurality of frames, based on the depth information of the plurality of subjects.

In various embodiments, the depth information of the plurality of subjects may include distance information between the camera 330 and each of the plurality of subjects and/or histogram information according to the distance information. The histogram information may include the number of pixels for each distance to a plurality of subjects included in each frame.

In various embodiments, the first subject having the first depth information may be a subject located at a short distance from the camera 330 among a plurality of subjects and/or a specified number of pixels among a plurality of subjects included in each frame. It may include a subject (or a subject having a small number of pixels among a plurality of subjects included in each frame).

In an embodiment, in operation 407, the processor 360 may generate an image by synthesizing a plurality of frames from which the first subject is removed. For example, the processor 360 checks coordinate values of an area from which the first subject is removed in each of a plurality of frames, and based on the coordinate values of the area from which the first subject is removed, RGB values among the plurality of frames exist. At least one other frame containing a pixel that The processor 360 may interpolate an area from which the first subject is removed in each frame based on the identified RGB values of at least one other frame. The processor 360 may generate an image by synthesizing a plurality of interpolated frames.

In relation to each operation of FIG. 4A described above according to various embodiments, various embodiments will be described in FIG. 4B to be described later.

4B is a flowchart 450 illustrating a method of processing an image based on depth information, according to various embodiments.

Referring to FIG. 4B , a processor (eg, the processor 360 of FIG. 3 ) of an electronic device (eg, the electronic device 301 of FIG. 3 ) may execute a camera application in operation 451 . For example, the processor 360 displays a preview image obtained through a camera (eg, the camera 330 of FIG. 3 ) based on detecting an input for executing a camera application (eg, the display 351 of FIG. 3 ). ) can be displayed.

In one embodiment, the processor 360 may provide a photographing button on the preview image displayed on the display 351 based on detecting an input for executing the camera application. When a user input for selecting a photographing button is detected, the processor 360 may start photographing.

In one embodiment, in operation 453, the processor 360 may check whether the blockage removal mode is set. For example, the processor 360 may provide a plurality of shooting modes. The plurality of shooting modes may include normal (or automatic), video, selfie, live focus, slow motion, pro, and/or panorama. In one embodiment, the obstruction removal mode may be provided in normal shooting and/or panoramic shooting. However, it is not limited thereto.

Various embodiments of the obstruction removal mode in the above-described panoramic photographing will be described with reference to FIGS. 5A to 7B to be described later. Also, the obstruction removal mode in normal shooting will be described in various embodiments in FIGS. 8A to 9B to be described later.

In one embodiment, if it is confirmed that the obstacle removal mode is set (eg, YES in operation 453), the processor 360 may acquire a plurality of frames including a plurality of subjects through the camera 330 in operation 455. have. For example, the plurality of subjects may include a first subject located at a first distance (eg, short distance) from the camera 330 and a second subject located at a second distance (eg, far distance) greater than the first distance. have. In one embodiment, in the case of an obstruction removal mode in panoramic shooting, the processor 360 may display a panoramic view path guiding movement in a specific direction (eg, a horizontal direction and/or a vertical direction) on the display 351. have.

In one embodiment, the processor 360 may detect movement of the electronic device 301 in a specific direction (eg, a horizontal direction and/or a vertical direction). As the electronic device 301 moves, the composition of the camera 330 may change. A plurality of frames including a plurality of subjects are a plurality of frames (e.g., acquired at different points in time (e.g., shooting time) according to movement of the electronic device 301 (e.g., movement in a horizontal direction and/or a vertical direction). Example: the first frame, the second frame, the third frame, ..., the n−2 th frame, the n−1 th frame, and the n th frame). For example, the processor 360 obtains a first frame at a first time point, a second frame at a second time point, and a third frame at a third time point based on the detection of movement of the electronic device 301 . can be obtained. Subsequently, based on the detection of the movement of the electronic device 301, the processor 360 obtains the n-2th frame at the n-2th time point, and obtains the n-1th frame at the n-1th time point. and an n-th frame may be acquired at an n-th time point.

In one embodiment, the first point of view may include a time point when a photographing request is detected (eg, a time point when a user input for selecting the above-described photographing button is detected), and the n-th point of time may include a time point when photographing ends. can However, it is not limited thereto.

In one embodiment, each of the plurality of frames may have some overlapping area. For example, the first frame, the second frame, and/or the third frame may have partially overlapped areas, and the n-2th frame, the n-1th frame, and/or the nth frame may have partially overlapped areas. have.

In another embodiment, in the case of an obstruction removal mode in normal shooting, the processor 360 performs a motion gesture (eg, a gesture of shaking the electronic device 301 in a specific direction) through the sensor circuit 340 of the electronic device 301 . can be detected. In response to detecting the motion gesture, the processor 360 may obtain a plurality of frames including a plurality of subjects at different viewpoints through the camera 330 .

In an embodiment, the processor 360 may generate a depth map for a plurality of frames through a sensor circuit (eg, the sensor circuit 340 of FIG. 3 ) in operation 457 . For example, the processor 360 may generate a depth map for a plurality of frames acquired through the camera 330 using the sensor circuit 340 , for example, a depth sensor (eg, the depth sensor 341 ). For example, the processor 360 generates a depth map for a first frame acquired at a first point of view using the depth sensor 341 based on the movement of the electronic device 301 in a specific direction, and the second point of view A depth map for the second frame acquired at , and a depth map for the n-th frame obtained at the n-th point of view may be generated.

In an embodiment, in operation 459, the processor 360 may check depth information on a plurality of subjects based on the generated depth maps of the plurality of frames. Depth information about the plurality of subjects may include distance information between the camera 330 and each of the plurality of subjects. The present invention is not limited thereto, and the depth information of the plurality of subjects may further include histogram information according to distance information between the camera 330 and each of the plurality of subjects. Histogram information according to distance information between the camera 330 and each of the plurality of subjects may include the number of pixels for each distance to the plurality of subjects included in each frame.

In an embodiment, in operation 461, the processor 360 may identify a first subject having first depth information among a plurality of subjects based on the depth information of the plurality of subjects.

For example, the processor 360 may identify a first subject having first depth information among a plurality of subjects based on distance information between the camera 330 and each of the plurality of subjects and/or histogram information according to the distance information. have. For example, the first subject having the first depth information may be a subject located at a short distance from the camera 330 among a plurality of subjects and/or a subject whose number of pixels is less than a specified number among the plurality of subjects included in each frame (or , an object of a specific distance having a small number of pixels among a plurality of objects included in each frame).

In an embodiment, in operation 463, the processor 360 may remove the first subject having the first depth information from the plurality of frames. In operation 465, the processor 360 may check a coordinate value of an area from which the first subject is removed in each of a plurality of frames. For example, the processor 360 may activate an inertial sensor (eg, the inertial sensor 343 of FIG. 3 ) when acquiring a plurality of frames through the camera 330 . The processor 360 may set coordinate values on 3D space coordinates for a plurality of frames using the inertial sensor 343 . The 3D spatial coordinates of each of the plurality of frames may be matched with pixel coordinates and RGB values of each frame. The processor 360 may store in a memory (eg, the memory 320 of FIG. 3 ) coordinate values on 3D spatial coordinates for each of a plurality of allocated frames, and coordinates and RGB values of pixels matched thereto.

In an embodiment, in operation 467, the processor 360 may identify at least one other frame including a pixel having an RGB value based on the coordinate values of the region from which the first subject is removed in each frame. In operation 469, the processor 360 may interpolate an area from which the first subject is removed in each frame based on the identified RGB values of at least one other frame.

For example, in the case of the obstruction removal mode in panoramic photography, the processor 360 moves the camera 330 (eg, changes the composition of the camera 330) to obtain a plurality of frames at different points in time (eg, a shooting point). Among them, a frame including a pixel having an RGB value at a coordinate corresponding to a coordinate value of an area from which the first subject is removed may be identified. The processor 360 may interpolate an area from which the first subject is removed in each frame based on frames including pixels having RGB values.

In another embodiment, in the case of the obstacle removal mode in normal shooting, the processor 360 may include coordinates of an area from which the first subject is removed among a plurality of frames obtained in response to detecting the motion gesture of the electronic device 301. A frame including a pixel having an RGB value at a coordinate corresponding to the value can be checked. The processor 360 may interpolate an area from which the first subject is removed in each frame based on frames including pixels having RGB values.

In one embodiment, in operation 471, the processor 360 may generate an image by synthesizing a plurality of interpolated frames. The processor 360 may display the generated image in operation 473.

In one embodiment, if it is determined that the obstruction removal mode is not set (eg, NO in operation 453), the processor 360 may perform a normal photographing mode in operation 475. For example, the general photographing mode may include, for example, a mode for obtaining an image including a first subject located at a first distance from the camera 330 and a second subject located at a second distance greater than the first distance. have.

An image processing method based on depth information of an electronic device 301 according to various embodiments includes an operation of obtaining a plurality of frames including a plurality of subjects through a camera 330 and the plurality of frames through a sensor circuit 340. Checking depth information on the plurality of subjects included in the frames of, based on the depth information on the plurality of subjects, first depth information among the plurality of subjects from the plurality of frames The holding may include an operation of removing the first subject, and an operation of generating an image by synthesizing a plurality of frames from which the first subject is removed.

In various embodiments, the plurality of frames obtained through the camera 330 may include a plurality of frames captured at different points of time according to the movement of the electronic device 301 .

In various embodiments, the depth information of the plurality of subjects may include distance information between each of the plurality of subjects from the camera 330 and/or histogram information according to the distance information.

In various embodiments, the histogram information may include the number of pixels for each distance to a plurality of subjects included in each of the plurality of frames.

The operation of removing the first subject having the first depth information according to various embodiments may include distance information between the camera 330 and each of the plurality of subjects and/or the plurality of subjects according to the distance information An operation of checking the number of pixels by distance to the plurality of subjects, and based on the distance information and / or the number of pixels by distance to the plurality of subjects, the first having the first depth information among the plurality of subjects An operation of determining a subject may be included.

The operation of determining the first subject having the first depth information according to various embodiments may include determining a subject located at a short distance from the camera among the plurality of subjects and/or a small number of pixels among the plurality of subjects. An operation of determining a subject having the first depth information as a first subject having the first depth information.

The operation of checking the depth information of the plurality of subjects according to various embodiments may include an operation of setting coordinate values on 3D space coordinates for the plurality of frames through the sensor circuit 340. can

An image processing method based on depth information of an electronic device 301 according to various embodiments includes an operation of mapping coordinate values on 3D spatial coordinates of each of the plurality of frames to pixel coordinates and RGB values of each frame may further include.

In the image processing method based on the depth information of the electronic device 301 according to various embodiments, after removing the first subject having the first depth information, the first subject having the first depth information is removed Based on the coordinate values of the region, an operation of identifying at least one other frame including a pixel having an RGB value among the plurality of frames, and based on the identified RGB values of the at least one other frame, each An operation of interpolating a region from which the first subject is removed from the frame may be further included.

The generating of the image according to various embodiments may include generating the image by synthesizing a plurality of frames obtained by interpolating a region from which the first subject is removed.

In various embodiments, the generated image may include a second subject having second depth information among the plurality of subjects.

5A and 5B are diagrams 500 and 550 for explaining a method of acquiring a plurality of frames including a plurality of subjects at different viewpoints in an obstruction removal mode in panoramic photography, according to various embodiments.

Referring to FIG. 5A , an electronic device (eg, the electronic device 301 of FIG. 3 ) may acquire a plurality of frames including a plurality of subjects through a camera (eg, the camera 330 of FIG. 3 ). For example, the plurality of subjects may include a first subject 505 and a second subject 510 . The first subject 505 may include a subject located at a first distance 520 from the camera 330 . The second subject 510 may include a subject located at a second distance 515 greater than the first distance 520 from the camera 330 .

In one embodiment, when set to the obstruction removal mode in panoramic photography, the electronic device 301 may provide a panorama view path for guiding movement in a specific direction (eg, a horizontal direction and/or a vertical direction). The electronic device 301 may detect movement 525 of the electronic device 301, and based on the movement 525 of the electronic device 301, a plurality of subjects photographed at different times through the camera 330. It is possible to obtain a plurality of frames including . For example, a plurality of frames photographed at different viewpoints may include a first frame of the first range 530 obtained at a first viewpoint, a second frame of the second range 535 obtained at a second viewpoint, and/or a second frame of the second range 535 obtained at a second viewpoint. It may include the third frame of the third range 540 obtained at the third viewpoint. In one embodiment, the first range 530, the second range 535, and the third range 540 are obtained at different points in time (eg, a shooting point) based on the movement 525 of the electronic device 301. It may refer to the range of the angle of view of the camera 330 to be.

In various embodiments, it has been described assuming that the plurality of frames are the first frame of the first range 530, the second frame of the second range 535, and/or the third frame of the third range 540. , but is not limited thereto. The plurality of frames may include frames exceeding three frames.

In various embodiments, as the electronic device 301 moves 525 in a specific direction, for example, a horizontal direction (and/or a vertical direction), a plurality of frames obtained at different times may have areas overlapping each other. . For example, the first frame of the first range 530 and the second frame of the second range 535 may include partially overlapping regions, and the second frame of the second range 535 and the third frame 540 of the second range 535 may include overlapping regions. The third frame of ) may include some overlapping regions.

Referring to FIG. 5B , the electronic device 301, for example, a first subject 505 and a second subject 505 in a first range 530 through a camera 330 as shown by reference number <551>. A first frame including the subject 510 may be obtained. In the first frame according to an embodiment, the second subject 510 may include numbers “2”, “3”, and “4”. A partial area of the second subject 510 including the numbers “2”, “3”, and “4” may be covered by the first subject 505 .

For another example, the electronic device 301 moves 525 of the electronic device 301 as shown in reference numeral <553> through the camera 330 to view a plurality of subjects in the second range 535, for example , a second frame including the first subject 505 and the second subject 510 may be acquired. In the second frame according to an embodiment, the second subject 510 may include the numbers “3”, “4”, and “5”. A partial area of the second subject 510 including the numbers “3”, “4”, and “5” may be covered by the first subject 505 .

As another example, as shown in reference number <555>, the electronic device 301 moves 525 to capture a plurality of subjects in the third range 540 through the camera 330. For example, a third frame including the first subject 505 and the second subject 510 may be acquired. In the third frame according to an embodiment, the second subject 510 may include parts of the numbers “4”, “5”, “6”, and “7”. A partial area of the second subject 510 including the numbers “4”, “5”, “6”, and “7” may be covered by the first subject 505 .

In various embodiments, the electronic device 301 based on depth information on a plurality of subjects included in a plurality of frames through a sensor circuit (eg, the sensor circuit 340 of FIG. 3 ), A subject corresponding to an obstacle may be determined among a plurality of subjects.

Various embodiments will be described with reference to FIGS. 5C and 6 to be described later with respect to an operation of determining a subject corresponding to an obstacle among a plurality of subjects based on depth information of the plurality of subjects described above.

FIG. 5C is a diagram 580 for explaining a method of generating a depth map for a plurality of frames including a plurality of subjects in an obstruction removal mode in panoramic photography, according to various embodiments.

Referring to FIG. 5C, an electronic device (eg, the electronic device 301 of FIG. 3) uses a sensor circuit (eg, the sensor circuit 340), for example, a depth sensor (eg, the depth sensor 341 of FIG. 3). Thus, depth information of a plurality of subjects may be checked. For example, the electronic device 301 may generate a depth map for a plurality of frames obtained through a camera (eg, the camera 330 of FIG. 3 ) using the depth sensor 341 .

For example, as shown in reference number <581>, the electronic device 301 provides a depth map for the first frame of the first range 530 (eg, the first frame shown in reference number <551> in FIG. 5B). depth map) can be created. For another example, as shown in reference number <583>, the electronic device 301 provides a depth map for the second frame of the second range 535 (eg, the depth map shown in reference number <553> in FIG. 5B). A depth map for the second frame) may be generated. For another example, as shown in reference number <585>, the electronic device 301 provides a depth map for the third frame of the third range 540 (eg, shown in reference number <555> in FIG. 5B). A depth map for the third frame) may be generated.

In FIG. 5C according to various embodiments, a first subject 505 positioned at a first distance 520 from the camera 330 among a plurality of subjects may be represented by a light shade, and may be larger than the first distance 520. The second subject 510 located at the second distance 515 may be represented by a dark shade.

In an embodiment, the electronic device 301 provides a depth map for a first frame in the first range 530 (eg, see reference number <581>) and a depth map for a second frame in the second range 535. (eg, reference number <583>), and/or depth map for the third frame of the third range 540 (eg, reference number <585>), depth information about a plurality of subjects can be checked.

In an embodiment, the depth information of the plurality of subjects may include distance information between the camera 330 and each of the plurality of subjects. For example, the electronic device 301 detects the entire area of the camera 330 and an image sensor included in the camera 330 (eg, the image sensor 230 of FIG. 2 ) or Distance information between each of a plurality of subjects corresponding to a partial area may be extracted. The electronic device 301 determines at least one subject (eg, a first distance 520 from the camera 330) located at a short distance among the plurality of subjects, based on distance information between the camera 330 and each of the plurality of subjects. It is possible to check the first subject 505 located at .

In an embodiment, the electronic device 301 may identify at least one subject having the same property in a frame including a plurality of subjects by using an RGB sensor (not shown) included in the camera 330 . For example, attributes may include color and/or shape (eg, shape and/or area). The electronic device 301 may check whether at least one subject having the same attribute satisfies a specified condition. For example, the specified condition may include a condition in which the distance between the camera 330 and at least one subject having the same attribute information is equal to or smaller than the distance between the camera 330 and at least one other subject. For example, when distance information of at least one subject having the same property satisfies a specified condition, the electronic device 301 designates an area including at least one subject having the same property, and designates an area including at least one subject having the same property, and designates an area including at least one subject having the same property, may be identified as a subject of (eg, a first subject 505 located at a first distance 520 from the camera 330).

In various embodiments, depth information on a plurality of subjects may further include histogram information according to distance information. The histogram information may include the number of pixels for each distance to a plurality of subjects included in each frame. The electronic device 301 determines at least one subject (eg, a first distance 520 from the camera 330) located at a short distance based on distance information and/or histogram information according to the distance information for a plurality of subjects. A positioned first subject 505 may be identified.

A configuration for checking at least one subject (eg, a first subject 505 located at a first distance 520 from the camera 330) based on the number of pixels for each distance of the plurality of subjects described above In this regard, various embodiments will be described in FIG. 6 to be described later.

FIG. 6 is a graph 600 for explaining a method of determining whether an object located at a short distance from the camera 330 is an obstacle based on the number of pixels for each distance of a plurality of objects, according to various embodiments.

Referring to FIG. 6 , an electronic device (eg, the electronic device 301 of FIG. 3 ) generates a depth map (eg, FIG. 5C ) for a first frame of a first range (eg, the first range 530 of FIG. 5A ). Reference number <581> of), a depth map for the second frame of the second range (eg, second range 535 of FIG. 5A) (eg, reference number <583> of FIG. 5C), and/or Based on the depth map (eg, see reference number <585> of FIG. 5C) of the third frame of the third range (eg, the third range 540 of FIG. 5A), depth information, for example, included in each frame The number of pixels of a plurality of subjects may be checked. For example, the electronic device 301 may check the number of pixels 620 (eg, depth distribution) according to distance information 610 between the camera 330 and each of a plurality of subjects.

In an embodiment, the electronic device 301 selects a plurality of subjects at a short distance (eg, a first distance (eg, a first distance ( 520)) and a subject located at a long distance (eg, the second distance 515) may be distinguished.

For example, as shown in FIG. 6 , a plurality of subjects may be separated at regular intervals as they are located at different distances from the camera 330 . The electronic device 301 includes a first subject 505 (eg, a subject located at a short distance (eg, about 30 to 50 cm)) located at a first distance 520 from the camera 330 and a subject located at a distance 520 from the camera 330. A second subject 510 (eg, a subject located at a long distance (eg, about 180 to 250 cm)) located at a second distance 515 greater than the first distance 520 may be identified.

In an embodiment, the electronic device 301 determines the first subject located at the first distance 520 based on the number of pixels 620 of the plurality of subjects included in each frame according to the distance information 610. 505 (eg, a subject located at a short distance (eg, about 30 to 50 cm)) may be determined as an obstacle (eg, the first subject 505 having the first depth information determined in operation 461 of FIG. 4B).

In various embodiments, the electronic device 301 may determine the first subject 505 located at the first distance 520 as an obstacle and perform operations 463 to 473 of FIG. 4B .

7A is a method of removing a first subject having first depth information from a plurality of frames in an obstruction removal mode in a panoramic photographing and interpolating a region of a frame from which the first subject is removed, according to various embodiments; It is a drawing 700 for explaining.

In various embodiments, an electronic device (eg, electronic device 301 of FIG. 3 ) is determined to be an obstacle according to embodiments of FIGS. 5A to 6 (eg, camera 330 of FIG. 3 ) at a short distance ( Example: A subject located at a first distance 520 of FIG. 5A , for example, a first subject (eg, first subject 505 of FIG. 5A ) may be removed from a plurality of frames.

In various embodiments, the electronic device 301 may activate an inertial sensor (eg, the inertial sensor 343 of FIG. 3 ) when acquiring a plurality of frames through the camera 330 . The processor 360 may set coordinate values on spatial coordinates for a plurality of frames using the inertial sensor 343 . The 3D spatial coordinates allocated to each frame may be matched with pixel coordinates and RGB values of each frame. For example, a part of a first frame (eg, a first frame shown by reference numeral <551> in FIG. 5B) of a first range (eg, first range 530 of FIG. 5A), a second range (eg, figure Part of the second frame (e.g., the second frame shown in reference numeral <553> of FIG. 5B) of the second range 535 of FIG. 5A, and/or the third range (e.g., the third range of FIG. 5A ( 540) may have overlapping regions, and the overlapping regions may have the same coordinate values.

In an embodiment, the electronic device 301 detects a first subject having first depth information in each of a plurality of frames (eg, 711a, 711b, and 711c in the first frame, 721a, 721b, and 721c in the second frame, and A background pixel value of an area from which 731a, 731b, and 731c) of the third frame are removed may be interpolated and generated. For example, the electronic device 301 may identify at least one other frame including a pixel having an RGB value among a plurality of frames based on the coordinate values of the region from which the first subject is removed in each frame. The electronic device 301 determines the first subject having the first depth information in each frame (eg, 711a, 711b, and 711c in the first frame, and 721a in the second frame) based on the identified RGB values of at least one other frame. , 721b, 721c, and 731a, 731b, 731c of the third frame) may be interpolated.

For example, referring to FIG. 7A , the electronic device 301, as shown by reference number <710>, first subjects 711a and 711b having first depth information from the first frame of the first range 530 , 711c) can be obtained. The electronic device 301 identifies at least one other frame in which RGB values exist based on the coordinate values of the regions 711a, 711b, and 711c from which the first subject having the first depth information is removed from the first frame, The areas 711a, 711b, and 711c from which the first subject is removed may be reconstructed based on at least one other frame having RGB values. For example, the electronic device 301 based on the coordinate values of the regions 711a, 711b, and 711c from which the first subject having the first depth information in the first frame shown in reference number <710> has been removed, reference number <710>. 720> and/or the third frame shown in reference number <730>, pixels having RGB values corresponding to coordinate values of regions 711a, 711b, and 711c from which the first subject is removed are selected. You can check the containing frame. The electronic device 301 converts the regions 711b and 711c from which the first subject having the first depth information is removed in the first frame to at least one other frame region corresponding thereto, for example, the region 722a of the second frame. 722b) and interpolation based on the area 732a of the third frame.

For another example, as shown in reference number <720>, the electronic device 301 selects the first subjects 721a, 721b, and 721c having the first depth information from the second frame of the second range 535. Removed frames can be obtained. The electronic device 301 determines reference number <710> based on the coordinate values of the areas 721a, 721b, and 721c from which the first subject having the first depth information is removed in the second frame shown in reference number <720>. RGB values corresponding to coordinate values of regions 721a, 721b, and 721c from which the first subject having the first depth information is removed in the first frame shown in and/or the third frame shown in reference number <730> are You can check frames that contain existing pixels. For example, the electronic device 301 converts the areas 721a, 721b, and 721c from which the first subject having the first depth information is removed in the second frame to at least one other frame corresponding to the areas 721a, 721b, and 721c of the first frame. Interpolation may be performed based on the regions 712a and 712b and the regions 733a and 733b of the third frame.

As another example, the electronic device 301, as shown in reference number <730>, the first subject 731a, 731b, 731c having the first depth information from the third frame of the third range 540 It is possible to obtain a frame from which is removed. The electronic device 301 is shown in reference number <710> based on the coordinate values of the regions 731a and 731b from which the first subject having the first depth information is removed in the third frame shown in reference number <730>. In the first frame and/or the second frame indicated by reference number <720>, a frame including pixels having RGB values corresponding to coordinate values of regions 731a and 731b from which the first subject is removed may be identified. have. For example, the electronic device 301 converts the areas 731a and 731b from which the first subject having the first depth information is removed in the third frame to at least one other frame area corresponding thereto, for example, the area of the first frame ( 713a) and interpolation may be performed based on the regions 723a and 723b of the second frame.

FIG. 7B is a diagram 735 for explaining a method of generating an image by synthesizing interpolated frames in an obstruction removal mode in panoramic photography, according to various embodiments.

Referring to FIG. 7B, an electronic device (for example, the electronic device 301 of FIG. 3) is a first frame in which an area from which a first subject is removed shown by reference number <740> is interpolated, shown by reference number <750>. An image (e.g., a panoramic image) is synthesized by synthesizing a second frame in which the area from which the first subject has been removed is interpolated and/or a third frame in which the area from which the first subject is removed shown in reference number <760> is interpolated. can create

In an embodiment, the electronic device 301 may display the created image (eg, a panoramic image) on a display (eg, the display 351 of FIG. 3 ), as shown by reference number <770>. For example, in an image (eg, a panoramic image) displayed on the display 351, a first subject (eg, the first subject 505 of FIG. 5A ) having first depth information is removed, and a second subject having second depth information An image including only the second subject (eg, the second subject 510 of FIG. 5A) may be included.

8A and 8B are diagrams 800 and 850 for explaining a method of obtaining a plurality of frames including a plurality of subjects at different viewpoints in an obstacle removal mode in normal photographing, according to various embodiments.

Referring to FIG. 8A , an electronic device (eg, the electronic device 301 of FIG. 3 ) may acquire a plurality of frames including a plurality of subjects through a camera (eg, the camera 330 ). For example, the plurality of subjects may include a first subject 805 and a second subject 810 . The first subject 805 may include a subject located at a first distance 820 from the camera 330 . The second subject 810 may include a subject located at a second distance 815 greater than the first distance 820 from the camera 330 .

In one embodiment, the electronic device 301 performs a motion gesture (eg, in a specific direction) in a specific direction (eg, a horizontal direction and/or a vertical direction) via a sensor circuit (eg, the sensor circuit 340 of FIG. 3 ). In response to detecting a gesture of shaking the electronic device 301 , a plurality of frames photographed at different viewpoints may be obtained through the camera 330 . For example, a plurality of frames photographed at different viewpoints may include a first frame of the first range 830 obtained at a first viewpoint, a second frame of the second range 835 obtained at a second viewpoint, and/or a second frame of the second range 835 obtained at a second viewpoint. It may include the third frame of the third range 840 obtained at the third viewpoint.

In various embodiments, a plurality of frames acquired at different times by a motion gesture in a specific direction (eg, a horizontal direction and/or a vertical direction) may have areas overlapping each other. For example, a partial area of the first frame, a partial area of the second frame, and/or a partial area of the third frame may overlap each other.

Referring to FIG. 8B , in response to detecting a motion gesture of the electronic device 301, the electronic device 301 uses a camera 330 as shown in reference numerals <851>, <853>, and <855>. A first frame including a plurality of subjects (eg, the first subject 805 and the second subject 810) of the first range 830 and a plurality of subjects (eg, the second range 835) through : A second frame including the first subject 805 and the second subject 810, and a plurality of subjects in the third range 840 (eg, the first subject 805 and the second subject 810) ) It is possible to obtain a third frame including.

In a plurality of frames including a plurality of subjects acquired at different viewpoints according to various embodiments, the second subject 810 includes the numbers “3”, “4”, “5”, and/or “6”. can do. A partial area of the second subject 810 including the numbers “3”, “4”, “5”, and/or “6” may be covered by the first subject 805 .

In various embodiments, the electronic device 301 based on depth information on a plurality of subjects included in a plurality of frames through a sensor circuit (eg, the sensor circuit 340 of FIG. 3 ), A subject corresponding to an obstacle may be determined among a plurality of subjects.

FIG. 8C is a diagram 880 for explaining a method of generating a depth map for a plurality of frames including a plurality of subjects in an obstruction removal mode in normal shooting, according to various embodiments.

Referring to FIG. 8C , an electronic device (eg, the electronic device 301 of FIG. 3 ) uses a sensor circuit (eg, the sensor circuit 340 ), for example, a depth sensor (eg, the depth sensor 341 of FIG. 3 ). Thus, depth information of a plurality of subjects may be checked. For example, the electronic device 301 may generate a depth map for a plurality of frames acquired through the camera 330 using the depth sensor 341 .

For example, as shown in reference number <881>, the electronic device 301 generates a depth map for the first frame of the first range 830 (eg, a depth map for reference number <851> in FIG. 8B). can create For another example, as shown in reference number <883>, the electronic device 301 provides a depth map for the second frame of the second range 835 (eg, a depth map for reference number <583> in FIG. 8B). map) can be created. For another example, as shown in reference number <885>, the electronic device 301 provides a depth map for a third frame of the third range 840 (eg, a depth map for reference number <855> in FIG. 8B). depth map).

In FIG. 8C according to various embodiments, a first subject 805 located at a first distance 820 from the camera 330 may be represented by a light shade among a plurality of subjects, and a distance greater than the first distance 820 The second subject 810 located at the second distance 815 may be represented by a dark shade.

In an embodiment, the electronic device 301 provides a depth map for a first frame in the first range 830 (eg, see reference number <881>) and a depth map for a second frame in the second range 835. (eg, see reference number <883>) and/or based on a depth map (eg, see reference number <885>) for the third frame of the third range 840, depth information about a plurality of subjects can be checked.

In an embodiment, the depth information of the plurality of subjects may include distance information between the camera 330 and each of the plurality of subjects and/or histogram information according to the distance information (eg, a plurality of subjects included in each frame). number of pixels per distance). The electronic device 301 selects at least one subject (eg, a camera ( 330), a first subject 805 located at a first distance 820 may be identified.

In various embodiments, after checking at least one subject (eg, a first subject 805 located at a first distance 820 from the camera 330) located at a short distance among a plurality of subjects, Operations 463 to 473 of 4b may be performed.

9A is a method of removing a first subject having first depth information from a plurality of frames in an obstruction removal mode in normal shooting and interpolating a region of a frame from which the first subject is removed, according to various embodiments; It is a drawing 900 for explaining.

In various embodiments, an electronic device (eg, electronic device 301 of FIG. 3 ) is at a short distance from a camera (eg, camera 330 of FIG. 3 ) that is determined to be an obstruction according to the embodiments of FIGS. 8A-8C . A positioned subject, for example, a first subject (eg, a first subject having first depth information) may be removed from a plurality of frames.

In various embodiments, when acquiring a plurality of frames through the camera 330, the electronic device 301 uses an inertial sensor (eg, the inertial sensor 343 of FIG. 3 ) to obtain 3 frames for the plurality of frames. Coordinate values on dimensional space coordinates can be set. The electronic device 301 may allocate coordinate values on 3D space coordinates to a plurality of frames, and match pixel coordinates and RGB values of each frame to the allocated coordinate values.

In an embodiment, the electronic device 301 may display a first subject having first depth information in each of a plurality of frames (eg, 911a, 911b, and 911c in the first frame, 921a, 921b, and 921c in the second frame, and A background pixel value of an area where 931a and 931b) of the third frame are removed may be interpolated and generated. For example, the electronic device 301 may identify at least one other frame including a pixel having an RGB value based on the coordinate values of the region from which the first subject is removed in each frame. The electronic device 301 determines the first subject having the first depth information in each frame (eg, 911a, 911b, and 911c in the first frame, and 921a in the second frame) based on the identified RGB values of at least one other frame. , 921b, 921c, and 931a, 931b of the third frame) may be interpolated.

For example, referring to FIG. 9A , the electronic device 301, as shown by reference number <910>, first subjects 911a and 911b having first depth information from the first frame of the first range 830 , 911c) can be obtained. The electronic device 301 identifies at least one other frame in which RGB values exist based on the coordinate values of the regions 911a, 911b, and 911c from which the first subject having the first depth information is removed from the first frame, The regions 911a, 911b, and 911c from which the first subject is removed may be reconstructed based on at least one other frame having RGB values. For example, the electronic device 301 based on the coordinate values of the regions 911a, 911b, and 911c from which the first subject having the first depth information is removed in the first frame shown in reference number <910>, reference number < 920> and/or the third frame shown in reference number <930>, pixels having RGB values corresponding to coordinate values of regions 911a, 911b, and 911c from which the first subject has been removed are selected. You can check the containing frame. The electronic device 301 converts the areas 911a, 911b, and 911c from which the first subject having the first depth information is removed in the first frame to at least one other frame area 922a corresponding to the area 922a of the second frame. ) and the regions 932a and 932b of the third frame.

For another example, as shown in reference number <920>, the electronic device 301 selects the first subjects 921a, 921b, and 921c having the first depth information from the second frame of the second range 835. Removed frames can be obtained. The electronic device 301 determines reference number <910> based on the coordinate values of the regions 921a, 921b, and 921c from which the first subject having the first depth information is removed in the second frame shown in reference number <920>. RGB values corresponding to coordinate values of regions 921a, 921b, and 921c from which the first subject having the first depth information is removed in the first frame shown in and/or the third frame shown in reference number <930> are You can check frames that contain existing pixels. For example, the electronic device 301 converts the areas 921a, 921b, and 921c from which the first subject having the first depth information is removed in the second frame to at least one other frame area corresponding thereto, for example, the first frame area. Interpolation may be performed based on (912a, 912b, 912c) and areas (933a, 932c) of the third frame.

For another example, as shown in reference number <930>, the electronic device 301 removes the first subjects 931a and 931b having the first depth information from the third frame of the third range 840. frame can be obtained. The electronic device 301 is shown in reference number <910> based on the coordinate values of the regions 931a and 931b from which the first subject having the first depth information is removed in the third frame shown in reference number <930>. In the first frame and/or the second frame indicated by reference number <920>, a frame including pixels having RGB values corresponding to the coordinate values of the areas 931a and 931b from which the first subject has been removed may be identified. have. For example, the electronic device 301 converts the areas 931a and 931b from which the first subject having the first depth information is removed in the third frame to at least one other frame area corresponding thereto, for example, the area 913a of the first frame. ) and the area 923a of the second frame.

9B is a diagram 935 for explaining a method of generating an image by synthesizing interpolated frames in an obstruction removal mode in normal shooting, according to various embodiments.

Referring to FIG. 9B , an electronic device (e.g., the electronic device 301 of FIG. 3) is a first frame in which an area from which a first subject is removed, indicated by reference number <940>, is interpolated, shown by reference number <950>. An image may be generated by synthesizing a second frame obtained by interpolating an area from which the first subject is removed, and/or a third frame obtained by interpolating an area from which the first subject is removed, shown in reference number <960>.

In one embodiment, the electronic device 301 may display the generated image on a display (eg, the display 351 of FIG. 3 ), as indicated by reference number <970>. For example, in the image displayed on the display 351, the first subject having the first depth information (eg, the first subject 805 of FIG. 8A ) is removed, and the second subject having the second depth information (eg, the first subject 805 of FIG. 8A ) An image including only the second subject 810 of FIG. 8A may be included.

In various embodiments, the composition of the camera 330 is determined by a distance difference between a second subject (eg, a target subject located at a distance) and a first subject (eg, an obstacle located at a short distance) covering the second subject. By using the fact that the area of the second subject covered by the first subject can change, a plurality of frames acquired at different times according to the movement (or motion gesture) of the electronic device 301 through the camera 330 are displayed. By removing the first subject and interpolating the area from which the first subject is removed based on the other frame, an image including only the main subject can be easily obtained.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

301: electronic device 310: communication circuit
320: memory 330: camera
340: sensor circuit 350: touch screen display
360: processor

Claims (20)

In electronic devices,
camera;
sensor circuit; and
a processor operatively connected with the camera and the sensor circuit;
the processor,
Obtaining a plurality of frames including a plurality of subjects through the camera,
Checking depth information on the plurality of subjects included in the plurality of frames through the sensor circuit;
removing a first subject having first depth information among the plurality of subjects from the plurality of frames based on the depth information of the plurality of subjects; and
An electronic device configured to generate an image by synthesizing a plurality of frames from which the first subject is removed. According to claim 1,
The plurality of frames acquired through the camera include a plurality of frames captured at different points of time according to the movement of the electronic device. According to claim 1,
The electronic device of claim 1 , wherein the depth information of the plurality of subjects includes distance information between each of the plurality of subjects from the camera and/or histogram information according to the distance information. According to claim 3,
The histogram information includes the number of pixels for each distance to the plurality of subjects included in each of the plurality of frames. According to claim 4,
the processor,
Checking distance information between the camera and each of the plurality of subjects and/or the number of pixels for each distance to the plurality of subjects according to the distance information, and
The electronic device configured to determine the first subject having the first depth information among the plurality of subjects based on the distance information and/or the number of pixels per distance to the plurality of subjects. According to claim 5,
the processor,
The electronic device configured to determine a subject located at a short distance from the camera among the plurality of subjects and/or a subject having a small number of pixels among the plurality of subjects as the first subject having the first depth information. According to claim 1,
the processor,
An electronic device that sets coordinate values on three-dimensional space coordinates for the plurality of frames through the sensor circuit. According to claim 7,
the processor,
An electronic device configured to map coordinate values on three-dimensional spatial coordinates of each of the plurality of frames to pixel coordinates and RGB values of each frame. According to claim 8,
the processor,
Identifying at least one other frame including a pixel having an RGB value among the plurality of frames based on coordinate values of an area from which the first subject having the first depth information is removed; and
The electronic device configured to interpolate an area from which the first subject is removed in each frame based on the identified RGB values of the at least one other frame. According to claim 9,
the processor,
It is set to generate the image by synthesizing a plurality of frames obtained by interpolating an area from which the first subject is removed,
The generated image includes a second subject having second depth information among the plurality of subjects. A method for processing an image based on depth information of an electronic device,
obtaining a plurality of frames including a plurality of subjects through a camera;
checking depth information of the plurality of subjects included in the plurality of frames through a sensor circuit;
removing a first subject having first depth information among the plurality of subjects from the plurality of frames based on the depth information of the plurality of subjects; and
and generating an image by synthesizing a plurality of frames from which the first subject is removed. According to claim 11,
The plurality of frames acquired through the camera include a plurality of frames captured at different points of time according to the movement of the electronic device. According to claim 11,
The depth information of the plurality of subjects includes distance information between each of the plurality of subjects from the camera and/or histogram information according to the distance information. According to claim 13,
The histogram information includes the number of pixels for each distance to a plurality of subjects included in each of the plurality of frames. 15. The method of claim 14,
The operation of removing the first subject having the first depth information,
checking distance information between the camera and each of the plurality of subjects and/or the number of pixels for each distance to the plurality of subjects according to the distance information; and
and determining the first subject having the first depth information among the plurality of subjects based on the distance information and/or the number of pixels per distance to the plurality of subjects. According to claim 15,
The operation of determining the first subject having the first depth information,
and determining, among the plurality of subjects, a subject located at a short distance from the camera and/or a subject having a small number of pixels among the plurality of subjects as a first subject having the first depth information. According to claim 11,
The operation of checking the depth information for the plurality of subjects,
and setting coordinate values on 3D spatial coordinates for the plurality of frames through the sensor circuit. 18. The method of claim 17,
The method further comprises mapping coordinate values on 3D spatial coordinates of each of the plurality of frames to pixel coordinates and RGB values of each frame. According to claim 18,
After removing the first subject having the first depth information, pixels having RGB values among the plurality of frames are selected based on coordinate values of an area from which the first subject having the first depth information is removed. checking at least one other frame including; and
The method further comprises interpolating an area from which the first subject is removed in each frame based on the identified RGB values of the at least one other frame. According to claim 19,
The operation of generating the image,
generating the image by synthesizing a plurality of frames obtained by interpolating an area from which the first subject is removed;
The generated image includes a second subject having second depth information among the plurality of subjects.

Images