WO2020171450A1 - Electronic device and method for generating depth map - Google Patents

Abstract

Disclosed is an electronic device. The electronic device comprises: a lens assembly; a first pixel array aligned along a first direction; a second pixel array aligned along the first direction; a first micro lens disposed between the lens assembly and the first pixel array and between the lens assembly and the second pixel array, and covering the first pixel array and the second pixel array; a processor; and a memory operatively connected to the processor, wherein the memory stores instructions that, when executed, cause the processor to provide boundary information identifying a first boundary of a subject, by using a first image signal and a second image signal, wherein, as light emitted from at least one light-emitting element is reflected by the subject and made incident upon the lens assembly, the first image signal is generated by sensing, by the first pixel array, a first incident light collected through the first micro lens, from among the incident light, and the second image signal is generated by sensing, by the second pixel array, the first incident light; provide first distance information identifying a first distance to a first point of the subject, by comparing a second incident light among the incident light, to the light; and generate a depth map on the basis of the first distance information and the boundary information. In addition, various other embodiments identified through the specification are possible.

Description

Electronic device and method for generating depth map

Embodiments disclosed in this document relate to an electronic device and a method for generating a depth map.

Recently, portable devices (eg, cameras, mobile communication terminals, etc.) including an image sensor have been developed. The image sensor may include at least one or more pixels. A pixel of the image sensor may generate an image signal by sensing incident light incident on a lens and converting photons corresponding to a specific color into electrons.

The 3D image data may include depth map information including distance information of a subject. The 3D image data can be used in various technical fields, such as motion detection based on images and user recognition based on images. In order to generate more accurate 3D image data, more accurate depth map generation methods are being studied.

For generating the depth map, various methods can be used. For example, the electronic device may use a plurality of sensors capable of generating stereoscopic information corresponding to binocular parallax. For another example, the electronic device may generate a depth map based on the reflection time of the reflected light. For another example, in order to improve the accuracy of the depth map, the electronic device may generate the depth map using both stereoscopic information and echo time. In general, different types of distance information (eg, stereoscopic information and reverberation time information) use different physical characteristics. Accordingly, sensors for sensing different types of distance information may have separate hardware configurations. In this case, the electronic device may perform complex procedures to generate a depth map based on a combination of different types of distance information.

According to the exemplary embodiment disclosed in this document, by separately identifying distance information for a point of a subject and boundary information at the boundary of the subject, a sophisticated depth map in which the boundary of the subject is clearly expressed may be obtained.

An electronic device according to an embodiment disclosed in this document includes: a lens assembly, a first pixel array aligned along a first direction, a second pixel array aligned along the first direction, the lens assembly and the first pixel A first microlens disposed between arrays and between the lens assembly and the second pixel array and covering the first pixel array and the second pixel array, a processor, and a memory operatively connected to the processor. The memory includes, when executed, the processor causes the first incident light to be collected through the first microlens among incident light incident on the lens assembly by reflecting light emitted from at least one light emitting element from a subject. Identifying the first boundary of the subject using a first image signal generated by sensing by the first pixel array and a second image signal generated by sensing the first incident light by the second pixel array Provides boundary information, and provides first distance information for identifying a first distance to a first point of the subject by comparing the light and a second incident light of the incident light, and the first distance information and the boundary information It stores instructions to create a depth map based on.

In addition, the method according to an exemplary embodiment disclosed in the present document includes light emitted from at least one light emitting device and the light reflected from a subject to a lens assembly by using an edge detection circuitry of an electronic device. An operation of identifying the first boundary of the subject and providing boundary information based on the intensity of the first incident light among the incident light, using a distance measurement circuitry of the electronic device, Providing first distance information for identifying a first distance to a first point of the subject by comparing the second incident light with and based on the first distance information and the boundary information, the depth map It may include an operation of generating.

In addition, the electronic device according to an embodiment disclosed in the present document includes, based on the intensity of the first incident light among incident light incident on the lens assembly by reflecting light emitted from at least one light emitting device from a subject, the A plenoptic circuitry that identifies a first boundary of a subject to provide boundary information, and a first to identify a first distance to a first point of the subject by comparing the light with a second incident light of the incident light A time of flight (TOF) circuit for providing distance information, a processor, and a memory operatively connected to the processor, the memory configured to cause the processor to, when executed, cause the first distance information and the boundary Instructions for generating a depth map using the information may be stored.

According to an embodiment disclosed in this document, distance information for identifying a distance to a point of a subject and border information for identifying a boundary of a subject are separately provided by using incident light incident on the subject by reflecting light emitted from the light emitting device. By detecting, an elaborate depth map capable of identifying the boundary of the subject can be implemented.

In addition to this, various effects that are directly or indirectly identified through this document can be provided.

1 is a block diagram of an electronic device in a network environment according to an embodiment disclosed in this document.

2 is a diagram illustrating an electronic device according to an embodiment disclosed in this document.

3A is a view for explaining the lens assembly and the image sensor of FIG. 2.

FIG. 3B is a diagram illustrating an arrangement of a lens assembly, a micro lens array, and an image sensor of FIG. 2.

4 is a diagram for describing the lens assembly and the image sensor of FIG. 2.

5A is a diagram illustrating the processor of FIG. 2.

FIG. 5B is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information on a subject according to an exemplary embodiment disclosed in the present document.

FIG. 5C is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information about a subject according to an embodiment disclosed in this document.

6 is a flowchart illustrating a method of generating a depth map by an electronic device according to an embodiment disclosed in this document.

7 is a diagram for explaining incident light according to an embodiment disclosed in this document.

8 is a view for explaining the operation of FIG. 6.

9 is a diagram for explaining the operation of FIG. 6.

10A is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.

10B is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.

10C is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.

10D is a diagram illustrating a user interface provided by an electronic device according to an embodiment disclosed in this document.

In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present invention.

1 is a block diagram of an electronic device 101 in a network environment 100 according to an embodiment disclosed in this document. Referring to FIG. 1, in a network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (for example, a short-range wireless communication network), or a second network 199 It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) Can be included. In some embodiments, at least one of these components may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, a program 140) to implement at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to an embodiment, as at least a part of data processing or operation, the processor 120 stores commands or data received from other components (eg, the sensor module 176 or the communication module 190) to the volatile memory 132 The command or data stored in the volatile memory 132 may be processed and result data may be stored in the nonvolatile memory 134. According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and an auxiliary processor 123 (eg, a graphics processing unit, an image signal processor) that can be operated independently or together. , A sensor hub processor, or a communication processor). Additionally or alternatively, the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.

The coprocessor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, an application is executed). ) While in the state, together with the main processor 121, at least one of the components of the electronic device 101 (for example, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the functions or states related to. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176). The data may include, for example, software (eg, the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive a command or data to be used for a component of the electronic device 101 (eg, the processor 120) from an outside (eg, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output an sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry set to sense a touch, or a sensor circuit (eg, a pressure sensor) set to measure the strength of a force generated by the touch. have.

The audio module 170 may convert sound into an electric signal or, conversely, convert an electric signal into sound. According to an embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device directly or wirelessly connected to the electronic device 101 (for example, Sound may be output through the electronic device 102) (for example, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 176 is, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used for the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that a user can perceive through a tactile or motor sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture a still image and a video. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment, the power management module 388 may be implemented as at least a part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, electronic device 102, electronic device 104, or server 108). It is possible to support establishment and communication through the established communication channel. The communication module 190 operates independently of the processor 120 (eg, an application processor), and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A LAN (local area network) communication module, or a power line communication module) may be included. Among these communication modules, a corresponding communication module is a first network 198 (for example, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 199 (for example, a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (for example, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information stored in the subscriber identification module 196 (eg, International Mobile Subscriber Identifier (IMSI)) within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be checked and authenticated.

The antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive from the outside. According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, provided by the communication module 190 from the plurality of antennas. Can be chosen. The signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as a part of the antenna module 197.

At least some of the components are connected to each other through a communication method (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices and signals ( E.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be a device of the same or different type as the electronic device 101. According to an embodiment, all or part of the operations executed by the electronic device 101 may be executed by one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 does not execute the function or service by itself. In addition or in addition, it is possible to request one or more external electronic devices to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 101. The electronic device 101 may process the result as it is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology Can be used.

2 is a diagram illustrating an electronic device 201 according to an embodiment disclosed in this document.

Referring to FIG. 2, an electronic device 201 (eg, the electronic device 101 of FIG. 1) includes a lens assembly 210, a light emitting device 220, an image sensor 230, and a memory 250 (eg: Buffer memory), a micro lens array 270, or a processor 260. The electronic device 201 according to the embodiment disclosed in this document may be a device of various types. The electronic device 201 may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment disclosed in this document is not limited to the above-described devices.

The lens assembly 210 may collect light emitted from a subject to be imaged. The lens assembly 210 may include one or more lenses. According to an embodiment, the electronic device 201 may include a plurality of lens assemblies 210. According to an embodiment, when the electronic device 201 is a camera, the electronic device 201 may form, for example, a dual camera, a 360 degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may be a different lens assembly It may have one or more lens properties different from the lens properties of. The lens assembly 210 may include, for example, a wide-angle lens and/or a telephoto lens.

The micro lens array 270 may include a plurality of micro lenses. Each of the plurality of micro lenses included in the micro lens array 270 may correspond to at least two pixel arrays. The micro lens array 270 may be disposed between the lens assembly 210 and the image sensor 230. The micro lens array 270 may collect incident light incident on the lens assembly 210 by reflecting light emitted from the light emitting device 220 from a subject.

The light-emitting device 220 may emit light used to enhance light emitted or reflected from the subject. The light-emitting device 220 may include at least one light-emitting device. According to an embodiment, the light emitting device 220 may include one or more light emitting diodes (eg, red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. In the drawing, the light emitting element 220 is shown to be disposed inside the electronic device 201, but is not limited thereto. For example, it goes without saying that the light emitting element 220 may be disposed outside the electronic device 201.

The image sensor 230 senses the light emitted or reflected from the subject and transmitted through the lens assembly 210 and the microlens array 270 and converts it into an electric signal, thereby obtaining an image signal corresponding to the subject. . According to an embodiment, the image sensor 230 is, for example, one image sensor selected from image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor. It may include a plurality of image sensors having attributes, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor 230 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

Memory 250 may be operatively connected to a processor. For example, when executed, the memory 250 may store instructions that enable the processor 260 to perform various operations according to embodiments disclosed in this document. The memory 250 may temporarily store at least a part of the image signal acquired through the image sensor 230 for the next image processing operation. For example, when image acquisition is delayed according to the shutter, or when a plurality of images are acquired at high speed, the acquired original image (eg, Bayer-patterned image or high resolution image) is stored in the memory 250 , A copy image corresponding thereto (eg, a low resolution image) may be previewed through a display device (display device 160 of FIG. 1 ). Thereafter, when a specified condition is satisfied (eg, user input or system command), at least a part of the original image stored in the memory 250 may be acquired and processed by, for example, the processor 260.

The processor 260 may perform one or more image processing on an image signal acquired through the image sensor 230 or an image signal stored in the memory 250. One or more image processing can be, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g. noise reduction, resolution adjustment, brightness adjustment, blurring). ), sharpening, or softening In addition or alternatively, the processor 260 includes at least one of the components included in the electronic device 201 (eg, the image sensor 230). ) Can be controlled (eg, exposure time control, readout timing control, etc.) The image processed by the processor 260 is stored again in the memory 250 for further processing or 201), such as the memory 130 of FIG. 1, the display device 160 of FIG. 1, the electronic device 102 of FIG. 1, the electronic device 104 of FIG. 1, or the server of FIG. 108)).

According to an embodiment, the lens assembly 210, the light emitting device 220, the image sensor 230, and the micro lens array 270 may be included in the camera module 180 of FIG. 1.

According to an embodiment, the memory 250 may be configured as at least a part of the memory 130 of FIG. 1 or as a separate memory that is independently operated.

According to an embodiment, the processor 260 may be configured as at least a part of the processor 120 of FIG. 1, or may be configured as a separate processor that is independently operated from the processor 120 of FIG. 1. When the processor 260 is configured as a separate processor from the processor 120 of FIG. 1, at least one image processed by the processor 260 is as is or undergoes additional image processing by the processor 120 of FIG. Then, it may be displayed through the display device 160 of FIG. 1.

According to an embodiment, the electronic device 201 may be the camera module 180 of FIG. 1. When the electronic device 201 is the camera module 180 of FIG. 1, the electronic device 101 of FIG. 1 includes a plurality of camera modules (eg, the camera module 180 of FIG. 1) each having different attributes or functions. Can include. In this case, for example, at least one of the plurality of camera modules (eg, the camera module 180 of FIG. 1) may be a wide-angle camera, and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules (eg, the camera module 180 of FIG. 1) may be a front camera, and at least the other may be a rear camera.

3A is a diagram illustrating the lens assembly 210 and the image sensor 230 of FIG. 2. 3B is a view for explaining the arrangement of the lens assembly 210, the micro lens array 270, and the image sensor 230 of FIG. 2.

2, 3A, and 3B, the image sensor 230 of the electronic device 201 according to the exemplary embodiment disclosed in this document includes a first pixel array PX1, a second pixel array PX2, and A third pixel array (PX3), a fourth pixel array (PX4), a fifth pixel array (PX5), a sixth pixel array (PX6), a seventh pixel array (PX7), and an eighth pixel array (PX8). I can.

First pixel array (PX1), second pixel array (PX2), third pixel array (PX3), fourth pixel array (PX4), fifth pixel array (PX5), sixth pixel array (PX6), seventh Each of the pixel array PX7 and the eighth pixel array PX8 may be aligned along the first direction X. For example, the first pixel array PX1 may include a first sub-pixel PXS1 and a second sub-pixel PXS2. The first sub-pixel PXS1 and the second sub-pixel PXS2 may be aligned along the first direction X.

The first to eighth pixel arrays PX1, PX2, PX3, PX4, PX5, PX6, PX7, and PX8 may be arranged side by side along the second direction Y. For example, the first pixel array PX1 and the second pixel array PX2 may be disposed along the second direction Y. For example, the second pixel array PX2 and the third pixel array PX3 may be disposed along the second direction Y. In other words, the first pixel array PX1, the second pixel array PX2, and the third pixel array PX3 may be arranged side by side along the second direction Y.

The micro lens array 270 may be disposed between the lens assembly 210 and the image sensor 230. The lens assembly 210, the micro lens array 270, and the image sensor 230 may be aligned along the third direction Z. The first direction (X), the second direction (Y), and the third direction (Z) may be directions that cross each other. The micro lens array 270 may include a first micro lens ML1, a second micro lens ML2, a third micro lens ML3, and a fourth micro lens ML4. The microlens array 270 is disposed on the image sensor 230 and includes at least one of the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) of the image sensor 230. It can cover two pixel arrays. For example, the lens assembly 210 includes a main lens, and the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) are light incident through one main lens. Can be sensed.

In one embodiment, the micro lens array 270 may be disposed directly on the image sensor 230. For example, other components may not be interposed between the micro lens array 270 and the image sensor 230.

In an embodiment, the micro lens array 270 may be disposed on the image sensor 230 and spaced apart from the image sensor 230. For example, other components may be interposed between the micro lens array 270 and the image sensor 230.

For example, the first micro lens ML1 may be disposed between the lens assembly 210 and the first pixel array PX1, and between the lens assembly 210 and the second pixel array PX2. The first micro lens ML1 may cover the first pixel array PX1 and the second pixel array PX2. The first micro lens ML1 may overlap the first pixel array PX1 and the second pixel array PX2 in a third direction Z.

For example, the second micro lens ML2 may be disposed between the lens assembly 210 and the third pixel array PX3, and between the lens assembly 210 and the fourth pixel array PX4. The second micro lens ML2 may cover the third pixel array PX3 and the fourth pixel array PX4. The second microlens ML2 may overlap the third pixel array PX3 and the fourth pixel array PX4 in the third direction Z. The description of the first micro lens ML1 and the second micro lens ML2 may be applied to the third micro lens ML3 and the fourth micro lens ML4.

Each of the first to fourth microlenses ML1, ML2, ML3, and ML4 may extend along the first direction X. Each of the first to fourth micro lenses ML1, ML2, ML3 and ML4 may be a cylindrical lens. Each of the first to fourth microlenses ML1, ML2, ML3, and ML4 may be disposed along the second direction Y. Since each of the first to fourth microlenses ML1, ML2, ML3, and ML4 is a cylindrical lens, the first to fourth microlenses ML1 and ML1 are compared to a lens other than a cylindrical lens (for example, the lens assembly 210). ML2, ML3, ML4) may have a relatively large curvature. Accordingly, the focal length may be shortened.

The first pixel array PX1 senses the incident light incident on the first microlens ML1 to provide a first image signal, and the second pixel array PX2 receives incident light incident on the first microlens ML1. A second image signal may be provided by sensing.

In FIG. 3A, it is illustrated that the image sensor 230 includes an array of eight pixels, but the present invention is not limited thereto. For example, the image sensor 230 may include at least two or more pixel arrays, if necessary. In FIG. 3A, it is illustrated that each of the first to eighth pixel arrays PX1, PX2, PX3, PX4, PX5, PX6, PX7, and PX8 includes five sub-pixels, but is not limited thereto. For example, each of the first to eighth pixel arrays (PX1, PX2, PX3, PX4, PX5, PX6, PX7, PX8) may include an arbitrary number of subpixels as necessary. In FIG. 3A, the micro lens array 270 is illustrated as including four micro lenses, but is not limited thereto. For example, if the micro lens array 270 covers two pixel arrays, an arbitrary number of micro lenses may be included according to the number of pixel arrays included in the image sensor 230.

4 is a diagram illustrating the lens assembly 210 and the image sensor 230 of FIG. 2. For clarity of description, what has been described above may be simplified or omitted.

2 and 4, the image sensor 230 of the electronic device 201 according to the embodiment disclosed in this document may include a micro lens array 270 and a pixel array PX. The micro lens array 270 may include a plurality of micro lenses. Each of the plurality of micro lenses may include a pixel array corresponding to each of the plurality of micro lenses. For example, the first micro lens ML1 may include a pixel array PX corresponding to the first micro lens ML1.

The pixel array PX may be aligned along the first direction X. For example, the pixel array PX may include a first sub-pixel PXS1 and a second sub-pixel PXS2. The first sub-pixel PXS1 and the second sub-pixel PXS2 may be aligned along the first direction X. The pixel array PX may be disposed along the second direction Y.

Two readout circuits may be connected to the pixel array PX. One of the two readout circuits may be connected to the first region PXR1 of the pixel array PX. The other one of the two readout circuits may be connected to the second region PXR2 of the pixel array PX. For example, the first area PXR1 may be an area on one side of the pixel array PX, and the second area PXR2 may be an area on the other side of the pixel array PX. The two readout circuits sense different regions of the pixel array PX (e.g., the first region PXR1 and the second region PXR2), so that an image signal (e.g., photoelectric conversion signal) in one pixel array Can be obtained differentially.

For example, unlike those described with reference to FIGS. 3A and 3B, the micro lens array 270 may cover one pixel array. According to the exemplary embodiment disclosed in this document, a readout circuit is connected to each of the first region PXR1 and the second region PXR2 of one pixel array PX, so that image signals may be differentially obtained. . For example, the first area PXR1 of the pixel array PX senses the incident light incident on the first microlens ML1 to provide a first image signal, and the second area PXR2 of the pixel array PX ) May provide a second image signal by sensing incident light incident on the first microlens ML1.

5A is a diagram illustrating the processor 260 of FIG. 2. The operation described below with reference to the boundary detection circuit 261 and the distance measurement circuit 263 may be executed (or performed) by the processor 260.

2 and 5A, the processor 260 of the electronic device 201 according to the embodiment disclosed in the present document includes an edge detection circuitry 261 and a distance measurement circuitry. (263) may be included.

The boundary detection circuit 261 may be referred to as a plenoptic circuitry. The boundary detection circuit 261 may identify the boundary of the subject and provide boundary information. The boundary detection circuit 261 may identify the boundary of the subject based on the intensity of the first incident light among incident light incident on the lens assembly 210. For example, the boundary detection circuit 261 may generate a disparity map based on the intensity of the first incident light and identify the boundary of the subject using the disparity map.

In an embodiment, the boundary detection circuit 261 may further refer to distance information of the distance measurement circuit 263 to generate a disparity map based on the intensity of the first incident light and identify the boundary of the subject. . In an embodiment, the boundary detection circuit 261 may further refer to face recognition information of the face recognition circuit, generate a disparity map based on the intensity of the first incident light, and identify the boundary of the subject. In an embodiment, the boundary detection circuit 261 may generate a disparity map based on the intensity of the first incident light by further referring to the autofocus information of the autofocus detection circuit and identify the boundary of the subject.

The incident light may be light incident on the lens assembly 210 by reflecting light emitted from the light emitting device 220 from a subject.

The first incident light may be, for example, light having a certain incident angle or within a certain range of incident light. The first incident light may be light incident on the lens assembly 210 by reflecting light from the boundary of the subject. The boundary of the subject may be, for example, a boundary between one subject and another subject in an image including a plurality of subjects. The boundary of the subject may be, for example, a boundary between a foreground subject and a background subject in an image including a foreground subject and a background subject.

The first incident light may be incident on a first micro lens, which is one of the micro lens arrays 270. The boundary detection circuit 261 includes a first image signal generated by sensing a first incident light by a first pixel array or a first region of a pixel array, and a first image signal generated by sensing a first pixel array or a first region of the pixel array. The intensity of the first incident light may be detected using the second image signal generated by sensing the incident light.

The distance measurement circuit 263 may be referred to as a time of flight (TOF) circuit. The distance measuring circuit 263 may provide distance information for identifying a distance to a point and/or a boundary of the subject.

The distance measuring circuit 263 may compare the light emitted from the light emitting element 220 and the incident light to identify a distance to a point of the subject. For example, the distance measuring circuit 263 may compare the light emitted from the light emitting element 220 with the second incident light to identify a distance to a point of the subject. The second incident light may be, for example, light having a certain incident angle or within a certain range of incident light. The second incident light may be, for example, light emitted from the light emitting device 220 is reflected from a point of the subject and incident on the lens assembly 210.

The distance to the point and/or boundary of the subject may be a value measured from an arbitrary point and/or boundary of the electronic device 201 in the third direction Z, which is a direction from the electronic device toward the subject. According to an embodiment, the distance to the point and/or the boundary of the subject may be a distance between the lens assembly 210 of the electronic device 201 and the point and/or the boundary of the subject. According to an embodiment, the distance to the point and/or the boundary of the subject may be a distance between the image sensor 230 of the electronic device 201 and the point and/or the boundary of the subject.

According to an embodiment, the point of the subject may be a point inside the subject. According to an embodiment, the point of the subject may be, for example, a point at the boundary of the subject.

The distance measuring circuit 263 may determine a distance to a point and/or boundary of the subject by measuring a time when light is reflected from the subject and incident on the lens assembly 210. For example, the distance measuring circuit 263 may identify a distance to a boundary of the subject based on a phase difference between the light and the first incident light. For example, the distance measuring circuit 263 may identify a distance to a point of the subject based on a phase difference between the light and the second incident light.

The boundary detection circuit 261 may generate a disparity map based on the intensity of image signals sensed by a pixel array covered by one microlens. When generating the disparity map, the boundary detection circuit 261 may use information on an initial pixel group.

For example, in performing a stereo matching method for generating a disparity map, the boundary detection circuit 261 may identify an initial pixel group based on distance information for an object. For example, the electronic device 201 includes a group of pixels from 1 to n (however, n>1, n is a natural number), and each pixel group may include a pixel array covered by one microlens. have. The boundary detection circuit 261 may identify that the x pixel group (where 1≦x<n, x is a natural number) is an initial pixel group corresponding to the distance to the subject based on distance information on the subject. When generating the disparity map, the boundary detection circuit 261 compares each of the n pixel groups from the x+1 pixel group and the initial pixel group (ie, x pixel group) to identify the disparity value of each pixel group. I can. At this time, the boundary detection circuit 261 compares each of the n pixel groups from the 2 pixel group and the 1 pixel group to detect the disparity value of each pixel group, and detects the disparity value from the initial pixel group, resulting in high speed. You can create a disparity map with. The identification of the disparity value of the pixel group may be based on the intensity of the first and second image signals sensed by the pixel group.

The boundary detection circuit 261 may identify a boundary of a subject by identifying a sudden change in the intensity difference (ie, disparity value) of an image signal between neighboring pixel groups. For example, an x pixel group (eg, a pixel group including a first pixel array PX1 and a second pixel array PX2, or a pixel array PX covered by the first micro lens ML1) and Display between the x+1 pixel group (e.g., the pixel group including the third pixel array PX3 and the fourth pixel array PX4, or the pixel array PX covered by the second microlens ML2). The difference between the parity value is smaller than the reference value, and the x+1 pixel group and the x+2 pixel group (e.g., a pixel group including the fifth pixel array PX5 and the sixth pixel array PX6, or the third microlens If the difference in disparity values between the pixel arrays PX covered by (ML3) is greater than the reference value, it may be identified that the x+1 pixel group corresponds to the boundary of the subject.

The electronic device 201 may generate a depth map based on boundary information, disparity map, and distance information.

In an embodiment, in performing a stereo matching method for generating a disparity map, the boundary detection circuit 261 may identify an initial pixel group based on distance information for a subject. Distance information on the subject may be distance information provided from the distance measuring circuit 263, for example.

FIG. 5B is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information on a subject according to an exemplary embodiment disclosed in the present document. For clarity of description, those overlapping with those described above may be simplified or omitted. The operation described below with reference to the boundary detection circuit 261 may be executed (or performed) by the processor 260.

Referring to FIG. 5B, a processor 260 (eg, the electronic device 201 and/or the boundary detection circuit 261 of FIG. 2) according to an embodiment disclosed in this document is configured to generate a disparity map. In performing a stereo matching method, an initial pixel group may be identified based on distance information for a subject. Distance information on the subject may be, for example, face recognition information provided from the face recognition circuit 264.

In an embodiment, the face recognition circuit 264 may be implemented as, for example, a circuit included in the processor 260. In an embodiment, the face recognition circuit 264 may be implemented in software. In an embodiment, the face recognition circuit 264 may be a component included in an external electronic device. The face recognition circuit 264 may provide, for example, face recognition information that is a result of recognizing a human face among subjects.

The boundary detection circuit 261 may obtain distance information for a face among subjects based on the face recognition information. The boundary detection circuit 261 may identify an initial pixel group corresponding to a distance to a face based on the face recognition information.

FIG. 5C is a diagram illustrating that an electronic device identifies an initial pixel group based on distance information about a subject according to an embodiment disclosed in this document. For clarity of description, those overlapping with those described above may be simplified or omitted. The operation described below with reference to the boundary detection circuit 261 may be executed (or performed) by the processor 260.

Referring to FIG. 5C, a processor 260 (eg, the electronic device 201 and/or the boundary detection circuit 261 of FIG. 2) according to an embodiment disclosed in the present document is configured to generate a disparity map. In performing a stereo matching method, an initial pixel group may be identified based on distance information for a subject. Distance information about the subject may be, for example, auto focus information provided from the auto focus detection circuit 265.

In an embodiment, the auto focus detection circuit 265 may be implemented as, for example, a circuit included in the processor 260. In one embodiment, the auto focus detection circuit 265 may be implemented in software. In an embodiment, the auto focus detection circuit 265 may be a component included in an external electronic device. The auto focus detection circuit 265 may provide, for example, auto focus information, which is focus information on one of the subjects.

The boundary detection circuit 261 may obtain distance information for any one of the subjects based on the auto focus information. The boundary detection circuit 261 may identify an initial pixel group corresponding to a distance to any one of the subjects based on the auto focus information.

6 is a flowchart illustrating a method of generating a depth map by the electronic device 201 according to an embodiment disclosed in this document. Hereinafter, it is assumed that the electronic device 201 of FIG. 2 performs the process of FIG. 6. An operation described as being performed by the electronic device may be implemented as instructions that can be executed (or executed) by the processor 260 of the electronic device 201. The instructions may be stored in, for example, a computer recording medium, the memory 130 of the electronic device 101 of FIG. 1, or the memory 250 of the electronic device 201 of FIG. 2.

5A and 6, in operation S101, the processor 260 may provide boundary information identifying a first boundary of a first subject by using at least two image signals. For example, in generating the disparity map to provide boundary information, the processor 260 is sensed and generated by the first pixel array PX1 or the first region PXR1 of the pixel array PX. The first image signal and a second image signal sensed and generated by the second pixel array PX2 or the second region PXR2 of the pixel array PX may be used. The first image signal and the second image signal may be signals generated from the first incident light among incident light reflected from the first subject and incident on the lens assembly (the lens assembly 210 of FIG. 2 ). The processor 260 may detect the intensity of the first incident light using the first image signal and the second image signal. The processor 260 may identify a first boundary of the first subject based on the intensity of the first incident light.

For example, the processor 260 may generate a disparity map based on the intensity of the first incident light, and may identify a first boundary of the first subject. The processor 260 may generate a disparity map based on the intensity of image signals sensed by the pixel array covered by the microlens by the first incident light incident on the microlens. When generating the disparity map, the processor 260 may identify an initial pixel group based on distance information for a subject. The processor 260 may generate a disparity map using an initial pixel group, identify a first boundary from the disparity map, and provide boundary information.

According to an embodiment, the boundary information may be provided by the boundary detection circuit 261.

In operation S103, the processor 260 may provide distance information identifying a distance to the first point of the first subject. According to an embodiment, the distance information may be provided by the distance measurement circuit 263.

In an embodiment, the operation S101 and the operation S103 may be simultaneously performed (or executed) by the processor 260. In an exemplary embodiment, after any one of the operations S101 and S103 is performed (or executed) by the processor 260, the remaining operations may be performed (or executed).

In operation S105, the processor 260 may generate a depth map based on boundary information and distance information.

7, 8, and 9 are diagrams for explaining operations S101, S103, and S105 of FIG. 6. The x-axis of FIG. 9 may represent time (unit: AU (arbitrary unit)), and the y-axis may represent light intensity (unit: AU).

Referring to FIG. 7, light EL may emit light from at least one light emitting device 220 toward a first subject OB1 and a second subject OB2. According to an embodiment, each of the first subject OB1 and the second subject OB2 may be one of a foreground subject and a background subject. According to an embodiment, each of the first subject OB1 and the second subject OB2 may be a foreground subject.

The incident light IL may be light incident on the lens assembly 210 by reflecting the light EL on the first and second subjects OB1 and OB2. The incident light IL may pass through the lens assembly 210 and may be collected through the micro lens array 270.

Referring to FIG. 8, the incident light IL may include a first incident light IL1 and a second incident light IL2, a third incident light and a fourth incident light. For clarity of illustration, illustration of the third incident light and the fourth incident light has been omitted.

The first incident light IL1 may be incident light by reflecting the light EL from the first boundary 401 of the first subject OB1. The first incident light IL1 may be collected through the first micro lens ML1. The first boundary 401 of the first subject OB1 may be, for example, a part of the boundary between the first subject OB1 and the second subject OB2.

The second incident light IL2 may be light incident by reflecting the light EL from the first point 402 of the first object OB1. The second incident light IL2 may be collected through the second micro lens ML2. The first point 402 of the first subject OB1 may be, for example, a point inside the first subject OB1, not a boundary between the first subject OB1 and the second subject OB2. .

The third incident light may be light incident by reflecting the light EL from the second boundary 403 of the first subject OB1. The third incident light may be collected through a third micro lens (eg, the third micro lens ML3 of FIG. 3A ). The second boundary 403 of the first subject OB1 may be, for example, another part of the boundary between the first subject OB1 and the second subject OB2. The first boundary 401 and the second boundary 403 may be located on the same X-Y plane, for example.

The electronic device 201 according to the embodiment disclosed in this document uses the microlens array 270 to divide the boundary between the first subject OB1 and the second subject OB2 into parts (for example, the first The boundary 401 and the second boundary 403, and incident light reflected from each portion of the boundary may be separately sensed. By separately sensing incident light reflected from each part of the boundary to generate boundary information, and by creating a depth map using the boundary information, the boundary of the subject of the depth map generated according to the embodiment disclosed in this document is clearly identified. Can be.

The fourth incident light may be light that is incident by reflecting the light EL from the second point 404 of the first subject OB1. The fourth incident light IL4 may be collected through a fourth micro lens (eg, the fourth micro lens ML4 of FIG. 3A ). The second point 404 of the first subject OB1 may be, for example, a point inside the first subject OB1, not a boundary between the first subject OB1 and the second subject OB2. . The first point 402 and the second point 404 may have different distances from the lens assembly 210 in the third direction Z.

The first incident light IL1, the second incident light IL2, the third incident light, and the fourth incident light may have different incident angles incident on each of the first to fourth microlenses. For example, the first incident light IL1 may be light incident on the first microlens ML1 at a first incident angle θ1 among incident light IL. For example, the second incident light IL2 may be light incident on the second microlens ML2 at a second incident angle θ2 among incident light IL. The first incident angle θ1 and the second incident angle θ2 may be different. The description of the first incident light IL1 and the second incident light IL2 may also be applied to the third incident light IL3 and the fourth incident light IL4.

According to an embodiment, each of the incident angles at which the first incident light IL1, the second incident light IL2, the third incident light IL3, and the fourth incident light IL4 are incident to each of the first to fourth microlenses is constant. It can have a range. For example, the first incident angle θ1 may be within a certain angle range. When the first incident angle θ1 has a certain angular range, the incident light may be referred to as the first incident light IL1 if it is light within the range of the first incident angle θ1.

The image sensor 230 may sense light collected through the micro lens array 270 among incident light IL incident through the lens assembly 210.

For example, the first pixel array PX1 of the image sensor 230 senses the first incident light IL1 collected through the first microlens ML1 among the incident light IL, and receives a first image signal. Can be generated. The second pixel array PX2 of the image sensor 230 may generate a second image signal by sensing the first incident light IL1 collected through the first microlens ML1 among the incident light IL. .

The first image signal and the second image signal may include the same, similar, or different information. For example, the information on the amount of charge included in the first image signal may be different from the information on the amount of charge included in the second image signal. In other words, the amount of charge sensed by the first pixel array PX1 and the amount of charge sensed by the second pixel array PX2 may be different. For example, the phase of the first image signal and the phase of the second image signal may be opposite to each other. For example, information on the intensity of the first incident light IL1 included in the first image signal and the information on the intensity of the first incident light IL1 included in the second image signal are substantially the same or It can be similar.

Since the first incident light IL1 is incident light reflected from the first boundary 401 of the first subject OB1, information on the first boundary 401 is included. For example, in the first incident light IL1, information on the intensity of the first incident light IL1, position information on the XYZ coordinates of the first boundary 401, and the first incident angle θ1 of the first incident light IL1 May contain information about.

The processor 260 may provide boundary information identifying the first boundary 401 of the first subject OB1 using the first image signal and the second image signal. For example, the processor 260 identifies an initial pixel group based on distance information on a subject, generates a disparity map based on the intensity of the first incident light IL1, and determines the first boundary 401 Can be identified.

According to an embodiment, the processor 260 may identify a distance to the first boundary 401 based on a phase difference between the light EL and the first incident light IL1 and include it in the distance information. Referring to FIG. 9, a first graph G1 may be a graph showing the intensity of light EL over time, and a second graph G2 is a graph showing the intensity of first incident light IL1 over time. Can be The processor 260 may identify a distance to the first boundary 401 based on the phase difference Δφ between the first graph G1 and the second graph G2. The distance to the first boundary 401 may be a distance in the third direction Z from an arbitrary point related to the electronic device 201 to the first boundary 401.

According to an embodiment, the processor 260 emits light EL from the at least one light emitting element 220 and is reflected from the first boundary 401 of the first subject OB1 to the lens assembly 210. By measuring the incident time to, the distance to the first boundary 401 may be identified and included in the distance information.

Referring back to FIG. 8, the third pixel array PX3 of the image sensor 230 senses the second incident light IL2 collected through the second microlens ML2 among the incident light IL, It can generate an image signal. The fourth pixel array PX4 of the image sensor 230 may generate a fourth image signal by sensing the second incident light IL2 collected through the second microlens ML2 among the incident light IL. .

The third image signal and the fourth image signal may include the same, similar, or different information. The description of the first and second image signals may be applied to the third and fourth image signals.

Since the second incident light IL2 is incident light reflected from the first point 402 of the first object OB1, information on the first point 402 is included. For example, in the second incident light IL2, information on the intensity of the second incident light IL2, position information on the XYZ coordinates of the first point 402, and the second incident angle θ2 of the second incident light IL2 May contain information about.

The processor 260 may identify the first point 402 of the first subject OB1 using the third image signal and the fourth image signal. For example, the processor 260, based on the intensity of the second incident light IL2, if the intensity of the second incident light IL2 is less than the reference value, the second incident light IL2 is the first subject OB1 It may be identified as the light reflected from the inner point (ie, the first point 402) of the first subject OB1, not the boundary.

According to an embodiment, the processor 260 may identify a distance to the first point 402 based on a phase difference between the light EL and the second incident light IL2 and include it in the distance information. Referring to FIG. 9, a first graph G1 may be a graph showing the intensity of light EL over time, and a second graph G2 is a graph showing the intensity of second incident light IL2 over time. Can be The processor 260 may identify the distance to the first point 402 based on the phase difference Δφ between the first graph G1 and the second graph G2. The distance to the first point 402 may be a distance in the third direction Z from an arbitrary point related to the electronic device 201 to the first point 402.

According to an embodiment, the processor 260 emits light EL from the at least one light emitting element 220 and is reflected from the first point 402 of the first subject OB1 to the lens assembly 210. By measuring the incident time to, the distance to the first point 402 may be identified and included in the distance information.

Referring to FIG. 8 again, the fifth pixel array of the image sensor 230 (for example, the fifth pixel array PX5 of FIG. 3A) is a third incident light collected through a third microlens among the incident light IL ( IL3) may be sensed to generate a fifth image signal. For example, the sixth pixel array of the image sensor 230 (for example, the sixth pixel array PX6 in FIG. 3A) is a third incident light IL3 collected through a third microlens among the incident light IL. By sensing, a sixth image signal may be generated.

The fifth image signal and the sixth image signal may include the same, similar, or different information. Descriptions of the first and second image signals may also be applied to the fifth and sixth image signals.

The processor 260 may identify the second boundary 403 of the first subject OB1 using the fifth image signal and the sixth image signal. A description of the processor 260 identifying the first boundary 401 using the first and second image signals is, for the processor 260 to identify the second boundary 403 using the fifth and sixth image signals. Can be applied for identifying.

The processor 260 may identify a distance to the second boundary 403. A description of the processor 260 identifying the distance to the first boundary 401 may be applied to the processor 260 identifying the distance to the second boundary 403.

The seventh pixel array of the image sensor 230 (for example, the seventh pixel array PX7 in FIG. 3A) senses the fourth incident light IL4 collected through the fourth microlens among the incident light IL, 7 Can generate image signals. The eighth pixel array of the image sensor 230 (for example, the eighth pixel array PX8 in FIG. 3A) senses the fourth incident light IL4 collected through the fourth microlens among the incident light IL, 8 Image signals can be generated.

The seventh image signal and the eighth image signal may include the same, similar, or different information. Descriptions of the first and second image signals may also be applied to the seventh and eighth image signals.

The processor 260 may identify the second point 404 of the first subject OB1 using the seventh image signal and the eighth image signal. A description of the processor 260 identifying the first point 402 using the first and second image signals is, for the processor 260 to use the seventh and eighth image signals, the second point 404 It can be applied for identifying.

The processor 260 may identify a distance to the second point 404. The description of the processor 260 identifying the distance to the first point 402 may be applied to the processor 260 identifying the distance to the second point 404.

The processor 260 can clearly display the first boundary 401 and the second boundary 403 in the depth map by using boundary information for each of the first boundary 401 and the second boundary 403. . For example, in the depth map, the first boundary 401 may be clearly distinguished from the surrounding area of the first boundary 401. For example, in the depth map, since the brightnesses of the first subject OB1 and the second subject OB2 are displayed differently, the first boundary 401 can be clearly distinguished.

In the depth map, the processor 260 determines a first point 402 that is relatively farther from the first boundary 401 and the second boundary 403 than the first boundary 401 and the second boundary 403. It can be displayed darkly. In the depth map, the processor 260 may display the second point 404, which is relatively farther away than the first point 402, to be darker than the first point 402 in the depth map.

The depth map may include information on a boundary between the first subject OB1 and the second subject OB2, location information on XYZ coordinates, and the like.

An electronic device (eg, the electronic device 201 of FIG. 2) according to an embodiment disclosed in this document includes an image sensor (eg, the image sensor 230 of FIG. 2) and a lens assembly (eg, the lens assembly of FIG. 2 ). By disposing a micro lens array (eg, the micro lens array 270 of FIG. 2) between (210)), incident light incident at various incident angles may be collected, respectively. In other words, incident light reflected from various points of the first subject (for example, the first boundary 401, the first point 402, the second boundary 403, and the second point 404 of FIG. 8) Silver may be classified and collected through a micro lens array. Accordingly, the processor of the electronic device (eg, the processor 260 of FIG. 2) may provide distance information and boundary information at various points of the first subject. Specifically, since the processor of the electronic device can generate a depth map based on the boundary information at the boundary of the subject as well as distance information at the boundary and the internal point of the subject, a sophisticated depth map capable of identifying the boundary of the subject Can be implemented. In addition, when generating a disparity map for identifying a boundary of a subject, the electronic device may generate a disparity map at a high speed by identifying an initial pixel group based on distance information about the subject. For example, in an electronic device that provides a live view mode to a user, according to an embodiment disclosed in this document, by generating a disparity map based on an initial pixel group, a boundary of a subject may be identified in real time. Furthermore, by using the micro lens array, the electronic device may implement a sophisticated depth map capable of identifying the boundary of the subject based on distance information and boundary information about at least one point located at the boundary of the subject.

10A, 10B, 10C, and 10D are diagrams for explaining a user interface provided by an electronic device (for example, the electronic device 201 and/or the processor 260) according to an embodiment disclosed in this document admit. For clarity of description, those overlapping with those described above may be simplified or omitted.

Referring to FIG. 10A, an electronic device according to an embodiment disclosed in this document may provide a live view mode and/or a video capture mode. The first user interface 1001 may be a screen on which a third object OB3 and a fourth object OB4 are displayed through an electronic device in a live view mode and/or a video capture mode.

Referring to FIG. 10B, an electronic device according to an embodiment disclosed in this document may receive a user input related to selecting an object. In an embodiment, the user may select the third object OB3 in the live view mode. In an embodiment, the user may select the third object OB3 while capturing a video in the video capturing mode.

Referring to FIG. 10C, the electronic device according to the embodiment disclosed in this document may change properties of the third object OB3 and/or the fourth object OB4 based on a user input. The second user interface 1002 may be a screen in which properties of the third object OB3 and/or the fourth object OB4 are changed based on a user input.

For example, the electronic device identifies the boundary between the third object OB3 and the fourth object OB4, and identifies distance information for each point of the third object OB3 and the fourth object OB4. can do. When the user selects the third object OB3, the electronic device determines the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4 based on the boundary information. And/or the attribute of the fourth object OB4 may be changed. On the basis of the boundary EG identified using boundary information, the electronic device provides the fourth object OB4 in a direction away from the third object OB3 with respect to a point inside the fourth object OB4 based on the distance information. You can increase or decrease the intensity of changing the properties of

For example, the electronic device may apply a blur effect to the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4. For example, the electronic device may change the color of the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4. For example, the electronic device may apply a mosaic effect to the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.

In an embodiment, while the second user interface 1002 is displayed in the live view mode of the electronic device, when the electronic device receives an input related to photographing from the user, the electronic device may be configured with a third object OB3 and a fourth object. The photographing may be performed in a state in which properties of the third object OB3 and/or the fourth object OB4 are changed based on the boundary EG between the objects OB4. In this case, the same image data as the second user interface 1002 may be stored in the electronic device.

In an embodiment, while the second user interface 1002 is displayed in the video recording mode of the electronic device, the electronic device is based on the boundary EG between the third object OB3 and the fourth object OB4, The photographing may be performed while the attributes of the third object OB3 and/or the fourth object OB4 are changed. In this case, the same video data as the second user interface 1002 may be stored in the electronic device.

Referring to FIG. 10D, the electronic device may receive a user input related to selection of the fourth object OB4 in the second user interface 1002. The electronic device may change attributes of the third object OB3 and the fourth object OB4 based on the boundary EG between the third object OB3 and the fourth object OB4.

For example, the electronic device applies a blur effect to the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4, and applies a blur effect to the fourth object OB4. Can be sharpened. For example, the electronic device may change the color of the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4. For example, the electronic device applies a mosaic effect to the third object OB3 based on the boundary EG between the third object OB3 and the fourth object OB4, and the fourth object OB4 Can be sharpened.

Various embodiments of the present document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the corresponding embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or a plurality of the items unless clearly indicated otherwise in a related context. In this document, “A or B”, “at least one of A and B”, “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “A Each of phrases such as "at least one of, B, or C" may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the component from other corresponding components, and the components may be referred to in other aspects (eg, importance or Order) is not limited. Some (eg, a first) component is referred to as “coupled” or “connected” with or without the terms “functionally” or “communicatively” to another (eg, second) component. When mentioned, it means that any of the above components can be connected to the other components directly (eg by wire), wirelessly, or via a third component.

The term "module" used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (for example, the program 140) including them. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This makes it possible for the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic wave), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in the present document may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities. Computer program products are distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g., Play Store™) or two user devices ( It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are sequentially, parallel, repeatedly, or heuristically executed, or one or more of the above operations are executed in a different order or omitted. Or one or more other actions may be added.

Claims (15)

1. In the electronic device,

   Lens assembly;

   A first pixel array aligned along a first direction;

   A second pixel array aligned along the first direction;

   A first microlens disposed between the lens assembly and the first pixel array and between the lens assembly and the second pixel array and covering the first pixel array and the second pixel array;

   Processor; And

   A memory operatively connected to the processor,

   The memory, when executed, causes the processor to:

   A first image signal generated by sensing a first incident light collected through the first microlens among incident light incident on the lens assembly by reflecting light emitted from at least one light emitting element from a subject by the first pixel array And, using a second image signal generated by sensing the first incident light by the second pixel array, providing boundary information identifying a first boundary of the subject,

   Provides first distance information for identifying a first distance to a first point of the subject by comparing the second incident light among the light and the incident light,

   An electronic device that stores instructions for generating a depth map based on the first distance information and the boundary information.
2. The method according to claim 1,

   The lens assembly includes a main lens,

   The electronic device, wherein the first incident light and the second incident light are incident through the main lens.
3. The method according to claim 1,

   The electronic device, wherein the first micro lens is a cylindrical lens.
4. The method according to claim 1,

   The electronic device includes a group of pixels from 1 to n (where n is a natural number greater than 1) and a plurality of micro lenses covering each of the pixel groups from 1 to n,

   Each of the pixel groups 1 to n includes at least two pixel arrays covered by each of the plurality of micro lenses,

   The first pixel array and the second pixel array constitute an x pixel group (where 1≦x<n, x is a natural number),

   The instructions, the processor,

   Comparing the light and the first incident light, providing second distance information for identifying a second distance to the first boundary,

   Based on the second distance information, identifying the x pixel group corresponding to the second distance as an initial pixel group,

   The electronic device, wherein the first boundary is identified by comparing each of the n pixel groups from the x+1 pixel group with the initial pixel group.
5. The method according to claim 1,

   The electronic device, wherein the first image signal and the second image signal include information on an intensity of the first incident light.
6. The method according to claim 1,

   The first micro lens, the electronic device extending along the first direction.
7. The method according to claim 1,

   The electronic device,

   A third pixel array aligned along the first direction;

   A fourth pixel array aligned along the first direction; And

   A second microlens disposed between the lens assembly and the third pixel array and between the lens assembly and the fourth pixel array and covering the third pixel array and the fourth pixel array,

   The first pixel array, the second pixel array, the third pixel array, and the fourth pixel array are disposed along a second direction different from the first direction.
8. The method of claim 7,

   Each of the first and second micro lenses extends along the first direction,

   The electronic device, wherein the first microlens and the second microlens are disposed along the second direction.
9. The method of claim 7,

   Each of the first micro lens and the second micro lens is a cylindrical lens.
10. The method of claim 7,

    The instructions, the processor,

    Among the incident light, a third image signal generated by sensing a third incident light collected through the second microlens by the third pixel array, and a third image signal generated by sensing the third incident light by the fourth pixel array. 4 Using the image signal, identify the second boundary of the subject and include it in the boundary information,

    The electronic device comprising: comparing a fourth incident light among the light and the incident light to identify a second distance to a second point of the subject to provide second distance information.
11. The method of claim 10,

    The instructions, the processor,

    The electronic device of claim 1, wherein a phase difference between the light and the fourth incident light is identified to compare the light and the fourth incident light.
12. The method according to claim 1,

    The instructions, the processor,

    An electronic device configured to compare the light and the second incident light based on a phase difference between the light and the second incident light.
13. In a method for generating a depth map of an electronic device,

    Based on the intensity of the first incident light among the light emitted from at least one light emitting device and the incident light reflected from the subject and incident to the lens assembly using the edge detection circuitry of the electronic device Thus, identifying the first boundary of the subject and providing boundary information;

    Providing first distance information for identifying a first distance to a first point of the subject by comparing the light and a second incident light of the incident light using a distance measurement circuitry of the electronic device action; And

    And generating the depth map based on the first distance information and the boundary information.
14. The method of claim 13,

    A method of generating a depth map, identifying a second distance to the first boundary and providing second distance information based on a phase difference between the light and the first incident light.
15. The method of claim 13,

    Comparing the light and the second incident light is to identify a phase difference between the light and the second incident light.

Images