KR102551261B1 - Method for generating depth information by using structured light pattern projected to external object and Electronic device using the same - Google Patents

Abstract

An electronic device comprising: a camera; a light emitting module including one or more light emitting bodies; Memory; and a processor, wherein the processor projects structured light corresponding to a designated structured light pattern to an external object using the light emitting module, and the designated structured light pattern is distributed according to a designated rule. An image including polygons and a plurality of second polygons connecting at least some of the plurality of first polygons based on a designated repeating grid pattern, and the structured light reflected by the external object using the camera Obtaining, based on the obtained image, the specified repeating grid pattern and at least some of the first polygons are identified, and based on the identified specified repeating grid pattern and at least some of the first polygons, the specified repeating grid pattern and at least some of the first polygons are not identified. restoring other portions of the first polygons, and generating depth information about the external object based on positions of the at least some of the first polygons and the restored first polygons of the other portion; can be set. Other embodiments are also possible.

Description

Method for generating depth information by using structured light pattern projected to external object and Electronic device using the same}

Various embodiments of the present disclosure relate to an image processing method and an electronic device using the same.

The use of functions such as augmented reality (AR) or virtual reality (VR) in electronic devices is gradually increasing. It may be necessary to perform 3D (three dimensional) modeling of an external object in order to implement such augmented reality or virtual reality functions more realistically.

In order for the electronic device to perform 3D modeling of the external object, depth information between the electronic device and the external object may be required. The present invention may provide a specific method of acquiring depth information to perform 3D modeling.

An electronic device according to various embodiments includes a camera; a light emitting module including one or more light emitting bodies; Memory; and a processor, wherein the processor projects structured light corresponding to a designated structured light pattern to an external object using the light emitting module, and the designated structured light pattern is distributed according to a designated rule. An image including polygons and a plurality of second polygons connecting at least some of the plurality of first polygons based on a designated repeating grid pattern, and the structured light reflected by the external object using the camera Obtaining, based on the obtained image, the specified repeating grid pattern and at least some of the first polygons are identified, and based on the identified specified repeating grid pattern and at least some of the first polygons, the specified repeating grid pattern and at least some of the first polygons are not identified. restoring other portions of the first polygons, and generating depth information about the external object based on positions of the at least some of the first polygons and the restored first polygons of the other portion; can be set.

A method of an electronic device according to various embodiments of the present disclosure includes: projecting structured light corresponding to a designated structured light pattern to an external object by using a light emitting module; acquiring an image in which the structured light is reflected by the external object using the camera; checking a specified repeating grid pattern and at least some first polygons based on the obtained image; restoring other portions of the first polygons that are not identified based on the confirmed designated repeating grid pattern and at least some of the first polygons; and generating depth information of the external object based on positions of the at least some of the first polygons and the restored first polygons of the other part, wherein the designated structured light pattern comprises: It may include a plurality of first polygons distributed according to a specified rule and a plurality of second polygons connecting at least some of the plurality of first polygons based on the specified repeating grid pattern.

An electronic device according to various embodiments of the present disclosure may obtain depth information of an external object and perform 3D modeling of the external object based on the acquired depth information.

1A is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.
1B is a block diagram of a camera module according to various embodiments of the present disclosure.
2 is a diagram schematically illustrating a 3D modeling system of an electronic device according to various embodiments of the present disclosure.
3 is a block diagram schematically illustrating a depth sensing system of an electronic device according to various embodiments of the present disclosure.
4 is a flowchart illustrating a method of detecting a depth of an external object in an electronic device according to various embodiments of the present disclosure.
5 is a diagram illustrating a logical reference pattern of an electronic device according to various embodiments of the present disclosure.
6 is a diagram illustrating a first structured light pattern of an electronic device according to various embodiments of the present disclosure.
7 is a diagram illustrating a composite structured light pattern in which a first structured light pattern and a second structured light pattern of an electronic device according to various embodiments of the present disclosure are combined.
8A to 8B are diagrams for explaining a method for generating a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.
9A to 9C are diagrams for explaining a method of utilizing cellular automata to generate a logical reference pattern of an electronic device according to various embodiments of the present disclosure.
10A is a diagram illustrating an image reflected by an external object obtained by projecting a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.
10B is a diagram illustrating a physical shape corresponding to at least a part of an external object obtained from an image reflected by an external object of an electronic device according to various embodiments of the present disclosure.
10C is a diagram illustrating a logical shape generated based on an acquired physical shape of an electronic device according to various embodiments of the present disclosure.
11A to 11E are diagrams illustrating a method of restoring at least a part of a pre-generated logical reference pattern from an image reflected by an external object using cellular automata of an electronic device according to various embodiments of the present disclosure; am.
12 is a diagram illustrating a method of generating a lookup map from a pre-generated logical reference pattern of an electronic device according to various embodiments of the present disclosure.
13 is a diagram illustrating a method of obtaining connected components through at least some restored logical reference patterns of an electronic device and obtaining at least one coding word included in the connected components according to various embodiments of the present disclosure.
14 is a diagram illustrating a method of generating depth information of an external object through a lookup map of an electronic device according to various embodiments of the present disclosure.
15 is a flowchart illustrating a method of obtaining a connected component from an image reflected by an external object of an electronic device according to various embodiments of the present disclosure.
16 is a flowchart illustrating a method of generating depth information corresponding to a connected component based on a coded word included in the connected component of an electronic device according to various embodiments of the present disclosure.
17 is a diagram illustrating performing 3D modeling using connected components of an electronic device according to various embodiments of the present disclosure.
18A to 18B are diagrams illustrating performing 3D modeling and self-calibration using a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.

1A is a block diagram of an electronic device 101 within a networked environment 100, in accordance with various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted or one or more other components may be added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , process commands or data stored in the volatile memory 132 , and store resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and a secondary processor 123 (eg, a graphics processing unit, an image signal processor) that may operate independently of or in conjunction therewith. , sensor hub processor, or communication processor). Additionally or alternatively, the secondary processor 123 may be configured to use less power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used for a component (eg, the processor 120) of the electronic device 101 from an outside of the electronic device 101 (eg, a user). The input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, a stylus pen).

The sound output device 155 may output sound signals to the outside of the electronic device 101 . The audio output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry set to detect a touch or a sensor circuit (eg, a pressure sensor) set to measure the intensity of force generated by the touch. there is.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 388 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, the corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (eg, a cellular network, the Internet, or It may communicate with an external electronic device via a computer network (eg, a telecommunications network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified and authenticated.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module may include one antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the electronic devices 102 and 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external devices among the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

1B is a block diagram 100 of a camera module 180, in accordance with various embodiments. Referring to FIG. 1B , the camera module 180 includes a lens assembly 181, a flash 182, an image sensor 183, an image stabilizer 184, a memory 185 (eg, a buffer memory), or an image signal processor. (186). The lens assembly 181 may collect light emitted from a subject that is an image capturing target. The lens assembly 181 may include one or more lenses. According to one embodiment, the camera module 180 may include a plurality of lens assemblies 181 . In this case, the camera module 180 may be, for example, a dual camera, a 360-degree camera, or a spherical camera. The plurality of lens assemblies 181 have the same lens properties (eg, angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly is at least as large as another lens assembly. It can have one other lens property. The lens assembly 181 may include, for example, a wide-angle lens or a telephoto lens. The flash 182 may emit a light source used to enhance light emitted from a subject. The flash 182 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.

The image sensor 183 may obtain an image corresponding to the subject by converting light transmitted from the subject through the lens assembly 181 into an electrical signal. According to one embodiment, the image sensor 183 is, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties. Each image sensor included in the image sensor 183 may be implemented as, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 184 reacts to the movement of the camera module 180 or the electronic device 101 including the camera module 180 to at least partially compensate for a negative effect (eg, image shaking) caused by the movement on a captured image. At least one lens or image sensor 183 included in the assembly 181 may be moved in a specific direction or controlled (eg, read-out timing is adjusted, etc.). According to one embodiment, the image stabilizer 184 may be implemented as, for example, an optical image stabilizer, and a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 180. ) to detect the movement.

The memory 185 may at least temporarily store at least a part of an image acquired through the image sensor 183 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or when a plurality of images are acquired at high speed, the acquired original image (eg, high resolution image) is stored in the memory 185, and a copy corresponding to it is stored in the memory 185. An image (eg, a low resolution image) may be previewed through the display device 186 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 185 may be obtained and processed by, for example, the image signal processor 186 . According to one embodiment, the memory 185 may be configured as at least a part of the memory 183 or as a separate memory operated independently of the memory 183 .

The image signal processor 186 performs image processing (eg, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, an image signal processor ( 186) may perform control (eg, exposure time control, lead-out timing control, etc.) for at least one of the components included in the camera module 180 (eg, the image sensor 183). Images processed by the signal processor 186 are stored back in the memory 185 for further processing or external components of the camera module 180 (e.g. memory 183, display device 186, electronic device 102 ), the electronic device 104, or the server 108) According to one embodiment, the image signal processor 186 is configured as at least a part of the processor 182, or independently of the processor 182 In the case of a separate processor, the images processed by the image signal processor 186 are displayed as they are or after additional image processing by the processor 182. can be displayed through

According to an embodiment, the electronic device 101 may include two or more camera modules 180 each having different properties or functions. In this case, for example, at least one camera module 180 may be a wide-angle camera or a front camera, and at least one other camera module may be a telephoto camera or a rear camera.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B," "A, B or C," "at least one of A, B and C," and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in this document may include a unit implemented by hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Device-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include a singular entity or a plurality of entities. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

2 is a diagram schematically illustrating a 3D modeling system of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the 3D modeling system of the electronic device 101 may include a projector 210, structured light pattern generators 215 and 216, and a camera 220. there is.

According to various embodiments, the projector 210 may project electromagnetic waves or light to external objects 231 and 232 as a source.

According to various embodiments, the structured light pattern generators 215 and 216 may include a first structured light pattern generator 215 and a second structured light pattern generator 216 . For example, the structured light pattern generating units 215 and 216 may generate the structured light pattern using an electromagnetic field filter or mask.

According to various embodiments, the first structured light pattern 215 may include a logical reference pattern generated using cellular automata (CA). Cellular automata is a method of interpreting dynamical systems. By dealing with space and time discretely, the state of each cell is automatically updated by local interaction, and specific patterns can be derived. The first structured light pattern 215 may include the logical reference pattern itself or a pattern obtained by partially modifying the logical reference pattern. The first structured light pattern 215 may be combined with the second structured light pattern 216 to generate a complex structured light pattern.

According to various embodiments, the second structured light pattern 216 may include a grid pattern that is repeatedly arranged. The second structured light pattern 216 may include at least two types of polygons, and the at least two types of polygons may have different shapes or colors. The second structured light pattern 216 may be combined with the first structured light pattern 215 to generate a composite structured light pattern.

According to various embodiments, the light projected by the projector 210 of the electronic device 101 may include a complex structured light pattern generated by the first structured light pattern 215 and the second structured light pattern 216. there is. The camera 220 of the electronic device 101 may obtain or capture an image 225 in which the projected complex structured light pattern is reflected by the objects 231 and 232 (acquired or captured). .

According to various embodiments, light projected from the projector 210 of the electronic device 101 may be reflected by the first object 231 or the second object 232, from the electronic device 101 to the first object. Since the distance to 231 or the second object 232 is different, the light paths 211 and 212 may be different. Due to the difference between the light paths 211 and 212, a parallax shift 240 may occur, and the electronic device 101 is based on the parallax shift 240, and the electronic device 101 and the object 231, 232) and depth information.

3 is a block diagram schematically illustrating a depth sensing system of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the electronic device 101 may generate depth information of an external object using the depth sensing system 300 . For example, the depth sensing system 300 may include a structured light pattern projection module 310 and a depth information generating module 320 .

According to various embodiments of the present disclosure, the structured light pattern projection module 310 may project a pre-designated or pre-generated structured light pattern onto an external object. For example, since generating a new structured light pattern whenever 3D modeling is performed in the electronic device 101 may continuously apply a load to a processor (eg, the processor 120 of FIG. 1 ), the structured light pattern to be projected is It may be created in advance by a manufacturer and stored in the memory of the electronic device 101 (eg, the memory 130 of FIG. 1 ).

According to various embodiments, the structured light pattern projection module 310 may project light based on a logical reference pattern generated by cellular automata. Alternatively, the structured light pattern projecting module 310 may project light obtained by transforming at least a part of a logical reference pattern generated by cellular automata.

According to various embodiments, the structured light pattern projection module 310 may generate a logical reference pattern generated by a cellular automata, a structured light pattern generated based on a logical reference pattern, or a structured light pattern generated based on a logical reference pattern. At least some of the structured light patterns among the structured light patterns obtained by latticing the structured light patterns may be projected. Since the above-mentioned structured light patterns are basically based on logical reference patterns, they may include structures that follow the rules of cellular automata.

According to various embodiments, the depth information generation module 320 may acquire an image reflected by light projected by the structured light pattern projection module 310 of an external object.

According to various embodiments, since parallax movement of the reflected image occurs according to the depth of the external object, the depth information generating module 320 checks the disparity of the parallax movement to generate depth information of the external object. can

According to various embodiments, when the external object forms a uniform plane, depth information may be generated simply by considering parallax movement of the structured light pattern, but when at least a part of the external object is not uniform, the structured light pattern Distortion may occur, and if the distortion is severe, it may lead to data loss.

According to various embodiments of the present disclosure, the depth information generating module 320 may restore a pre-generated structured light pattern or a pre-generated logical reference pattern from an acquired image even when at least a part of the external object is not uniform. The depth information generating module 320 may restore at least a part of the pre-generated logical reference pattern from at least a part of the acquired image by using a cellular automata rule used for the pre-generated logical reference pattern.

According to various embodiments, the depth information generating module 320 may obtain or check a connected component indicating the depth of the same level in at least some of the recovered logical reference patterns.

According to various embodiments, the depth information generation module 320 may obtain or verify at least one coding word included in the obtained connected component.

According to various embodiments, the depth information generation module 320 may determine which part of the pre-generated logical reference patterns corresponds to a corresponding connected component from coding words included in the connected component. For example, coded words corresponding to each cell may be stored as a lookup map in a pre-generated logical reference pattern. Accordingly, by comparing the coded words included in the connected component with a pre-stored lookup map, it is possible to determine which position of the pre-generated logical reference pattern the connected component corresponds to.

According to various embodiments of the present disclosure, the depth information generation module 320 may check how much parallax movement has occurred by comparing the connected component with a pre-generated logical reference, thus providing depth information on at least a part of the acquired image corresponding to the connected component. can create

In the above, functions performed by the depth sensing system 300 have been generally described, and specific embodiments will be described with reference to FIGS. 4 to 18 below.

4 is a flowchart illustrating a method of detecting a depth of an external object in an electronic device according to various embodiments of the present disclosure.

According to various embodiments, in operation 410, the electronic device 101 may project a structured light pattern onto an external object. The electronic device 101 may include a projector, and through the projector, a structured light pattern may be projected onto an external object using electromagnetic waves or light as a source. The structured light pattern projected by the electronic device 101 may include a logical reference pattern generated using cellular automata (CA).

According to various embodiments of the present disclosure, in operation 420, the electronic device 101 may obtain an image in which the structured light pattern is reflected by an external object. The electronic device 101 may include a camera, and obtain an image in which a structured light pattern projected from a projector is reflected by an external object.

According to various embodiments, in operation 430, the electronic device 101 may detect the depth of an external object based on the acquired image. When the external object has a curve, distances between some areas constituting the external object from the electronic device 101 may be different. Depending on the distance, the optical path may also be different, and parallax movement may be confirmed in the acquired image due to the difference in the optical path. The electronic device 101 may generate depth information of an external object based on the confirmed parallax movement.

5 is a diagram illustrating a logical reference pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the electronic device 101 may project a structured light pattern based on the logical reference pattern 500 onto an external object. The electronic device 101 may use an electromagnetic field filter or mask to project a structured light pattern based on the logical reference pattern 500 .

According to various embodiments, the electronic device 101 may generate the logical reference pattern 500 whenever 3D modeling is performed, but in order to efficiently use the limited resources of the processor, a pre-generated or stored logical reference pattern ( 500) can also be used.

According to various embodiments, the logical reference pattern 500 of the electronic device 101 may include a logical reference pattern 500 generated using cellular automata (CA). Cellular automata is a method of interpreting dynamical systems. By dealing with space and time discretely, the state of each cell is automatically updated by local interaction, and specific patterns can be derived. A detailed method of generating the logical reference pattern 500 disclosed in FIG. 5 will be described in detail with reference to FIGS. 9A to 9C.

6 is a diagram illustrating a first structured light pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the electronic device 101 may generate the first structured light pattern 600 based on the logical reference pattern 500 illustrated in FIG. 5 . For example, the first structured light pattern 600 of FIG. 6 emits the logical reference pattern 500 as it is, but may correspond to a drawing in which a partial area of the logical reference pattern 500 is enlarged to help understanding of FIG. 7 . can As another example, the first structured light pattern 600 may be generated by spacing the cells included in the logical reference pattern 500 at regular intervals. That is, the electronic device 101 may emit the pre-generated logical reference pattern 500 as it is or may emit the first structured light pattern 600 obtained by partially transforming the logical reference pattern 500 .

7 is a diagram illustrating a composite structured light pattern in which a first structured light pattern and a second structured light pattern of an electronic device according to various embodiments of the present disclosure are combined.

According to various embodiments, the electronic device 101 may project the composite structured light pattern 700 obtained by coating the first structured light pattern 600 with the second structured light pattern onto an external object. For example, the second structured light pattern may include a grid pattern. The lattice pattern may be regularly arranged and repeated over the entire area of the complex structured light pattern 700 . For example, the electronic device 100 may distribute first polygons to correspond to cells of the logical reference pattern 500 generated by cellular automata, but may use second polygons smaller than the spacing between cells. Accordingly, a separation space may naturally occur between the first polygons, and the second polygons may fill the separation space. For example, the first polygons may be divided into different polygons based on a difference in logical value (eg, 0 or 1), color (eg, black or white), and the like. The second polygons may also be classified in the same way as the first polygons.

According to various embodiments, the electronic device 100 may set a grid that can be a reference point even in a non-repetitive logical reference pattern by using the composite structured light pattern 700 . The electronic device 100 can discover rules even in irregularities, and through this, it can generate depth information from an image reflected on an external object. A specific embodiment of generating the composite structured light pattern 700 will be described in detail with reference to FIGS. 8A to 8C.

8A to 8C are views for explaining a method for generating a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 8A , a part of a pre-generated logical reference pattern may be disclosed. For example, a portion of the logical reference pattern may include nine different cells 801 to 809. Here, a value of 0 may be included in the cells 801, 803, and 806 displayed in black, and a value of 1 may be included in the cells 802, 804, 805, 807, 808, and 809 displayed in white.

According to various embodiments, referring to FIG. 8B , some of the logical reference patterns may be displayed apart from each other by a predetermined distance. Although it is shown as if there is no distance between cells in FIG. 8A, if the logical reference pattern is enlarged, the distance between cells may be spaced apart.

According to various embodiments, referring to FIG. 8C , grid patterns 811 to 819 may be filled in empty spaces between cells using polygons smaller than the distance between cells in FIG. 8B. This can be expressed as that the first structured light pattern is tessellated and fitted by the second structured light pattern. Since it is difficult to accurately determine which region the image reflected on the external object corresponds to and how much distortion has occurred with only the logical reference pattern, the second structured light pattern, which can be a reference point or frame, is mixed with the first structured light pattern. something may be needed Accordingly, the electronic device 101 may project a composite structured light pattern in which the first structured light pattern and the second structured light pattern are mixed to an external object.

9A to 9C are diagrams for explaining a method of utilizing cellular automata to generate a logical reference pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 9A , the logical reference pattern of the electronic device 101 may include a combination of a plurality of cells. Each cell can be filled with a value of 0 or 1. In the case of cellular automata, regularity may exist in that the state of another cell is determined by being influenced by the state (eg, value) of neighboring cells. For example, with respect to the cell 921 adjacent to the right of the cell 911, the cell ( 921) can be determined. See FIG. 9B for rules for generating cellular automata.

According to various embodiments, referring to FIG. 9B , a logical reference pattern of the electronic device 101 may be generated using Rule #25 920 of cellular automata. For example, s corresponding to the columns (s0, s1) of Rule #25 (920) may mean the state of the current cell (eg, 911, Cell(i)). In addition, k corresponding to rows (k0, k1, k2) of Rule #25 is a cell adjacent to the upper part of the current cell 911 (eg, 910, Cell(i-1)) and a cell neighboring to the lower part (eg, 910, Cell(i-1)). : 912, it may mean the sum of the states of Cell(i+1)).

According to various embodiments, referring to FIG. 9B , when s is 0 and k is 0, the state of the cell 921 right next to the current cell 911 may be determined to be 1. If s is 0 and k is 1, the state of the cell 921 right next to the current cell 911 may be determined to be 0. If s is 0 and k is 2, the state of the cell 921 right next to the current cell 911 may be determined to be 1. In addition, if s is 1 and k is 0, the state of the cell 921 right next to the current cell 911 may be determined to be 0. If s is 1 and k is 1, the state of the cell 921 right next to the current cell 911 may be determined to be 1. If s is 1 and k is 2, the state of the cell 921 right next to the current cell 911 may be determined to be 0.

According to various embodiments, referring to FIG. 9C , a logical reference pattern may be generated using Rule #25 (920). For example, Rule #25 (920) can be used to determine the state of the cell 941 right next to the current cell 931. Referring to FIG. 9C , the state of the current cell 931 is 1, and the sum of cells 930 and 932 adjacent to the current cell 931 is 0. Accordingly, since s is 1 and k is 0, the state of the cell 941 right next to the current cell 931 may become 0 when substituted into Rule #25.

According to various embodiments, Rule # 25 (920) of cellular automata disclosed in FIGS. 9A to 9C is only one embodiment, and it is possible to generate logical reference patterns using other rules. .

10A is a diagram illustrating an image reflected by an external object obtained by projecting a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 10A , the electronic device 101 may project a composite structured light pattern in which a first structured light pattern and a second structured light pattern are mixed to an external object. The electronic device 101 may obtain an image 1000 in which the complex structured light pattern is reflected by an external object. The complex structured light pattern is a two dimensional (2D) image, and the image 1000 acquired by the electronic device 101 may also be a 2D image. The electronic device 101 may perform 3D (three dimensional) modeling of an external object based on the obtained image.

10B is a diagram illustrating a physical shape corresponding to at least a part of an external object obtained from an image reflected by an external object of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 10B , an image reflected by an external object acquired by the electronic device 101 may be different from a uniform lattice pattern (eg, a second structured light pattern) of the logical reference pattern. can For example, if the external object does not have a shape with a uniform surface, distortion of the uniform grid pattern may occur due to curvature of the external object or data loss. Accordingly, the electronic device 101 may acquire an image having a distorted lattice pattern, which may be defined as a physical appearance (1010).

10C is a diagram illustrating a logical shape generated based on an acquired physical shape of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 10C , the electronic device 101 may move a physical shape 1010 obtained from a captured image to a computational space. For example, the electronic device 101 may generate a logical appearance 1020 by moving a lattice portion serving as a reference point in the physical shape 1010 to the calculation area.

According to various embodiments, the electronic device 101 may restore at least a part of a pre-generated logical reference pattern from an image obtained based on the logical shape 1020 . The electronic device 101 may verify the cells included in the logical shape 1020 by performing cellular automata again based on the state of each cell included in the logical shape 1020 . Since the initially generated logical reference pattern was generated using a specific rule (e.g., Rule #25) of cellular automata, each cell included in the logical shape 1020 generated from the reflected image is also cellular automata (eg Rule #25). This is because it can satisfy a specific rule (eg Rule #25) of cellular automata). Cells that have passed verification may be defined as connected components in a pre-generated logical reference pattern. For example, a connected component may mean a cell continuously connected in a pre-generated logical reference pattern, and it can be inferred that the depth of the corresponding part is a part having a uniform depth to the extent that distortion does not occur. Accordingly, the depth of each cell included in the connected component can be defined as being uniform. The electronic device 101 may calculate parallax movement from a logical reference pattern based on at least one connected component, and based on this, create depth information corresponding to the connected component. 12 to 14 may be referred to for a detailed method of generating depth information corresponding to connected components.

11A to 11E are diagrams illustrating a method of restoring at least a part of a pre-generated logical reference pattern from an image reflected by an external object using cellular automata of an electronic device according to various embodiments of the present disclosure; am.

According to various embodiments, referring to FIGS. 11A to 11E , the electronic device 101 may perform cellular automata using states of cells included in a logical shape. For example, the electronic device 101 may use the value 110101111 included in the first column of the logical shape as an initial value to perform cellular automata according to Rule #25. The electronic device 101 may allocate a separate operation column 1120 to recover the logical reference pattern. The electronic device 101 may store a value generated according to Rule #25 based on the first column in the calculation column 1120 . The electronic device 101 may compare the value of the calculation column 1120 with the value of the second column 1110 of the logical shape. For example, the state 1122 of the cell corresponding to the second row of the second column may correspond to 1 since s=1 and k=1.

According to various embodiments, referring to FIG. 11B , a value of an operation column 1120 corresponding to a second column in which cellular automata is performed based on the first column may be disclosed.

According to various embodiments of the present disclosure, referring to FIG. 11C , the electronic device 101 calculates a value included in a second column 1110 of a logical shape and a value of an operation column 1120 obtained through cellular automata. can be compared For example, the state of the cell 1111 corresponding to the first row of the second column 1110 of the logical shape is 0, and in the first row of the operation column 1120 obtained through cellular automata The state of the corresponding cell 1121 may also be the same as 0. In this case, the electronic device 101 may set the value of the cell 1131 corresponding to the first row of the first column 1130 as the reference column to 0. If the comparison results do not match, the value of the cell 1131 corresponding to the first row of the first column 1130 may be set to 1.

According to various embodiments of the present disclosure, referring to FIG. 11D , the electronic device 101 calculates a value included in a second column 1110 of a logical shape and a value of an operation column 1120 obtained through cellular automata. You can go ahead and compare. For example, the state of the cell 1112 corresponding to the second row of the second column 1110 of the logical shape is 1, and in the second row of the operation column 1120 obtained through cellular automata The state of the corresponding cell 1122 may also be the same as 1. In this case, the electronic device 101 may set the value of the cell 1132 corresponding to the second row of the first column 1130 as the reference column to 0. If the comparison results do not match, the value of the cell 1132 corresponding to the second row of the first column 1130 may be set to 1.

According to various embodiments of the present disclosure, referring to FIG. 11E , values included in the generated logical shape and values acquired through cellular automata, up to the eighth column 1140 of the image acquired by the electronic device 101 this can be shown. The electronic device 101 may continuously perform the comparison process over the entire area of the acquired image. After that, the electronic device 101 can check the area where the calculation value is 0 and determine it as a connected area.

12 is a diagram illustrating a method of generating a lookup map from a pre-generated logical reference pattern of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, referring to FIG. 12 , the electronic device 101 may record a coding word corresponding to each cell from a pre-generated logical reference pattern 1200 . For example, the electronic device 101 may store a coding word corresponding to each cell of a logical reference pattern, which may be defined as a look-up map 1210.

According to various embodiments, the electronic device 101 may check a plurality of patterns crossing at right angles including a specific cell of a pre-generated logical reference pattern. Patterns that cross at right angles can be designated as 8 lengths and widths. Coding words of unfilled parts of the intersection pattern may be stored as 0. For example, in the case of the first cross pattern, since it is 111 vertically, it may be 00000111, and it may be 00100000 horizontally. Accordingly, the first coded word 1210 corresponding to the first cross pattern may be 0000011100100000. For example, in the case of the second cross pattern, it may be 01111111 vertically and 00110111 horizontally. Accordingly, the second coded word 1220 corresponding to the second cross pattern may be 0111111100110111. For example, in the case of the third cross pattern, it may be 01111110 vertically and 00010000 horizontally. Accordingly, the third coded word 1230 corresponding to the third cross pattern may be 0111111000010000.

According to various embodiments, the electronic device 101 may map and store coded words corresponding to each cell in each cell included in the lookup map 1210 . For example, 36 is recorded in the cell 1201 of the lookup map 1210, which may mean that 36 different coding words are mapped.

13 is a diagram illustrating a method of obtaining connected components through at least some restored logical reference patterns of an electronic device and obtaining at least one coding word included in the connected components according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, referring to FIG. 13 , the electronic device 101 may check a connected component 1310 in a logical shape 1300 through the comparison method disclosed in FIG. 11 . The electronic device 101 may generate an intersection pattern included in the connected component 1310 . For example, the electronic device 101 may generate the first coded word 1321 (0000111100011001) and the second coded word 1322 (0000000000000100) through the first cross pattern. The electronic device 101 may store coded words 1320 that can be generated from connected components. Thereafter, the electronic device 101 compares the coded words 1320 that can be generated from the connected components with a lookup map corresponding to the pre-generated logical reference pattern to determine which part of the pre-generated logical reference pattern corresponds to the connected component. can

14 is a diagram illustrating a method of generating depth information of an external object through a lookup map of an electronic device according to various embodiments of the present disclosure.

According to various embodiments of the present disclosure, referring to FIG. 14 , the electronic device 101 compares a part of a logical reference pattern recovered from a connected component with a pre-generated logical reference pattern 1420 and part of an object corresponding to the connected component. It can be confirmed how much parallax shift has occurred. For example, the electronic device 101 may compare coded words and confirm that the first cell 1401 of the first connected component corresponds to the cell 1431 of the lookup map 1430 . The electronic device 101 can determine which part of the pre-generated logical reference pattern 1420 corresponds to the cell 1431 of the lookup map 1430, and can determine the degree of difference from the connected components. For example, in the case of the first connected component, it can be confirmed that parallax movement does not occur by comparison with a pre-generated logical reference pattern. For example, in the case of the second connected component including the second cell 1402, it can be confirmed that a disparity of 3 has occurred compared to a pre-generated logical reference pattern. The electronic device 101 may check the depth difference between the connected components by utilizing the separation between the different connected components. The electronic device 101 may perform 3D modeling by utilizing a plurality of connected components and respective depth differences. 17 and 18 may be referred to for 3D modeling of external objects of the electronic device 101 .

15 is a flowchart illustrating a method of obtaining a connected component from an image reflected by an external object of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, in the operation of FIG. 1510 , the electronic device 101 may obtain a physical shape of the structured light pattern based on an image reflected by an external object.

According to various embodiments, the electronic device 101 may arrange or transform the acquired physical shape into a logical shape in operation of FIG. 1520 .

According to various embodiments, the electronic device 101 may restore at least a part of the pre-generated logical reference pattern in the logical shape in operation of FIG. 1530 .

According to various embodiments, the electronic device 101 may obtain connected components based on at least some of the restored logical reference patterns in operation 1540 .

Thereafter, the electronic device 101 may perform the operation described in FIG. 15 and the subsequent operation in FIG. 16 .

16 is a flowchart illustrating a method of generating depth information corresponding to a connected component based on a coded word included in the connected component of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the electronic device 101 may obtain coded words included in connected components in operation of FIG. 1610 .

According to various embodiments, in operation of FIG. 1620, the electronic device 101 may compare acquired coded words with a lookup map stored based on a pre-generated logical reference pattern.

According to various embodiments of the present disclosure, in the operation of FIG. 1630 , based on the comparison, the electronic device 101 may determine which part of a pre-generated logical reference pattern corresponds to a connected component.

According to various embodiments of the present disclosure, in the operation of FIG. 1640 , the electronic device 101 may generate depth information corresponding to the connected component based on parallax movement of the connected component.

17 is a diagram illustrating performing 3D modeling using connected components of an electronic device according to various embodiments of the present disclosure.

According to various embodiments, the electronic device 101 may perform 3D modeling 1700 by attaching a plurality of different connected components. Depending on the relative parallax movement of each connected component, the connected components may be arranged at different depths. For example, referring to FIG. 17 , connected components may have different colors, and 3D modeling may be performed using the relativity of depth information of each connected component.

18A to 18B are diagrams illustrating performing 3D modeling and self-calibration using a complex structured light pattern of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 18A, the electronic device 101 projects a structured light pattern onto an external object 1810 through a projector 210, and obtains an image reflected on the external object 1810 through a camera 220. A drawing of a system for performing 3D modeling (1800) is disclosed.

Referring to FIG. 18B , a diagram for performing self-repair by the electronic device 101 using cellular automata is disclosed. The electronic device 101 may perform 3D modeling through connected components and perform self-correction on areas missing from the 3D modeling. The electronic device 101 can use cellular automata to deduce what state it is in in a normal case even to areas that have not been accurately restored due to data loss, and based on this, it can be used for self-calibration of 3D modeling. .

An electronic device according to various embodiments includes a camera; a light emitting module including one or more light emitting bodies; Memory; and a processor, wherein the processor projects structured light corresponding to a designated structured light pattern to an external object using the light emitting module, and the designated structured light pattern is distributed according to a designated rule. An image including polygons and a plurality of second polygons connecting at least some of the plurality of first polygons based on a designated repeating grid pattern, and the structured light reflected by the external object using the camera Obtaining, based on the obtained image, the specified repeating grid pattern and at least some of the first polygons are identified, and based on the identified specified repeating grid pattern and at least some of the first polygons, the specified repeating grid pattern and at least some of the first polygons are not identified. restoring other portions of the first polygons, and generating depth information about the external object based on positions of the at least some of the first polygons and the restored first polygons of the other portion; can be set.

The light emitting module may include at least one of a mask, a filter, and a metasurface formed to correspond to the designated structured light pattern.

The plurality of first polygons may be distributed based on a pattern generated through a designated cellular automata.

The plurality of first polygons may correspond to cells corresponding to a pattern generated through the designated cellular automata, and may be spaced apart from each other so as to have intervals between the plurality of first polygons.

The designated repeating grid pattern may be formed by filling empty spaces between adjacent first polygons among the plurality of second polygons.

The processor may be configured to identify a connected component that matches at least a portion of the specified structured light pattern, based on the identified repetitive grid pattern and at least some of the first polygons.

The processor may be configured to check a coding word included in the connected component.

Based on the coding word included in the connected component, it may be configured to determine which position of the designated structured light pattern the connected component corresponds to.

The processor compares a coded word included in the connected component with a lookup map stored corresponding to the designated structured light pattern, and identifies a location of the lookup map including the coded word included in the connected component, and Based on the determined location of the lookup map, a location of the connected component in the designated structured light pattern may be determined.

The processor checks a position shift between a position of the connected component in the acquired image and a position of a region corresponding to the connected component in the designated structured light pattern, and based on the position shift, the It can be set to generate depth information.

A method of an electronic device according to various embodiments of the present disclosure includes: projecting structured light corresponding to a designated structured light pattern to an external object by using a light emitting module; acquiring an image in which the structured light is reflected by the external object using the camera; checking a specified repeating grid pattern and at least some first polygons based on the obtained image; restoring other portions of the first polygons that are not identified based on the confirmed designated repeating grid pattern and at least some of the first polygons; and generating depth information of the external object based on positions of the at least some of the first polygons and the restored first polygons of the other part, wherein the designated structured light pattern comprises: It may include a plurality of first polygons distributed according to a specified rule and a plurality of second polygons connecting at least some of the plurality of first polygons based on the specified repeating grid pattern.

The light emitting module may include at least one of a mask, a filter, and a metasurface formed to correspond to the designated structured light pattern.

The plurality of first polygons may be distributed based on a pattern generated through a designated cellular automata.

The plurality of first polygons may correspond to cells corresponding to a pattern generated through the designated cellular automata, and may be spaced apart from each other so as to have intervals between the plurality of first polygons.

The designated repeating grid pattern may be formed by filling empty spaces between adjacent first polygons among the plurality of second polygons.

An operation of identifying a connected component that matches at least a part of the designated structured light pattern based on the confirmed designated repeating grid pattern and at least some of the first polygons.

An operation of checking a coding word included in the connected component may be included.

An operation of determining which position of the designated structured light pattern the connected component corresponds to based on a coding word included in the connected component.

comparing a coded word included in the connected component with a lookup map stored corresponding to the designated structured light pattern, and checking a location of the lookup map including the coded word included in the connected component; and determining a location of the connected component in the designated structured light pattern based on the confirmed location of the lookup map.

checking a position shift between a position of the connected component in the acquired image and a position of a region corresponding to the connected component in the designated structured light pattern; and generating the depth information based on the location movement.

Claims (20)

In electronic devices,
camera;
a light emitting module including one or more light emitting bodies;
Memory; and
It includes a processor, the processor comprising:
Projecting structured light corresponding to a designated structured light pattern to an external object using the light emitting module;
The designated structured light pattern includes a plurality of first polygons distributed according to a designated rule and a plurality of second polygons connecting at least some of the plurality of first polygons based on a designated repeating grid pattern. a complex structured light pattern including a structured light pattern generated based on a pattern and a logical reference pattern;
Obtaining an image in which the structured light is reflected by the external object using the camera;
Based on the obtained image, check the specified repeating grid pattern and at least some of the first polygons;
Reconstructing other unidentified first polygons based on the confirmed designated repeating grid pattern and at least some of the first polygons;
Based on the obtained image, at least a part of the logical reference pattern is restored, and
Based on the positions of the at least some of the first polygons and the restored first polygons of the other part, and the restored logical reference pattern, the electronic method configured to generate depth information of the external object. Device.
According to claim 1,
The light emitting module,
An electronic device including at least one of a mask, a filter, and a metasurface formed to correspond to the designated structured light pattern.
◈Claim 3 was abandoned when the registration fee was paid.◈ According to claim 1,
The plurality of first polygons,
Distributed electronic devices based on patterns generated through designated cellular automata.
According to claim 1,
The plurality of first polygons,
An electronic device that corresponds to cells corresponding to the pattern generated through the designated cellular automata and is spaced apart and distributed so as to have intervals between the plurality of first polygons.
According to claim 4,
The specified repeating grid pattern,
An electronic device formed by filling empty spaces between adjacent first polygons among the plurality of first polygons with the plurality of second polygons.
According to claim 1,
the processor,
An electronic device configured to identify a connected component matching at least a portion of the specified structured light pattern, based on the confirmed specified repeating grid pattern and at least a portion of the first polygons.
According to claim 6,
the processor,
An electronic device configured to check a coding word included in the connected component.
According to claim 7,
the processor,
An electronic device configured to determine which position of the designated structured light pattern the connected component corresponds to based on a coding word included in the connected component.
According to claim 8,
the processor,
comparing a coded word included in the connected component with a lookup map stored corresponding to the designated structured light pattern to determine a location of the lookup map including the coded word included in the connected component; and
An electronic device configured to determine a location of the connected component in the designated structured light pattern based on the confirmed location of the lookup map.
According to claim 9,
the processor,
Checking a positional shift between a position of the connected component in the acquired image and a position of a region corresponding to the connected component in the designated structured light pattern; and
An electronic device configured to generate the depth information based on the location movement.
In the electronic device method,
projecting structured light corresponding to a designated structured light pattern to an external object by using a light emitting module;
obtaining an image in which the structured light is reflected by the external object using a camera;
checking a specified repeating grid pattern and at least some first polygons based on the obtained image;
restoring other portions of the first polygons that are not identified based on the confirmed designated repeating grid pattern and at least some of the first polygons;
restoring at least a part of a logical reference pattern based on the obtained image; and
generating depth information of the external object based on the positions of the at least some of the first polygons and the restored first polygons of the other part, and the restored logical reference pattern; include,
The designated structured light pattern includes a plurality of first polygons distributed according to a designated rule and a plurality of second polygons connecting at least some of the plurality of first polygons based on the designated repeating grid pattern. A method of an electronic device comprising a structured light pattern and a complex structured light pattern including a structured light pattern generated based on the logical reference pattern.
◈Claim 12 was abandoned when the registration fee was paid.◈ According to claim 11,
The light emitting module,
A method of an electronic device including at least one of a mask, a filter, and a metasurface formed to correspond to the designated structured light pattern.
◈Claim 13 was abandoned when the registration fee was paid.◈ According to claim 11,
The plurality of first polygons,
A method of distributed electronics based on patterns generated through designated cellular automata.
◈Claim 14 was abandoned when the registration fee was paid.◈ According to claim 11,
The plurality of first polygons,
The method of the electronic device in which the cells corresponding to the pattern generated through the designated cellular automata are spaced apart and distributed so as to have intervals between the plurality of first polygons.
◈Claim 15 was abandoned when the registration fee was paid.◈ According to claim 14,
The specified repeating grid pattern,
The method of the electronic device formed by filling the empty space between adjacent first polygons among the plurality of first polygons with the plurality of second polygons.
◈Claim 16 was abandoned when the registration fee was paid.◈ According to claim 11,
and identifying a connected component that matches at least a portion of the specified structured light pattern, based on the confirmed specified repeating grid pattern and at least a portion of the first polygons.
◈Claim 17 was abandoned when the registration fee was paid.◈ According to claim 16,
The method of the electronic device comprising an operation of checking a coding word included in the connected component.
◈Claim 18 was abandoned when the registration fee was paid.◈ According to claim 17,
and determining which position of the designated structured light pattern the connected component corresponds to, based on a coding word included in the connected component.
◈Claim 19 was abandoned when the registration fee was paid.◈ According to claim 18,
comparing a coded word included in the connected component with a lookup map stored corresponding to the designated structured light pattern, and checking a location of the lookup map including the coded word included in the connected component; and
and determining a location of the connected component in the designated structured light pattern based on the confirmed location of the lookup map.
◈Claim 20 was abandoned when the registration fee was paid.◈ According to claim 19,
checking a position shift between a position of the connected component in the acquired image and a position of a region corresponding to the connected component in the designated structured light pattern; and
The method of the electronic device comprising an operation of generating the depth information based on the location movement.

Images