WO2023106711A1

WO2023106711A1 - Electronic device including lidar device and operation method thereof

Info

Publication number: WO2023106711A1
Application number: PCT/KR2022/018874
Authority: WO
Inventors: 조세현; 김철귀; 윤종민
Original assignee: 삼성전자 주식회사
Priority date: 2021-12-06
Filing date: 2022-11-25
Publication date: 2023-06-15
Also published as: KR20230084978A

Abstract

An electronic device according to the embodiments of the present disclosure may comprise a LiDAR device including a plurality of transmission photodiodes and a plurality of reception diodes, a processor for controlling an operation of the LiDAR device, and a memory operatively connected to the processor. The memory may store instructions causing, when executed, the processor to perform control so that, among the plurality of transmission photodiodes, some of first transmission photodiodes are turned on, and second transmission photodiodes excluding the first transmission photodiodes are turned off. Depth values of objects in a scanned space may be calculated by operating the plurality of reception diodes or some reception diodes. On the basis of the depth values of the objects, a first area having a high density of the objects and a second area having a low density of the objects may be distinguished. The processor may perform control so that the first area is determined to be an additional light-emitting area, and some third transmission photodiodes among the transmission photodiodes corresponding to the first area are turned on. The processor may perform control so that the depth values of the objects in an additionally scanned area are calculated by operating the plurality of reception diodes or some reception diodes. Various other embodiments are also possible.

Description

Electronic device including LiDAR device and its operating method

Various embodiments of the present disclosure relate to an electronic device including a lidar device capable of reducing power consumption of the lidar device and an operating method thereof.

An electronic device may measure a distance to objects using a lidar device. A plurality of transmitting photodiodes are disposed in the transmitting module of the lidar device, and a plurality of receiving diodes are disposed in the receiving module. When scanning a space with a LIDAR device, there may be a region with a high density of objects and an area with a low density of objects. In a typical LIDAR device, power consumption may be increased because all transmission photodiodes light up (on) regardless of the density of objects.

In a typical lidar device, power consumption of the lidar device is increased by turning on all transmission photodiodes regardless of the density of objects. Power consumption can be reduced by reducing the number of transmit photodiode arrays. However, resolution is lowered as much as the transmission photodiode array is reduced, making it impossible to perform detailed spatial mapping. Various embodiments of the present disclosure may provide an electronic device capable of reducing power consumption without deteriorating the resolution of a lidar device and an operating method thereof. According to various embodiments of the present disclosure, an additional light emitting area is set according to a change in the viewing direction of a human mounted device (HMD), and a part of transmission photodiodes are additionally emitted for the additional light emitting area, thereby increasing the resolution. It is possible to provide an electronic device capable of reducing power consumption of a lidar device and an operating method thereof.

The technical tasks to be achieved in this document are not limited to the technical tasks mentioned above, and other technical tasks not mentioned can be clearly understood by those skilled in the art from the description below. there is.

An electronic device according to embodiments of the present disclosure includes a lidar device including a plurality of transmit photodiodes and a plurality of receive diodes, a processor for controlling an operation of the lidar device, and a memory operatively connected to the processor. can include When the memory is executed, the processor causes some of the first transmission photodiodes among the plurality of transmission photodiodes to be turned on, and the second transmission photodiodes other than the first transmission photodiodes to be turned off ( off) can be controlled. Depth values of objects in the scanned space may be calculated by operating the plurality of receiving diodes or a portion of the receiving diodes. Based on the depth values of the objects, a first area having a high density of the objects and a second area having a low density of the objects may be distinguished. The first area may be determined as an additional light emitting area, and some third transmit photodiodes among transmit photodiodes corresponding to the first area may be controlled to be turned on. Instructions for controlling the operation of the plurality of receiving diodes or some of the receiving diodes to calculate depth values of objects within an additionally scanned area may be stored.

In the operating method of an electronic device according to embodiments of the present disclosure, some first transmission photodiodes among a plurality of transmission photodiodes included in a LIDAR device are turned on, except for the first transmission photodiodes. The second transmission photodiodes may be controlled to be turned off. A plurality of receiving diodes or some receiving diodes included in the LIDAR device are operated to calculate depth values of objects in the scanned space, and according to the depth values of the objects, the objects are compared to a first region with high density. It is possible to distinguish the second region with low . The first area may be determined as an additional light emitting area, and some third transmit photodiodes among transmit photodiodes corresponding to the first area may be controlled to be turned on. Depth values of objects in an additionally scanned area may be calculated by operating the plurality of receiving diodes or some of the receiving diodes.

An electronic device according to embodiments of the present disclosure may additionally emit light from some transmission photodiodes in an area of high density of objects, thereby increasing resolution and reducing power consumption of the LIDAR device.

An electronic device according to embodiments of the present disclosure sets an additional light emitting area according to a change in a direction in which a human mounted device (HMD) looks, and additionally emits light from some transmission photodiodes for the additional light emitting area, thereby increasing the resolution. It is possible to reduce the power consumption of the LiDAR device along with the height.

In addition to this, various effects identified directly or indirectly through this document may be provided.

Features and advantages of certain embodiments and other aspects of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

2 is a block diagram of a display module according to various embodiments.

3A is a diagram illustrating a first state (eg, an unfolded state or an open state) of an electronic device according to various embodiments of the present disclosure.

3B is a diagram illustrating a second state (eg, a folded state or a closed state) of an electronic device according to various embodiments of the present disclosure.

4A is a perspective view of the front of an electronic device according to various embodiments of the present disclosure.

4B is a perspective view of a rear surface of an electronic device according to various embodiments of the present disclosure.

5 is a diagram illustrating an electronic device according to various embodiments of the present disclosure.

6 is a diagram illustrating a lidar device according to various embodiments of the present disclosure.

FIG. 7 is a diagram showing that a high-density area and a low-density area of objects are scanned.

8 is a diagram showing that power consumption is reduced by emitting light from some of all transmission cells (transmission photodiodes).

9 is a diagram illustrating scanning of an additional light emitting area by setting an area having a high density of objects as an additional light emitting area.

10 is a diagram illustrating scanning of an additional light emitting area by setting it as an additional light emitting area according to movement of a human mounted device (HMD) (or augmented reality (AR) glass (or electronic device)).

11 is a diagram illustrating a method of operating an electronic device according to various embodiments of the present disclosure.

12 is a diagram illustrating an operating method of an electronic device according to various embodiments of the present disclosure.

13 is a diagram illustrating an operating method of an electronic device according to various embodiments of the present disclosure.

Like reference numbers are used throughout the drawings to indicate like elements.

The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and equivalents thereof. It contains various specific details for illustrative purposes, but these should be regarded as illustrative only. Accordingly, those skilled in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are not limited to their literature meanings and are only used by the inventors to provide a clear and consistent understanding of the present invention. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the present invention is provided for purposes of illustration only and is not intended to limit the present invention as defined by the appended claims and equivalents thereof.

Expressions for the singular form are to be understood as including the plural referent unless the context clearly dictates otherwise. Thus, for example, reference to “a component surface” may include reference to one or more of such surfaces.

1 is a block diagram of an electronic device 101 within a network environment 100 according to various embodiments of the present disclosure.

Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into one component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers commands or data received from other components (eg, sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to an embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of this document is not limited to the aforementioned devices.

Various embodiments of the present disclosure and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited. A (eg, first) component is said to be "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and is interchangeably with terms such as, for example, logic, logical blocks, parts, or circuits. can be used A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure provide one or more instructions stored in a storage medium (eg, the internal memory 136 or the external memory 138) readable by a machine (eg, the electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include a single object or a plurality of objects, and some of the multiple objects may be separately disposed in other components. . According to various embodiments, one or more components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by modules, programs, or other components are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

According to an embodiment, the display module 160 shown in FIG. 1 is a variable display (eg, a y-axis direction of FIG. 4A) configured to expand an area (eg, screen size) in a first direction (eg, the y-axis direction of FIG. 4A). Flexible display) (eg, stretchable display) (eg, expandable display) may be included.

According to an embodiment, the display module 160 shown in FIG. 1 has an area (eg, the y-axis direction of FIG. 4A) in a second direction (eg, the x-axis direction of FIG. 4A) orthogonal to the first direction (eg, the y-axis direction). screen size) may include a flexible display (eg, a flexible display) (eg, a stretchable display) (eg, an expandable display) configured to be expandable.

According to an embodiment, the display module 160 shown in FIG. 1 extends an area (eg, screen size) in the first direction (eg, the y-axis direction) and the second direction (eg, the x-axis direction). It may include a flexible display (eg, flexible display) (eg, stretchable display) (eg, expandable display) configured to be

According to one embodiment, the display module 160 illustrated in FIG. 1 may include a flexible display configured to be folded or unfolded.

According to an embodiment, the display module 160 illustrated in FIG. 1 may include a flexible display that is slidably disposed to provide a screen (eg, a display screen).

According to an embodiment, the display module 160 may also be referred to as a flexible display (eg, a stretchable display), an expandable display, or a slide-out display.

According to one embodiment, the display module 160 shown in FIG. 1 may include a bar-type or plate-type display.

2 is a block diagram of a display module 160 according to various embodiments.

Referring to FIG. 2 , the display module 160 includes a display 200 and a display driver IC 230 (hereinafter referred to as 'DDI 230') for controlling the display 200. can include

The DDI 230 may include an interface module 231 , a memory 233 (eg, a buffer memory), an image processing module 235 , and/or a mapping module 237 .

According to an embodiment, the DDI 230 transmits image data or image information including image control signals corresponding to commands for controlling the image data to an electronic device (eg, FIG. 1 ) through the interface module 231. It can be received from other components of the electronic device 101).

According to an embodiment, the image information is independent of the function of a processor (eg, the processor 120 of FIG. 1) (eg, the main processor 121 of FIG. 1) (eg, the application processor) or the main processor 121. It may be received from an operating auxiliary processor (eg, the auxiliary processor 123 of FIG. 1 ) (eg, a graphic processing unit).

According to an embodiment, the DDI 230 may communicate with the touch circuit 250 or the sensor module 176 through the interface module 231 . Also, the DDI 230 may store at least some of the received image information in the memory 233 . As an example, the DDI 230 may store at least some of the received image information in the memory 233 in units of frames.

According to an embodiment, the image processing module 235 pre-processes or post-processes at least a portion of the image data based on characteristics of the image data or characteristics of the display 200 (eg, resolution, brightness, or size adjustment). ) can be performed.

According to an embodiment, the mapping module 237 may generate a voltage value or a current value corresponding to the image data pre-processed or post-processed through the image processing module 235 . As an embodiment, the generation of the voltage value or the current value is at least partially dependent on the properties of the pixels of the display 200 (eg, the arrangement of pixels (RGB stripe or pentile structure) or the size of each sub-pixel). can be performed based on

As an example, at least some pixels of the display 200 are driven based at least in part on the voltage value or current value, so that visual information (eg, text, image, and/or icon) corresponding to the image data is displayed. It can be displayed through (200).

According to one embodiment, the display module 160 may further include a touch circuit 250 . The touch circuit 250 may include a touch sensor 251 and a touch sensor IC 253 for controlling the touch sensor 251 .

As an example, the touch sensor IC 253 may control the touch sensor 251 to detect a touch input or a hovering input to a specific location of the display 200 . For example, the touch sensor IC 253 may detect a touch input or a hovering input by measuring a change in a signal (eg, voltage, light amount, resistance, or charge amount) for a specific position of the display 200 . The touch sensor IC 253 may provide information (eg, position, area, pressure, or time) related to the sensed touch input or hovering input to a processor (eg, the processor 120 of FIG. 1 ).

According to an embodiment, at least a part of the touch circuit 250 (eg, the touch sensor IC 253) may be included as a part of the display driver IC 230 or the display 200.

According to an embodiment, at least a part of the touch circuit 250 (eg, the touch sensor IC 253) is included as part of other components (eg, the auxiliary processor 123) disposed outside the display module 160. can

According to an embodiment, the display module 160 further includes at least one sensor of the sensor module 176 (eg, an extension detection sensor, a fingerprint sensor, an iris sensor, a pressure sensor, or an illumination sensor), or a control circuit therefor. can do. In this case, the at least one sensor or a control circuit thereof may be embedded in a part of the display module 160 (eg, the display 200 or the DDI 230) or a part of the touch circuit 250. For example, when the sensor module 176 embedded in the display module 160 includes a biometric sensor (eg, a fingerprint sensor), the biometric sensor provides biometric information associated with a touch input through a partial area of the display 200. (e.g. fingerprint image) can be acquired. For another example, if the sensor module 176 embedded in the display module 160 includes a pressure sensor, the pressure sensor may acquire pressure information associated with a touch input through a part or the entire area of the display 200. can For another example, when the sensor module 176 embedded in the display module 160 includes an expansion detection sensor, the expansion detection sensor detects a change in area (eg, screen size) of a display (eg, a flexible display). can sense According to an embodiment, the touch sensor 251 or the sensor module 176 may be disposed between pixels of a pixel layer of the display 200 or above or below the pixel layer.

3A is a diagram illustrating a first state (eg, an unfolded state or an open state) of an electronic device according to various embodiments of the present disclosure. 3B is a diagram illustrating a second state (eg, a folded state or a closed state) of an electronic device according to various embodiments of the present disclosure.

Referring to FIGS. 3A and 3B , an electronic device 300 (eg, the electronic device 101 of FIG. 1 ) includes a housing 310 and a display 320 disposed in a space formed by the housing 310. can include As an example, the display 320 may include a flexible display or a foldable display.

The surface on which the display 320 is disposed may be defined as a first surface or a front surface of the electronic device 300 (eg, a surface on which a screen is displayed when unfolded). Also, a surface opposite to the front surface may be defined as a second surface or a rear surface of the electronic device 300 . Also, a surface surrounding the space between the front and rear surfaces may be defined as a third surface or a side surface of the electronic device 300 . For example, the electronic device 300 may fold or unfold the folding area 323 in a first direction (eg, an x-axis direction) based on a folding axis (eg, an A-axis).

As an embodiment, the housing 310 includes a first housing structure 311, a second housing structure 312 including a sensor area 324, a first rear cover 380, a hinge cover 313, And it may include a second rear cover (390). The housing 310 of the electronic device 101 is not limited to the shape and combination shown in FIGS. 3A and 3B , and may be implemented by other shapes or combinations and/or combinations of parts. For example, in another embodiment, the first housing structure 311 and the first rear cover 380 may be integrally formed, and the second housing structure 312 and the second rear cover 390 may be integrally formed. can be formed

As an embodiment, the first housing structure 311 and the second housing structure 312 are disposed on both sides of the folding axis A, and may have a generally symmetrical shape with respect to the folding axis A. . The first housing structure 311 and the second housing structure 312 determine whether the state of the electronic device 300 is an unfolded state (eg, a first state), a folded state (eg, a second state), or an intermediate state ( Example: the third state), an angle or a distance formed from each other may be different.

As an embodiment, the second housing structure 312, unlike the first housing structure 311, the sensor area 324 where various sensors (eg, an illuminance sensor, an iris sensor, and/or an image sensor) are disposed. ), but may have mutually symmetrical shapes in other regions.

As an embodiment, at least one sensor (eg, a camera module, an illuminance sensor, an iris sensor, and/or an image sensor) may be disposed in the lower portion and/or bezel area of the display as well as the sensor area 324 .

As an example, the first housing structure 311 and the second housing structure 312 may form a recess accommodating the display 320 together. In the illustrated embodiment, due to the sensor area 324 , the recess may have two or more different widths in a direction orthogonal to the folding axis A (eg, an x-axis direction).

For example, the recess is formed at the edge of the sensor region 324 of the first portion 311a of the first housing structure 311 and the second housing structure 312. It may have a first width W1 between the first portions 312a. The recess corresponds to the second portion 311b of the first housing structure 311 parallel to the folding axis A of the first housing structure 311 and the sensor area 324 of the second housing structure 312. It may have a second width W2 formed by the second part 312b of the second housing structure 312 parallel to the folding axis A without being bent. In this case, the second width W2 may be longer than the first width W1. In other words, the first part 311a of the first housing structure 311 and the first part 312a of the second housing structure 312 having mutually asymmetric shapes form the first width W1 of the recess. can do. The second portion 311b of the first housing structure 311 and the second portion 312b of the second housing structure 312 having mutually symmetrical shapes may form the second width W2 of the recess. .

As an example, the first part 312a and the second part 312b of the second housing structure 312 may have different distances from the folding axis A. The width of the recess is not limited to the illustrated example. In various embodiments, the recess may have a plurality of widths due to the shape of the sensor area 324 or the asymmetrical shape of the first housing structure 311 and the second housing structure 312 .

As an example, at least a portion of the first housing structure 311 and the second housing structure 312 may be formed of a metal material or a non-metal material having a rigidity of a size selected to support the display 320 .

As an example, the sensor area 324 may be formed to have a predetermined area adjacent to one corner of the second housing structure 312 . However, the arrangement, shape, and size of the sensor area 324 are not limited to the illustrated example. For example, in another embodiment, the sensor area 324 may be provided at another corner of the second housing structure 312 or any area between the top corner and the bottom corner.

As an embodiment, components for performing various functions embedded in the electronic device 300 are electronically transmitted through the sensor area 324 or through one or more openings provided in the sensor area 324. It may be exposed on the front surface of the device 300 . In various embodiments, the components may include various types of sensors. The sensor may include, for example, at least one of an illuminance sensor, a front camera (eg, a camera module), a receiver, or a proximity sensor.

The first rear cover 380 is disposed on one side of the folding axis A on the rear side of the electronic device, may have, for example, a substantially rectangular periphery, and the first housing structure 311 ) The edge may be wrapped by. Similarly, the second rear cover 390 may be disposed on the other side of the folding axis A on the rear surface of the electronic device, and its edge may be wrapped by the second housing structure 312 .

In the illustrated embodiment, the first rear cover 380 and the second rear cover 390 may have substantially symmetrical shapes around the folding axis A. However, the first rear cover 380 and the second rear cover 390 do not necessarily have symmetrical shapes, and in another embodiment, the electronic device 300 includes various shapes of the first rear cover 380 and A second rear cover 390 may be included. In another embodiment, the first rear cover 380 may be integrally formed with the first housing structure 311, and the second rear cover 390 may be integrally formed with the second housing structure 312. there is.

As an embodiment, the first rear cover 380, the second rear cover 390, the first housing structure 311, and the second housing structure 312 are various parts of the electronic device 300 (eg : A space in which a printed circuit board or battery) can be disposed can be formed. As an example, one or more components may be disposed or visually exposed on the back of the electronic device 300 . For example, at least a portion of the sub display 330 may be visually exposed through the first rear area 382 of the first rear cover 380 . In another embodiment, one or more parts or sensors may be visually exposed through the second rear area 392 of the second rear cover 390 . In various embodiments, the sensor may include an illuminance sensor, a proximity sensor, and/or a rear camera.

As an example, the hinge cover 313 may be disposed between the first housing structure 311 and the second housing structure 312 to cover an internal part (eg, a hinge structure). . The hinge cover 313 may cover a portion where the first housing structure 311 and the second housing structure 312 come into contact with each other when the electronic device 300 is expanded and folded.

As an embodiment, the hinge cover 313 may include a first housing structure 311 and a second housing structure (according to a state (flat state or folded state) of the electronic device 101). 312) or may be exposed to the outside.As an embodiment, when the electronic device 101 is in an unfolded state, the hinge cover 313 is formed by the first housing structure 311 and the second housing structure. It may not be exposed because it is covered by 312. As an example, when the electronic device 101 is in a folded state (eg, fully folded state), the hinge cover 313 is a first housing structure. 311 and may be exposed to the outside between the second housing structure 312. As an embodiment, the first housing structure 311 and the second housing structure 312 form a predetermined angle (folded with a In the case of an intermediate state at a certain angle, the hinge cover 313 may be partially exposed to the outside between the first housing structure 311 and the second housing structure 312. However, in this case, the exposed area may be less than a completely folded state As an embodiment, the hinge cover 313 may include a curved surface.

The display 320 may be disposed on a space formed by the housing 310 . For example, the display 320 is seated on a recess formed by the housing 310 and may constitute most of the front surface of the electronic device 300 .

Accordingly, the front surface of the electronic device 300 may include the display 320 , a partial area of the first housing structure 311 adjacent to the display 320 , and a partial area of the second housing structure 312 . Further, the rear surface of the electronic device 300 includes the first rear cover 380, a partial area of the first housing structure 311 adjacent to the first rear cover 380, the second rear cover 390, and the second rear cover. A portion of the second housing structure 312 adjacent to 390 may be included.

The display 320 may refer to a display in which at least a partial area may be deformed into a flat or curved surface. As an embodiment, the display 320 includes a folding area 323, a first area 321 disposed on one side (eg, left side in FIG. 3A) and the other side (right side in FIG. 3A) based on the folding area 323. It may include a second area 322 disposed on.

As an example, the display 320 may include a top emission or bottom emission type OLED display. An OLED display may include a low temperature color filter (LTCF) layer, window glass (e.g., ultra-thin glass (UTG) or polymer window), and an optical compensation film (e.g., optical compensation film (OCF)). . Here, the polarizing film (or polarizing layer) can be replaced with the LTCF layer of the OLED display.

The area division of the display 320 is exemplary, and the display 320 may be divided into a plurality of (eg, two or more) areas according to a structure or function. As an embodiment, the area of the display 320 may be divided by the folding area 323 extending parallel to the y-axis or the folding axis A, but in another embodiment, the display 320 may have another folding area ( For example, the region may be divided based on a folding region parallel to the x-axis) or another folding axis (eg, a folding axis parallel to the x-axis).

As an example, the first region 321 and the second region 322 may have generally symmetrical shapes around the folding region 323 .

Hereinafter, the operation and display 320 of the first housing structure 311 and the second housing structure 312 according to the state of the electronic device 300 (eg, a flat state and a folded state) Each area of is described.

As an example, when the electronic device 300 is in a flat state (eg, FIG. 3A ), the first housing structure 311 and the second housing structure 312 form an angle of about 180 degrees and move in the same direction. can be placed facing up. The surface of the first area 321 and the surface of the second area 322 of the display 320 form an angle of about 180 degrees to each other and may face the same direction (eg, the front surface of the electronic device). The folding region 323 may form substantially the same plane as the first region 321 and the second region 322 .

As an example, when the electronic device 300 is in a folded state (eg, FIG. 3B ), the first housing structure 311 and the second housing structure 312 may face each other. The surface of the first area 321 and the surface of the second area 322 of the display 320 form a narrow angle (eg, between 0 degrees and about 10 degrees) and may face each other. At least a portion of the folding region 323 may be formed of a curved surface having a predetermined curvature.

As an example, when the electronic device 300 is in a half folded state, the first housing structure 311 and the second housing structure 312 may be disposed at a certain angle to each other. there is.

Electronic devices according to various embodiments of the present document may include electronic devices such as a bar type, a foldable type, a rollable type, a sliding type, a wearable type, a tablet PC, and/or a notebook PC. The electronic device 300 according to various embodiments of the present document is not limited to the above example and may include various other electronic devices.

4A is a perspective view of the front of an electronic device according to various embodiments of the present disclosure. 4B is a perspective view of a rear surface of an electronic device according to various embodiments of the present disclosure.

Referring to FIGS. 4A and 4B , an electronic device 400 (eg, the electronic device 101 of FIG. 1 ) according to various embodiments of the present disclosure has a first surface (or front surface) 410A and a second surface. (or rear) 410B, and housing 410 . A display 401 (eg, the display module 160 of FIG. 1 ) may be disposed in the space formed by the housing 410 . The housing 410 may include a side surface 410C surrounding a space between the first surface 410A and the second surface 410B. In another embodiment, the housing 410 may refer to a structure forming some of the first face 410A, the second face 410B, and the side face 410C.

According to one embodiment, the first surface 410A may be formed by at least a portion of a substantially transparent front plate 402 (eg, a glass plate or a polymer plate including various coating layers).

According to one embodiment, the second surface 410B may be formed by a substantially opaque back plate 411 . The rear plate 411 is formed, for example, of coated or tinted glass, ceramic, polymer, metal (eg, aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the foregoing materials. It can be. However, it is not limited thereto, and the back plate 411 may be formed of transparent glass.

According to one embodiment, the side surface 410C is coupled to the front plate 402 and the back plate 411 and is formed by a side bezel structure 418 (or "side member") including metal and/or polymer. can be formed In some embodiments, the back plate 411 and the side bezel structure 418 may be integrally formed and include the same material (eg, a metal material such as aluminum).

As an example, the front plate 402 may include two first regions 410D that are curved and seamlessly extended from the first surface 410A toward the rear plate 411 . The two first regions 410D may be disposed at both ends of a long edge of the front plate 402 .

As an example, the rear plate 411 may include two second regions 410E that are curved and seamlessly extended from the second surface 410B toward the front plate 402 .

In some embodiments, the front plate 402 (or the rear plate 411) may include only one of the first regions 410D (or the second regions 410E). In some embodiments, some of the first regions 410D or the second regions 410E may not be included. In embodiments, when viewed from the side of the electronic device 400, the side bezel structure 418 has the first area 410D or the second area 410E. It has a thickness (or width), and may have a second thickness smaller than the first thickness at a side surface including the first regions 410D or the second regions 410E.

According to an embodiment, the electronic device 400 includes a display 401 (eg, the display module 160 of FIG. 1 ), an audio input device 403 (eg, the input module 150 of FIG. 1 ), and Output device 407, 414 (eg, sound output module 155 in FIG. 1), sensor module 404, 419 (eg, sensor module 176 in FIG. 1), camera module 405, 412 (eg, : It may include at least one of the camera module 180 of FIG. 1), a flash 413, a key input device 417, an indicator (not shown), and

connectors

408 and 409. In some embodiments, the electronic device 400 may omit at least one of the components (eg, the key input device 417) or may additionally include other components.

According to one embodiment, the display 401 (eg, the display module 160 of FIG. 1 ) may be visually visible through an upper portion of the front plate 402 . In some embodiments, at least a portion of the display 401 may be visible through the front plate 402 forming the first surface 410A and the first area 410D of the side surface 410C. The display 401 may be combined with or disposed adjacent to a touch sensing circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and/or a digitizer that detects a magnetic field type stylus pen.

In some embodiments, at least one of a sensor module 404, a camera module 405 (eg, an image sensor), an audio module 414, and a fingerprint sensor is included on the rear surface of the screen display area of the display 401. can do.

According to some embodiments, the display 401 may be combined with or disposed adjacent to a touch sensing circuit, a pressure sensor capable of measuring the strength (pressure) of a touch, and/or a digitizer detecting a magnetic stylus pen. there is.

According to some embodiments, at least a portion of the

sensor modules

404 and 419 and/or at least a portion of the key input device 417 are located in the first areas 410D and/or the second area 410E. can be placed in the field.

According to one embodiment, the audio input device 403 may include a microphone. In some embodiments, the input device 403 may include a plurality of microphones arranged to sense the direction of sound. The

sound output devices

407 and 414 may include an external speaker 407 and a receiver for communication (eg, the audio module 414). In some embodiments, the audio input device 403 (eg, a microphone), the

audio output devices

407 and 414, and the

connectors

408 and 409 are disposed in the internal space of the electronic device 400 and formed in the housing 410. It may be exposed to the external environment through at least one hole. In some embodiments, the hole formed in the housing 410 may be commonly used for the audio input device 403 (eg, a microphone) and the

audio output devices

407 and 414 . In some embodiments, the

sound output devices

407 and 414 may include a speaker (eg, a piezo speaker) operated while excluding holes formed in the housing 410 .

According to an embodiment, the sensor modules 404 and 419 (eg, the sensor module 176 of FIG. 1 ) may provide electrical signals or data corresponding to an internal operating state of the electronic device 400 or an external environmental state. value can be created. The

sensor modules

404 and 419 may include, for example, a first sensor module 404 (eg, a proximity sensor) disposed on the first surface 410A of the housing 410 and/or a first sensor module 404 (eg, a proximity sensor) of the housing 410. A second sensor module 419 (eg, an HRM sensor) and/or a third sensor module (not shown) (eg, a fingerprint sensor) disposed on the second surface 410B may be included. For example, the fingerprint sensor may be disposed on the first surface 410A (eg, the display 401 ) and/or the second surface 410B of the housing 410 . The electronic device 400 includes a sensor module (not shown), for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a bio sensor, a temperature sensor, At least one of a humidity sensor and an illuminance sensor may be further included.

According to an embodiment, the

camera modules

405 and 412 include the first camera module 405 disposed on the first surface 410A of the electronic device 400 and the second camera module 405 disposed on the second surface 410B of the electronic device 400. A camera module 412 may be included. A flash 413 may be disposed around the

camera modules

405 and 412 . The

camera modules

405 and 412 may include one or more lenses, an image sensor, and/or an image signal processor. The flash 413 may include, for example, a light emitting diode or a xenon lamp.

As an example, the first camera module 405 may be disposed below the display panel of the display 401 in an under display camera (UDC) method. In some embodiments, two or more lenses (wide-angle and telephoto lenses) and image sensors may be disposed on one surface of the electronic device 400 . In some embodiments, a plurality of first camera modules 405 may be disposed on a first surface (eg, a surface on which a screen is displayed) of the electronic device 400 in an under display camera (UDC) method.

As an example, the key input device 417 may be disposed on the side surface 410C of the housing 410 . In another embodiment, the electronic device 400 may not include some or all of the above-mentioned key input devices 417, and the key input devices 417 that are not included may include other key input devices such as soft keys on the display 401. can be implemented in the form In some embodiments, the key input device 417 may be implemented using a pressure sensor included in the display 401 .

As an embodiment, the

connectors

408 and 409 include a first connector hole 408 capable of receiving a connector (eg, a USB connector) for transmitting and receiving power and/or data to and from an external electronic device; and / or a second connector hole 409 (or earphone jack) capable of receiving a connector for transmitting and receiving an audio signal to and from an external electronic device may be included. The first connector hole 408 may include a universal serial bus (USB) A type port or a USB C type port. When the first connector hole 408 supports USB type-C, the electronic device 400 (eg, the electronic device 101 of FIG. 1 ) may support USB power delivery (PD) charging.

As an embodiment, some of the camera modules 405 of the

camera modules

405 and 412 and/or some of the sensor modules 404 of the

sensor modules

404 and 419 are visually visible through the display 401. can be placed. As another example, when the camera module 405 is disposed in an under display camera (UDC) method, the camera module 405 may not be visually visible to the outside.

As an embodiment, the camera module 405 may be disposed overlapping with the display area, and the screen may be displayed in the display area corresponding to the camera module 405 . Some sensor modules 404 may be arranged to perform their functions without being visually exposed through the front plate 402 in the internal space of the electronic device.

Referring to FIG. 5 , an electronic device 500 according to various embodiments of the present disclosure may include augmented reality (AR) glasses. For example, the electronic device 500 may include a human mounted device (HMD).

As an embodiment, the electronic device 500 (eg, AR glasses) according to various embodiments of the present disclosure includes a visor unit 510, a display unit 520, a recognition camera unit 530, and ET (Eye Tracking) A camera unit 540, an LED light 550, a PCB unit 560, a battery unit 570, a speaker unit 580, a microphone unit 590, and a lidar device 600 may be included.

As an embodiment, the lidar device 600 may include a transmission module 601 and a reception module 602. For example, the lidar device 600 may be located in the center of the electronic device 500 (eg, AR glasses) to match the user's line of sight.

As an embodiment, the visor unit 510 may include a first visor 511 (eg, a visor for the right eye) and a second visor 512 (eg, a visor for the left eye). For example, the visor unit 510 may be located on the front or rear side of the display unit 520 (eg, a wave guide (or screen display unit)) to protect the display unit 520.

For example, the visor unit 510 may control transmission of external light incident to the display unit 520 (eg, a waveguide (or screen display unit)). The visor unit 510 may control transmission of external light through an electrochromic function in which a redox reaction occurs and the color is changed by applied power.

As an example, the display unit 520 may include screen display units 521 and 522 (or waveguides) and

display driving units

523 and 524 . For example, the

screen display units

521 and 522 may include a screen display unit 521 for the right eye and a screen display unit 522 for the left eye. For example, the

screen display units

521 and 522 may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), and an organic light emitting diode. (organic light emitting diode; OLED) or micro LED (micro light emitting diode; micro LED) may be included. For example, when the

screen display units

521 and 522 are formed of one of a liquid crystal display, a digital mirror display, and a silicon liquid crystal display, the electronic device 500 may include a light source for radiating light to the screen output area of the display. there is. As another example, when the

screen display units

521 and 522 can generate light by themselves, a good quality virtual image can be provided to the user even without a separate light source.

As an example, if the

screen display units

521 and 522 are implemented with organic light emitting diodes or micro LEDs, since a light source is unnecessary, the electronic device 500 can be lightened in weight. The user may use the electronic device 500 while wearing it on the face.

As an example, the

screen display units

521 and 522 may operate as waveguides (eg, waveguides) that transmit light to the user's eyes. The waveguide may be made of glass, plastic, or polymer, and may include a nanopattern formed on an inner or outer surface, for example, a polygonal or curved grating structure. According to an embodiment, light incident to one end of the waveguide may be propagated inside the display optical waveguide by nanopatterns and provided to the user. In addition, a waveguide composed of a free-form type prism may provide incident light to a user through a reflection mirror. For example, the waveguide may include at least one of at least one diffractive element (eg, a Diffractive Optical Element (DOE) or a Holographic Optical Element (HOE)) or a reflective element (eg, a reflective mirror). For example, the waveguide may guide the display light emitted from the light source unit to the user's eyes using at least one diffractive element or reflective element.

As an embodiment, the recognition camera unit 530 may include a first recognition camera 531 and a second recognition camera 532 . For example, the recognition camera unit 530 is a camera used for 3DoF, 6DoF head tracking, hand detection and tracking, and space recognition, and may include a Global Shutter (GS) camera. The camera unit 530 for recognition consists of two GS cameras (eg, a first camera 531 for recognition and a second camera 532 for recognition) because a stereo camera is required for head tracking and spatial recognition. It can be.

As an embodiment, the ET (Eye Tracking) camera unit 540 may include a first ET camera 541 (eg, a camera for recognizing a right eye) and a second ET camera 542 (eg, a camera for recognizing a left eye). can For example, the ET camera unit 540 may detect pupils (eg, right and left eyes) of the user and track the movement of the pupils. The center of a virtual image projected on the AR glasses by using the ET camera unit 540 may be positioned according to a direction in which a user wearing the AR glasses gazes.

As an example, the LED light 550 may be attached to the frame of the AR glasses. The LED light 550 may irradiate infrared wavelengths so that the pupil of the user is smoothly detected when photographing the pupil of the user with the ET (Eye Tracking) camera 540 . As an embodiment, the LED light 550 may be used as a means of supplementing the brightness of the surroundings when photographing the surroundings with the camera unit 530 for recognition.

As an embodiment, the PCB unit 560 may be disposed on a leg portion of the AR glass and may include a first PCB 561 and a second PCB 562. For example, the PCB unit 560 includes a visor unit 510, a display unit 520, a recognition camera unit 530, an ET (Eye Tracking) camera unit 540, an LED light 550, and a speaker unit 580. ), and at least one driving unit for controlling the microphone unit 590. Through the PCB unit 560, the visor unit 510, the display unit 520, the recognition camera unit 530, the ET (Eye Tracking) camera unit 540, the LED light 550, the speaker unit 580, And an electrical signal may be transmitted to the microphone unit 590 .

As an embodiment, the battery unit 570 may be disposed on a leg portion of the AR glasses and may include a first battery 571 and a second battery 572. through the battery unit 570

A visor unit 510, a display unit 520, a recognition camera unit 530, an ET (Eye Tracking) camera unit 540, an LED light 550, a PCB unit 560, a speaker unit 580, and Power for driving the microphone unit 590 may be supplied.

As an example, the speaker unit 580 may include a first speaker 581 (eg, a right speaker) and a second speaker 582 (eg, a left speaker). For example, the speaker unit 580 may output sound according to the control of the driving unit of the PCB unit 560. As an example,

As an embodiment, the microphone unit 590 includes a first microphone 591 (eg, a top microphone), a second microphone 592 (eg, a right microphone), and a third microphone 593 (eg, a top microphone). left microphone). For example, the user's voice and external sound may be converted into electrical signals through the microphone unit 590 . As an example, the first microphone 591 (eg, top microphone), the second microphone 592 (eg, right microphone), and the third microphone 593 (eg, left microphone) are condenser, dynamic ( moving coil and ribbon), a piezoelectric element, or a micro-electromechanicalsystems (MEMS) type microphone. FIG. 6 is a diagram illustrating a lidar device according to various embodiments of the present disclosure.

Referring to FIGS. 5 and 6 , a lidar apparatus 600 according to various embodiments of the present disclosure may include a transmission module 601 and a reception module 602 .

According to an embodiment, the transmission module 601 may include a transmission array unit 610 including a plurality of transmission photodiodes 611 and a diffractive optical unit 620 (eg, a diffractive optical element (DOE)). there is.

As an embodiment, a plurality of transmit photodiodes 611 may be arranged in an array form in the transmit array unit 610, and the plurality of transmit photodiodes 611 may be individually operated (eg, turned on/off). there is. For example, a plurality of transmit photodiodes 611 may be grouped into a plurality of channels and operated (eg, turned on/off). A plurality of transmission photodiodes 611 can be grouped into four channels and operated (eg, on/off) for each channel. Here, four channels may be simultaneously operated (eg, on/off), or only one of the four channels may be operated (eg, on/off).

As an example, the plurality of transmit photodiodes 611 may include vertical-cavity surface-emitting lasers (VCSELs). As an example, the plurality of transmit photodiodes 611 may include a laser, single-sided emitting laser or surface light emitting laser.

For example, the diffractive optical unit 620 may divide the laser beam generated by the transmission array unit 610 into a plurality of beams. For example, the diffraction optical unit 620 may expand or expand the laser beam generated by the transmission array unit 610 in one direction.

A laser beam emitted from the transmitting module 601 may be incident to the receiving module 602 after being reflected from an object.

According to one embodiment, the receiving module 602 may include a receiving array unit 630 and a calculating unit 640 .

For example, in the receiving array unit 630, a plurality of receiving diodes (eg, single photon photodiodes) may be arranged in an array (eg, single photon avalanche diode array (SPDA) form). For example, the plurality of receiving diodes may be They may operate individually (eg, on/off) or/or grouped into a plurality of channels (eg, on/off) The plurality of receiving diodes may include avalanche photo diodes (APDs). The receiving array unit 630 may amplify photoelectrons of the laser beam received by the receiving diodes through the APD, and transmit a measurement value according to the laser beam received by the receiving diodes to the calculating unit 640 .

For example, the calculator 640 may include a time to digital converter (TDC) logic for measuring the arrival time of each laser beam. The calculation unit 640 may measure the arrival time of each laser beam based on the measurement value of each laser beam. The calculation unit 640 may transfer the measurement result of the arrival time to a processor (eg, the processor 120 of FIG. 1 ).

As an embodiment, the processor (eg, the processor 120 of FIG. 1 ) may detect the distance to the object based on the measurement result of the arrival time of the laser beams received from the operation unit 640 .

Referring to FIGS. 5, 6, and 7, as an embodiment, a processor (eg, the processor 120 of FIG. 1) determines depth values of objects in a frame of an image based on distances to the objects. can be calculated. A processor (eg, the processor 120 of FIG. 1 ) may distinguish an area 710 with a high density of objects and an area 720 with a low density of objects within the scanned space 700 based on the depth values of the objects. there is. A processor (eg, the processor 120 of FIG. 1 ) may control the operation of the LIDAR device 600 by dividing an area 710 in which the density of objects is high and an area 720 in which the density of objects is low.

As another embodiment, a separate signal processing processor may be disposed inside the lidar device 600. The signal processing processor may include a microprocessor. The signal processing processor may detect distances to objects based on measurement results of arrival times of laser beams. The signal processing processor may transfer distance information to objects to a processor (eg, the processor 120 of FIG. 1 ). A processor (eg, the processor 120 of FIG. 1 ) may calculate depth values of objects in a frame of an image based on distances to the objects. A processor (eg, the processor 120 of FIG. 1 ) may distinguish an area 710 with a high density of objects and an area 720 with a low density of objects within the scanned space 700 based on the distance values of the objects. there is. A processor (eg, the processor 120 of FIG. 1 ) may control the operation of the LIDAR device 600 by dividing an area 710 in which the density of objects is high and an area 720 in which the density of objects is low.

Referring to FIGS. 5, 6, and 8 , as an embodiment, a processor (eg, the processor 120 of FIG. 1 ) may include first portions of all transmission cells 800 (eg, all transmission photodiodes). Driving of the LIDAR device 600 may be controlled so that the transmission cells 810 (eg, a plurality of first transmission photodiodes) are turned on. In addition, the processor (eg, the processor 120 of FIG. 1 ) may include the second transmission cells 820 (eg, a plurality of first transmission photodiodes) excluding the first transmission cells 810 (eg, a plurality of first transmission photodiodes). Driving of the lidar device 600 may be controlled so that the second transmission photodiodes) are turned off.

As an embodiment, the lidar device 600 performs first transmission of some of all transmission cells 800 (eg, all transmission photodiodes) under the control of a processor (eg, the processor 120 of FIG. 1 ). Cells 810 (eg, a plurality of first transmitting photodiodes) may be turned on. The lidar device 600 is a second transmission cell excluding the first transmission cells 810 (eg, a plurality of first transmission photodiodes) based on the control of a processor (eg, the processor 120 of FIG. 1 ). 820 (eg, a plurality of second transmitting photodiodes) may be turned off.

As an embodiment, the lidar device (eg, the lidar apparatus 600 of FIGS. 5 and 6 ) includes 1/2 first transmission cells among MxN transmission cells 800 (eg, all transmission photodiodes). 810 (eg, a plurality of first transmission photodiodes) are turned on, and 1/2 second transmission cells 820 (eg, a plurality of second transmission photodiodes) are turned off ( off) can be turned off.

For example, 1/2 first transmission cells 810 (eg, a plurality of first transmission photodiodes) among all transmission cells 800 (eg, all transmission photodiodes) are turned on; All receiving cells (eg, all receiving diodes) may be turned on. As another example, 1/2 first transmission cells 810 (eg, a plurality of first transmission photodiodes) among all transmission cells 800 (eg, all transmission photodiodes) are turned on , only 1/2 first receiving cells among all receiving cells (eg, all receiving diodes) may be turned on. For example, the calculation unit 640 may count arrival times of laser beams received from all receiving cells (or some receiving cells). Here, when the distance between the lidar device 600 and the objects is small, the count value of the arrival time is small, and the longer the distance between the lidar device 600 and the objects, the larger the count value of the arrival time. For example, the calculator 640 may convert a delay value of an arrival time of a laser beam received from all receiving cells (or some receiving cells) into a phase difference value.

For example, the calculation unit 640 converts the count value of the arrival time of all reception cells (or some reception cells) (or the phase difference value of the arrival time of all reception cells (or some reception cells)) to a processor (eg, the processor of FIG. 1 ). (120)). A processor (eg, the processor 120 of FIG. 1 ) based on the count value of the arrival time of all reception cells (or some reception cells) (or the phase difference value of the arrival time of all reception cells (or some reception cells)), An arrival time distribution or a phase difference distribution of all reception cells (or some reception cells) may be checked.

As an example, a processor (eg, the processor 120 of FIG. 1 ) performs 3D mapping of the scanned space based on the arrival time distribution of all receiving cells (or some receiving cells) and distance values (or depth values) of objects can do.

For example, a processor (eg, the processor 120 of FIG. 1 ) may calculate distance values of objects in a scanned space (eg, the scanned space 600 of FIG. 6 ), and an area in which the density of objects is high. (eg, the high-density region 710 of FIG. 7 ) and the low-density region of objects (eg, the low-density region 720 of FIG. 7 ) may be distinguished. In a part in which objects are densely located, count values of arrival times of the objects change densely, and accordingly, depth values of the objects may change densely. A processor (eg, the processor 120 of FIG. 1 ) determines an area where depth values of objects vary densely as an area of high density (eg, the area 710 of high density of FIG. 7 ), and An area in which the change in depth value is small may be determined as an area in which density of objects is low (eg, an area 720 in which density is low in FIG. 7 ).

Referring to FIG. 9 , as an embodiment, a processor (eg, the processor 120 of FIG. 1 ) determines an area in which the object density is high (eg, the area 710 in which the object density is high) and the density of the objects. It is possible to control the operation of the LIDAR device (eg, LIDAR device 600 in FIGS. 5 and 6 ) by dividing the area with low density (eg, the low density area 720 in FIG. 7 ).

As an embodiment, some (eg, 1/2) of the first transmission cells (eg, the first transmission cells 810 of FIG. 8 ) among all transmission cells 800 (eg, all transmission photodiodes) By turning on and scanning the space, the resolution of a high-density area may be lowered. A processor (eg, the processor 120 of FIG. 1 ) is configured to increase the resolution of an area where the object density is high (eg, the high density area 710 of FIG. 7 ). : The high-density region 710 of FIG. 7) may be determined as an additional region. A processor (eg, the processor 120 of FIG. 1 ) is a LIDAR device (eg, the LIDAR device 600 of FIGS. 5 and 6 ) so that the third transmission cells 830 corresponding to the additional area emit light (on). can control the operation of The lidar device (eg, the lidar apparatus 600 of FIGS. 5 and 6 ) is a third third corresponding to an additional area in which the density of objects is high based on the control of a processor (eg, the processor 120 of FIG. 1 ). Transmitting cells 830 may emit light (on). In addition, all receiving cells (eg, all receiving diodes) or some receiving cells corresponding to the third transmitting cells 830 may be turned on. In this way, the third transmission cells 830 additionally emit light for the additional emission area in the area where the density of objects is high (eg, the area 710 with high density in FIG. 7 ), so that the density of objects is high. A space and objects in an area (eg, a high-density area 710 in FIG. 7 ) may be scanned in detail.

Referring to FIGS. 5, 6, and 10 , according to an embodiment, an electronic device according to various embodiments of the present disclosure includes an inertial measurement device (eg, the sensor module 176 of FIG. 1) (eg, an inertial measurement unit)). The inertial measurement device (eg, the sensor module 176 of FIG. 1 ) may include a gyro sensor and an acceleration sensor.

As an embodiment, a processor (eg, processor 120 of FIG. 1 ) receives attitude and acceleration values from an inertial measurement device (eg, sensor module 176 of FIG. 1 ), and based on the attitude and acceleration values, It is possible to control the operation of the lidar device (eg, the lidar device 600 of FIGS. 5 and 6).

For example, the processor (eg, the processor 120 of FIG. 1 ) may determine that the attitude and position of the electronic device have changed when the attitude and acceleration values are equal to or greater than a preset reference value. Also, when the user wears the HMD (or AR glasses), the space scanned through the HMD (or AR glasses) may change according to the user's movement. The space to be scanned changes according to the movement of the user wearing the HMD, which can be detected by changes in posture and acceleration values.

As an embodiment, the processor (eg, the processor 120 of FIG. 1 ) may determine that the space scanned by the HMD (or AR glasses) is changed when the attitude and acceleration values are equal to or greater than preset reference values. When a space scanned by the HMD (or AR glasses) is changed, power consumption may increase if all

transmission cells

1010, 1020, and 1030 are turned on for the entire space. In order to reduce power consumption of the LIDAR device 600, the processor (eg, the processor 120 of FIG. 1 ) turns off the transmission cells 1010 corresponding to the existing light emitting region 1001 among all transmission cells 1000. (off), and the operation of the lidar device 600 may be controlled so that all or some of the

transmission cells

1020 and 1030 corresponding to the additional light emitting region 1002 to be additionally scanned emit light.

For example, the lidar device 600 turns off the transmission cells 1010 corresponding to the existing light emitting region 1001 based on the control of a processor (eg, the processor 120 of FIG. 1), and adds Some of the transmission cells 1020 among the

transmission cells

1020 and 1030 corresponding to the emission region 1002 may emit light. The lidar device 600 may turn off the remaining transmission cells 1030 except for some transmission cells 1020. In addition, all of the receiving cells (eg, all of the receiving diodes) or some of the receiving cells corresponding to the additional foot and area 1002 may be turned on. In this way, as the HMD (or AR glasses) moves, some transmission cells 1020 corresponding to the additional light emitting area may additionally emit light, so that space and objects in a new area to be scanned may be scanned in detail.

Referring to FIG. 11 , in operation 1110, a processor (eg, the processor 120 of FIG. 1 ) may determine a light emitting area from among the entire area.

In operation 1120, the processor (eg, the processor 120 of FIG. 1 ) selects some first transmission cells (eg, all transmission photodiodes) from among all transmission cells (eg, all transmission cells 800 of FIG. 8 ). The operation of the lidar device (eg, the lidar device 600 of FIGS. 5 and 6 ) may be controlled so that the transmit photodiodes (eg, the first transmit cells 810 of FIG. 8 ) emit light.

For example, the lidar apparatus 600 may select some first transmission cells 810 (eg, a plurality of transmission cells 800) among all transmission cells 800 (eg, all transmission photodiodes) under the control of the processor 120. first transmit photodiodes) may be turned on. Under the control of the processor 120, the LiDAR device 600 is a second transmission cell excluding the first transmission cells 810 (eg, a plurality of first transmission photodiodes) (eg, the second transmission cell of FIG. 8 ). Transmission cells 820) may be turned off.

In operation 1130, the processor 120 may control the operation of the LIDAR device 600 so that all or some of the receiving cells are turned on. LiDAR device 600 may turn on all or some of the receiving cells based on the control of the processor 120 .

In operation 11400, the calculation unit (eg, the calculation unit 640 of FIG. 6) of the lidar device 600 may count arrival times of laser beams received from all reception cells (or some reception cells). Here, when the distance between the lidar device 600 and the objects is small, the count value of the arrival time is small, and the longer the distance between the lidar device 600 and the objects, the larger the count value of the arrival time. The calculation unit 640 may transfer the count value of the arrival times of all reception cells (or some reception cells) to the processor 120 .

In operation 1150, the processor 120 calculates an arrival time distribution of all reception cells (or some reception cells) (or all reception cells (or some reception cells)) based on the count value of arrival times of all reception cells (or some reception cells). cells) and distance values of objects.

In operation 1160, the processor 120 may determine whether the count value of the arrival time of each receiving cell is greater than or equal to a preset reference value.

As a result of determination in operation 1160, when the distribution of the count values of the arrival times of the plurality of receiving cells is less than a predetermined reference value (no), in operation 1170, the processor 120 determines the arrival of all the receiving cells (or some of the receiving cells). 3D mapping of the scanned space may be performed based on time distribution (or phase difference values of arrival times of all receiving cells (or some receiving cells)) and distance values (or depth values) of objects.

As a result of the determination in operation 1160, when the distribution of the count values of the arrival times of the plurality of receiving cells is greater than or equal to a preset reference value (yes), in operation 1180, the processor 120 determines an area where the density of objects is high (eg, a map). An area 710 having a high density of objects in 8 may be determined as an additional light emitting area.

In operation 1190, the processor 120 may control the operation of the LIDAR device 600 so that some transmission cells (eg, the third transmission cells 830 of FIG. 8) corresponding to the additional emission area emit light (on). there is. In addition, the processor 120 may control the operation of the LIDAR device 600 so that all receiving cells (or some receiving cells) are turned on. The LIDAR device 600 may emit light (on) of some transmission cells (eg, the third transmission cells 830 of FIG. 9 ) corresponding to the additional emission area under the control of the processor 120 . In addition, the lidar device 600 may operate (on) all receiving cells (or some receiving cells) based on the control of the processor 120 .

In operation 1200, the processor 120 calculates the arrival time distribution of all reception cells (or some reception cells) for the additional zero (or the phase difference value of the arrival time of all reception cells (or some reception cells)) and the distance value of objects ( Alternatively, 3D mapping of the scanned space may be performed based on the depth value).

12 is a diagram illustrating an operating method of an electronic device according to various embodiments of the present disclosure. In describing the operating method of the electronic device of FIG. 12 , description of the same (or similar) operation as that of the electronic device shown in FIG. 11 may be omitted.

Referring to FIG. 12 , operations 1110 to 1160 may be identical to (or similar to) operations of the electronic device shown in FIG. 11 .

In operation 1160, the processor 120 may determine whether a distribution of count values of arrival times of a plurality of receiving cells is greater than or equal to a preset reference value.

As a result of the determination in operation 1160, when the distribution of the count values of the arrival times of the plurality of receiving cells is greater than or equal to a preset reference value (yes), in operation 1180, the processor 120 determines an area where the density of objects is high (eg, a map). An area 710 having a high density of objects in 9 may be determined as an additional light emitting area.

In operation 1190, the processor 120 may control the operation of the LIDAR device 600 so that some transmission cells (eg, the third transmission cells 830 of FIG. 9) corresponding to the additional emission region emit light (on). there is. In addition, the processor 120 may control the operation of the LIDAR device 600 so that all receiving cells (or some receiving cells) are turned on. The LIDAR device 600 may emit light (on) of some transmission cells (eg, the third transmission cells 830 of FIG. 9 ) corresponding to the additional emission area under the control of the processor 120 . In addition, the lidar device 600 may operate (on) all receiving cells (or some receiving cells) based on the control of the processor 120 .

In operation 1170, the processor 120 calculates the arrival time distribution of all reception cells (or some reception cells) for the additional zero (or the phase difference value of the arrival time of all reception cells (or some reception cells)) and the distance value of objects ( Alternatively, 3D mapping of the scanned space may be performed based on the depth value).

In operation 1170, after performing 3D mapping of the scanned space, in operation 1210, the processor 120 determines whether the attitude and acceleration values of the human mounted device (HMD) (or augmented reality (AR) glasses) have been changed. can judge When the user wears the HMD (or AR glasses), the space scanned through the HMD may change according to the user's movement. The space to be scanned changes according to the movement of the user wearing the HMD, which can be detected by changes in posture and acceleration values.

As a result of the determination in operation 1210, when the posture and acceleration values of the HMD are not changed, the processor 120 may return to operation 1110 and perform subsequent operations.

As a result of the determination in operation 1210, when the posture and acceleration values of the HMD are changed, in operation 1220, the processor 120 may determine an additional light-emitting area where the direction of the HMD is changed.

In operation 1230, the processor (eg, the processor 120 of FIG. 1 ) in order to reduce power consumption of the LIDAR device 600, all transmission cells (eg, the existing light emitting region among all transmission cells 1000 of FIG. 10 ) Transmission cells (eg, transmission cells 1010 in Fig. 9) corresponding to (eg, the existing light emitting region 1001 in Fig. 9) may be turned off. Controls the operation of the lidar device 600 so that all or part of the transmission cells (eg, the

transmission cells

1020 and 1030 in FIG. 10 ) corresponding to the light-emitting region (eg, the additional light-emitting region 1002 in FIG. 10 ) emit light. For example, in the LIDAR device 600, based on the control of a processor (eg, the processor 120 of FIG. 1), the transmission cells 1010 corresponding to the existing light emitting region 1001 are turned off. ), and some of the transmission cells 1020 among the

transmission cells

1020 and 1030 corresponding to the additional light emitting region 1002 may emit light. The cells 1030 may be turned off. At the same time, all reception cells (eg, all reception diodes) or some reception cells corresponding to the additional foot and area 1002 may be turned on. .

In operation 1240, the processor 120 calculates the arrival time distribution of all reception cells (or some reception cells) for the additional zero (or the phase difference value of the arrival time of all reception cells (or some reception cells)) and the distance value of objects ( Alternatively, 3D mapping of the scanned space may be performed based on the depth value). In this way, according to the movement of the HMD (or AR glasses) (or the electronic device 300 of FIG. 3A or the electronic device 400 of FIG. 4A), some transmission cells 1020 corresponding to the additional light emitting area emit additional light, , the space and objects in the area to be newly scanned can be scanned in detail.

13 is a diagram illustrating an operating method of an electronic device according to various embodiments of the present disclosure. In describing the operating method of the electronic device of FIG. 13 , a detailed description of the same (or similar) operation as that of the electronic device shown in FIG. 11 may be omitted.

Referring to FIG. 13 , operations 1110 to 1160 may be identical to (or similar to) operations of the electronic device shown in FIG. 11 .

In operation 1170, after performing 3D mapping of the scanned space, in operation 1310, the processor 120 determines whether the attitude and acceleration values of the human mounted device (HMD) (or augmented reality (AR) glass) have been changed. can judge When the user wears the HMD (or AR glasses), the space scanned through the HMD may change according to the user's movement. The space to be scanned changes according to the movement of the user wearing the HMD, which can be detected by changes in posture and acceleration values.

As a result of the determination in operation 1310, if the attitude and acceleration values of the HMD are not changed, the processor 120 may return to operation 1110 and perform subsequent operations.

As a result of the determination in operation 1310, when the posture and acceleration value of the HMD are changed, in operation 1320, the processor 120 may determine an additional light-emitting area where the direction of the HMD is changed.

In operation 1330, the processor (eg, the processor 120 of FIG. 1) reduces the power consumption of the LIDAR device 600 by all transmission cells (eg, the existing light emitting region among all transmission cells 1000 of FIG. 10). Transmission cells (eg, transmission cells 1010 in Fig. 10) corresponding to (eg, the existing light emitting region 1001 in Fig. 10) may be turned off. Controls the operation of the lidar device 600 so that all or part of the transmission cells (eg, the

transmission cells

In operation 1340, the processor 120 calculates the arrival time distribution of all reception cells (or some reception cells) for the additional zero (or the phase difference value of the arrival times of all reception cells (or some reception cells)) and the distance value of objects ( Alternatively, 3D mapping of the scanned space may be performed based on the depth value). In this way, according to the movement of the HMD (or AR glass) (or the electronic device 300 in FIG. 3A or the electronic device 400 in FIG. 4A), some transmission cells 1020 corresponding to the additional light emitting area additionally emit light. Thus, it is possible to precisely scan the space and objects in the area to be newly scanned.

At least some of the operations shown in FIGS. 11 to 13 may be omitted. Before or after the at least some operations shown in FIGS. 11 to 13 , at least some operations mentioned with reference to other drawings in this document may be added and/or inserted. As an embodiment, the operations shown in FIGS. 11 to 13 are performed by a processor (eg, the processor 120 of FIG. 1 ) or a lidar device (eg, the lidar apparatus 600 of FIGS. 5 and 6 ). ) may be performed by a signal processing processor (eg, microprocessor) separately disposed in the As an embodiment, the operations shown in FIGS. 11 to 13 are separately disposed in a processor (eg, the processor 120 of FIG. 1 ) and a lidar device (eg, the lidar apparatus 600 of FIGS. 5 and 6 ). It may be distributed and performed in a signal processing processor (e.g., microprocessor). For example, a memory (eg, memory 130 of FIG. 1 ), when executed, a processor (eg, processor 120 of FIG. 1 ), and/or a signal processing processor as shown in FIGS. 11-13 It can store instructions to perform actions.

Electronic devices according to various embodiments of the present disclosure (eg, the electronic device 101 of FIG. 1 , the electronic device 300 of FIGS. 3A and 3B , the electronic device 400 of FIGS. 4A and 4B , and the electronic device of FIG. 5 The device 500 is a lidar device including a plurality of transmitting photodiodes 611 and a plurality of receiving diodes (eg, the lidar device 600 of FIGS. 5 and 6), the lidar device 600 ) may include a processor (eg, the processor 120 of FIG. 1 ) that controls the operation of the processor 120 and a memory operatively connected to the processor 120 (eg, the memory 130 of FIG. 1 ). When the memory 130 is executed, the processor 120 allows some of the first transmission photodiodes (eg, the first transmission cells 810 of FIG. 8 ) among the plurality of transmission photodiodes 611 to be stored. is on, and the second transmission photodiodes (eg, the second transmission cells 820 in FIG. 8) except for the first transmission photodiodes (eg, the first transmission cells 810 in FIG. 8) are turned off. (off) can be controlled. Depth values of objects in the scanned space may be calculated by operating the plurality of receiving diodes or a portion of the receiving diodes. Based on the depth values of the objects, a first area having a high density of the objects and a second area having a low density of the objects may be distinguished. The first area may be determined as an additional light emitting area, and some of the third transmit photodiodes 611 among the transmit photodiodes 611 corresponding to the first area may be controlled to be turned on. Instructions for controlling the operation of the plurality of receiving diodes or some of the receiving diodes to calculate depth values of objects within an additionally scanned area may be stored.

According to an embodiment, the processor 120 may control the LIDAR device 600 so that the plurality of transmit photodiodes 611 are individually turned on or off.

According to an embodiment, the processor 120 causes first receiving diodes corresponding to the first region to be turned on among the plurality of receiving diodes, and second receiving diodes corresponding to the second region to be turned on. It can be controlled to turn off.

According to one embodiment, the processor 120 may control the LIDAR device 600 so that the plurality of receiving diodes are individually turned on or off.

According to an embodiment, the lidar device 600 may include an arithmetic unit (eg, the arithmetic unit 640 of FIG. 6 ). The plurality of receiving diodes measure the received laser beams reflected from the object. A value may be transferred to the calculation unit (eg, the calculation unit 640 of FIG. 6 ). The calculating unit (eg, the calculating unit 640 of FIG. 6 ) may measure arrival times of the laser beams and transmit the measured arrival times to the processor 120 . The processor 120 may calculate a depth value of the object based on the measurement value of the arrival time.

According to an embodiment, the processor 120 may determine the density of the objects by checking the arrival time distribution or the phase difference distribution for all of the plurality of receiving diodes.

According to an embodiment, the processor 120 may compare an arrival time distribution or a phase difference distribution of the plurality of receiving diodes with a preset reference value. When the arrival time distribution or the phase difference distribution of the receiving cell is less than a predetermined reference value, 3D mapping of the scanned space may be performed.

According to an embodiment, the processor 120 may compare arrival times or phase difference distributions of the plurality of receiving diodes with a preset reference value. When the arrival time distribution or phase difference distribution of the plurality of receiving diodes is equal to or greater than a preset reference value, a region having an arrival time distribution or phase difference distribution of the plurality of receiving diodes equal to or greater than the reference value may be determined as the additional light emitting region.

According to an embodiment, the processor 120 may receive the attitude and acceleration values of a human mounted device (HMD) (eg, the electronic device 500 of FIG. 5 ). Based on the attitude and acceleration values, a change in attitude and position of the HMD (eg, the electronic device 500 of FIG. 5 ) may be determined. When the attitude and position of the HMD (eg, the electronic device 500 of FIG. 5 ) is changed, a new additional light-emitting area may be determined. Some of the transmission photodiodes 611 among the transmission photodiodes 611 corresponding to the new additional light emitting region may be controlled to be turned on.

According to an embodiment, the processor 120 operates the plurality of receiving diodes or some receiving diodes corresponding to the new additional light emitting area to calculate depth values of objects scanned in the new additional light emitting area. can

Methods of operating

electronic devices

101, 300, 400, and 500 according to various embodiments of the present disclosure include first transmission photodiodes among a plurality of transmission photodiodes 611 included in a lidar device 600 (Example: The first transmission cells 810 of FIG. 8 are turned on, and the second transmission photodiodes excluding the first transmission photodiodes (eg, the first transmission cells 810 of FIG. 8) ( Example: It is possible to control the second transmission cells 820 of FIG. 8 to be turned off. Depth values of objects in the scanned space may be calculated by operating a plurality of receiving diodes or some receiving diodes included in the lidar device 600 . Depending on the depth values of the objects, a first area with a high density of objects and a second area with a low density of the objects may be distinguished. The first area may be determined as an additional light emitting area, and some of the third transmit photodiodes 611 among the transmit photodiodes 611 corresponding to the first area may be controlled to be turned on. Depth values of objects in an additionally scanned area may be calculated by operating the plurality of receiving diodes or some of the receiving diodes.

According to an embodiment, the plurality of transmission photodiodes 611 may be individually controlled to be turned on or off.

According to an embodiment, the first receiving diodes corresponding to the first area among the plurality of receiving diodes are turned on, and the second receiving diodes corresponding to the second area are controlled to be turned off. can do.

According to an embodiment, the plurality of receiving diodes may be individually controlled to be turned on or off.

According to an embodiment, the plurality of receiving diodes may transfer measurement values according to laser beams reflected from the object and received to the calculating unit of the lidar device 600 (eg, calculating unit 640 of FIG. 6 ). . The calculating unit (eg, the calculating unit 640 of FIG. 6 ) may measure arrival times of the laser beams and transmit the measured arrival times to the processor 120 . The processor 120 may calculate a depth value of the object based on the measurement value of the arrival time.

According to an embodiment, the density of the objects may be determined by checking an arrival time distribution or a phase difference distribution for all of the plurality of receiving diodes.

According to an embodiment, the arrival time distribution or the phase difference distribution of the plurality of receiving diodes may be compared with a preset reference value.

According to an embodiment, when the arrival time distribution or the phase difference distribution of the plurality of receiving diodes is less than the set reference value, 3D mapping of the scanned space may be performed.

According to an embodiment, the arrival time distribution or the phase difference distribution of the plurality of receiving diodes may be compared with a preset reference value. When the arrival time distribution or phase difference distribution of the plurality of receiving diodes is greater than or equal to the set reference value, a region having an arrival time distribution or phase difference distribution greater than or equal to the set reference value may be determined as the additional light-emitting area.

According to an embodiment, the processor 120 may receive the attitude and acceleration values of a human mounted device (HMD) (eg, the electronic device 500 of FIG. 5 ). Based on the attitude and acceleration values, a change in attitude and position of the HMD (eg, the electronic device 500 of FIG. 5 ) may be determined. When the attitude and position of the HMD (eg, the electronic device 500 of FIG. 5 ) is changed, a new additional light emitting area may be determined. Some of the transmission photodiodes 611 among the transmission photodiodes 611 corresponding to the new additional light emitting region may be controlled to be turned on.

According to an embodiment, depth values of objects scanned in the new additional light emitting area may be calculated by operating the plurality of receiving diodes or some receiving diodes corresponding to the new additional light emitting area.

Although the disclosure has been shown and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the following disclosure. defined by the appended claims and their equivalents.

Claims

In electronic devices,

A lidar device including a plurality of transmitting photodiodes and a plurality of receiving diodes;

A processor for controlling the operation of the LIDAR device; and

a memory operatively coupled with the processor;

The memory, when executed, the processor,

controlling some first transmission photodiodes among the plurality of transmission photodiodes to be turned on and second transmission photodiodes excluding the first transmission photodiodes to be turned off;

operating the plurality of receiving diodes or some receiving diodes to calculate depth values of objects in the scanned space;

Distinguishing a first area with a high density of the objects and a second area with a low density based on the depth values of the objects;

determining the first region as an additional light emitting region, and controlling some third transmission photodiodes among transmission photodiodes corresponding to the first region to be turned on;

Storing instructions for controlling to operate the plurality of receiving diodes or some receiving diodes to calculate depth values of objects in an additionally scanned area,

electronic device.
According to claim 1,

Controlling the lidar device so that the plurality of transmission photodiodes are individually turned on or off.

electronic device.
According to claim 1,

Among the plurality of receiving diodes, first receiving diodes corresponding to the first region are turned on, and second receiving diodes corresponding to the second region are controlled to be turned off.

electronic device.
According to claim 3,

Controlling the lidar device so that the plurality of receiving diodes are individually turned on or off,

electronic device.
According to claim 3,

The lidar device includes a calculation unit,

The plurality of receiving diodes transfer measurement values according to laser beams reflected from the object and received to the calculation unit;

The calculation unit measures the arrival time of the laser beams and transmits the measured value of the arrival time to the processor;

The processor calculates a depth value of the object based on the measured value of the arrival time.

electronic device.
According to claim 5,

The processor determines the density of the objects by checking the arrival time distribution or the phase difference distribution for all of the plurality of receiving diodes.

electronic device.
According to claim 6,

The processor compares an arrival time distribution or a phase difference distribution of the plurality of receiving diodes with a preset reference value,

Performing 3D mapping of the scanned space when the arrival time distribution or phase difference distribution of the receiving cell is less than a preset reference value,

electronic device.
According to claim 6,

The processor compares the arrival time or phase difference distribution of the plurality of receiving diodes with a preset reference value,

When the arrival time distribution or phase difference distribution of the plurality of receiving diodes is equal to or greater than a preset reference value, determining a region having an arrival time distribution or phase difference distribution of the plurality of receiving diodes equal to or greater than the set reference value as the additional light emitting region.

electronic device.
According to claim 3,

The processor receives attitude and acceleration values of a human mounted device (HMD),

Based on the posture and acceleration values, determining a change in posture and position of the HMD;

determining a new additional light-emitting area when the posture and position of the HMD are changed;

Controlling some of the transmission photodiodes among the transmission photodiodes corresponding to the new additional light emitting region to be turned on.

electronic device.
According to claim 9,

calculating depth values of objects scanned in the new additional light emitting area by operating the plurality of receiving diodes or some receiving diodes corresponding to the new additional light emitting area;

electronic device.
In the operating method of the electronic device,

Among the plurality of transmission photodiodes included in the lidar device, some of the first transmission photodiodes are turned on, and the second transmission photodiodes other than the first transmission photodiodes are controlled to be turned off,

Operating a plurality of receiving diodes or some receiving diodes included in the lidar device to calculate depth values of objects in the scanned space,

According to the depth values of the objects, a first area with a high density of objects and a second area with a low density are distinguished;

determining the first region as an additional light emitting region, and controlling some third transmission photodiodes among transmission photodiodes corresponding to the first region to be turned on;

calculating depth values of objects in an additionally scanned area by operating the plurality of receiving diodes or some receiving diodes;

Methods of operating electronic devices.
According to claim 11,

Controlling the plurality of transmission photodiodes to be individually turned on or off,

Methods of operating electronic devices.
According to claim 11,

Among the plurality of receiving diodes, first receiving diodes corresponding to the first region are turned on, and second receiving diodes corresponding to the second region are controlled to be turned off.

Methods of operating electronic devices.
According to claim 13,

Controlling the plurality of receiving diodes to be individually turned on or off,

Methods of operating electronic devices.
According to claim 13,

The plurality of receiving diodes transfer measurement values according to laser beams reflected from the object and received to the calculation unit of the LIDAR device,

The calculation unit measures the arrival time of the laser beams and transmits the measured value of the arrival time to the processor;

The processor calculates a depth value of the object based on the measured value of the arrival time.

Methods of operating electronic devices.