WO2022025630A1

WO2022025630A1 - Electronic device including distance sensor, and autofocusing method

Info

Publication number: WO2022025630A1
Application number: PCT/KR2021/009817
Authority: WO
Inventors: 이존기; 박경완
Original assignee: 삼성전자 주식회사
Priority date: 2020-07-31
Filing date: 2021-07-28
Publication date: 2022-02-03
Also published as: KR20220015752A

Abstract

An electronic device according to one embodiment disclosed in the present document comprises: a camera module including a lens unit and an image sensor; a distance sensor; a processor; and a memory, wherein the processor can acquire first calibration data related to the characteristics related to the distance measurement of the distance sensor at a first distance and store same in the memory, acquire second calibration data related to crosstalk at a second distance and store same in the memory, execute the camera module, calculate an object distance from an external object on the basis of the first calibration data or the second calibration data, and determine, on the basis of the object distance, the position of the lens unit for autofocusing of the camera module.

Description

Electronic devices including distance sensors and methods of performing autofocus

Various embodiments disclosed in this document relate to an electronic device including a distance sensor and a method for performing autofocus using the distance sensor.

An electronic device such as a smartphone or a tablet PC may capture an image using a camera. The electronic device may perform auto focus (hereinafter, AF) on an external object. The electronic device may calculate a distance to the object using a distance sensor (eg, a time of flight (TOF) sensor) and perform AF using the calculated distance.

The distance sensor may include a light emitting unit and a light receiving unit. The light emitting unit may output light of an IR (infrared ray) pulse, and the light receiving unit may obtain light from which the light of an IR (infrared ray) pulse is reflected by an object. The electronic device may calculate the distance to the object by measuring the time for the light output from the light emitting unit to be reflected by the object and flow into the light receiving unit.

The electronic device may quickly determine a position of a lens for auto focus (AF) based on the calculated distance. The position of the lens for AF according to the distance to the object may be determined and stored.

The electronic device may perform calibration on the distance sensor during a manufacturing process or in a setting after manufacturing to increase accuracy of distance measurement with an object. In order to increase the accuracy of the distance measurement, the electronic device may simultaneously perform the offset calibration and the crosstalk calibration at a distance that can escape the influence of the crosstalk, for example, at a distance of 30 cm or more from the distance sensor. In this case, at a short distance (eg, less than 30 cm), an error in distance measurement with an object may be large, and AF accuracy may be reduced.

Various embodiments of the present disclosure provide an electronic device that increases accuracy of distance measurement with an object by performing offset calibration and crosstalk calibration of a distance sensor at different distances.

An electronic device according to various embodiments of the present disclosure includes a camera module including a lens unit and an image sensor, a distance sensor including a light emitting unit and a light receiving unit, a processor, and a memory, wherein the processor performs the distance at a first distance shorter than a reference distance. Obtaining first calibration data related to the characteristic related to the distance measurement of the sensor and storing it in the memory, at a second distance greater than or equal to the reference distance, obtaining and storing second calibration data related to crosstalk in the memory, Executes the camera module, calculates an object distance with an external object based on the first calibration data or the second calibration data, and performs the auto focus of the camera module based on the object distance The position of the lens unit may be determined.

The electronic device according to various embodiments disclosed herein performs offset calibration (correcting a difference between an actual distance and a measured distance) and crosstalk calibration (correcting an error due to crosstalk) of a distance sensor at different distances Thus, the accuracy of distance measurement with an object can be improved. Through this, the accuracy of the short-range AF may be increased.

The electronic device according to various embodiments of the present disclosure may output a user interface to allow a user to remove an obstacle when the crosstalk is at a level that affects object distance measurement.

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;

2 illustrates an electronic device according to various embodiments.

3 illustrates a distance sensor according to various embodiments.

4 illustrates a calibration operation of a distance sensor according to various embodiments.

5 is a flowchart illustrating preferential storage of first calibration data according to various embodiments of the present disclosure;

6 is a flowchart illustrating a method of performing auto-focusing according to various embodiments of the present disclosure;

7 illustrates an error rate according to a distance according to various embodiments.

8 illustrates a UI display according to an error in object distance measurement according to various embodiments of the present disclosure.

In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. However, it is not intended to limit the technology described in this document to specific embodiments, and it should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of this document. . In connection with the description of the drawings, like reference numerals may be used for like components.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments of the present disclosure. The electronic device according to various embodiments disclosed in this document may have various types of devices. Electronic devices include, for example, portable communication devices (eg, smartphones), computer devices (eg, personal digital assistants), tablet PCs (tablet PCs), laptop PCs (desktop PCs, workstations, or servers); It may include at least one of a portable multimedia device (eg, an e-book reader or an MP3 player), a portable medical device (eg, a heart rate, blood sugar, blood pressure, or body temperature monitor), a camera, or a wearable device. (e.g., watches, rings, bracelets, anklets, necklaces, eyeglasses, contact lenses, or head-mounted-devices (HMDs)); : skin pad or tattoo), or a bioimplantable circuit.In some embodiments, the electronic device is, for example, a television, a digital video disk (DVD) player, an audio device, an audio accessory. Devices (such as speakers, headphones, or headsets), refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air purifiers, set-top boxes, home automation control panels, security control panels, game consoles, electronic dictionaries, electronic keys, It may include at least one of a camcorder and an electronic picture frame.

In another embodiment, the electronic device is a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR) (eg, a black box for a vehicle/vessel/airplane), an automotive infotainment device. (e.g. head-up displays for vehicles), industrial or home robots, drones, automated teller machines (ATMs), point of sales (POS) instruments, metering instruments (e.g. water, electricity, or gas metering instruments); Alternatively, it may include at least one of an IoT device (eg, a light bulb, a sprinkler device, a fire alarm, a thermostat, or a street lamp). The electronic device according to the embodiment of this document is not limited to the above-described devices, and, for example, as in the case of a smartphone equipped with a function of measuring personal biometric information (eg, heart rate or blood sugar), a plurality of electronic devices The functions of the devices may be provided in a complex manner. In this document, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

In the network environment 100 , the electronic device 101 communicates with the electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 (eg, a long-distance wireless communication network). network) and communicate with the electronic device 104 or the server 108 . According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input device 150 , a sound output device 155 , a display device 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminals 178 ( 1 ( 1 )) may be omitted, or one or more other components may be added to the electronic device 101 . In , some of these components (eg, the sensor module 176 , the camera module 180 , or the antenna module 197 ) may be integrated into one component (eg, the display device 160 ). .

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor), or a secondary processor 123 (eg, a graphic processing unit or an image signal processor) that can operate independently or together with the main processor 121 . , a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 may use less power than the main processor 121 or may be set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the co-processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input device 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output a sound signal to the outside of the electronic device 101 . The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to an embodiment, the display device 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input device 150 , or an external electronic device (eg, a sound output device 155 ) connected directly or wirelessly with the electronic device 101 . A sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

2 illustrates an electronic device according to various embodiments.

Referring to FIG. 2 , an electronic device 201 (eg, the electronic device 101 of FIG. 1 ) includes a processor 210 , a memory 220 , a display 230 , a distance sensor 240 and/or a camera module ( 250) may be included. FIG. 2 is not limited thereto, focusing on a configuration related to measuring a distance from an object or capturing an image.

The processor 210 (eg, the processor 120 of FIG. 1 ) may perform various operations necessary for the operation of the electronic device 201 . For example, the processor 210 may calculate the distance to the object (hereinafter, object distance) by driving the distance sensor 240 . As another example, the processor 210 may drive the camera module 250 to perform AF on an object and capture an image.

The memory 220 (eg, the memory 120 of FIG. 1 ) may store various information used for driving the electronic device 201 . For example, the memory 220 may store calibration information of the distance sensor 240 , reference data for controlling the distance sensor 240 according to noise, or data regarding the power of the distance sensor 240 . For another example, the memory 220 may include information about the position of the lens unit of the camera module 250 according to the object distance for performing AF or information related to photographing an image (eg, data for recognizing a light source, intensity of a light source) and data related to color temperature or data for detecting noise of a light source).

Display 230 may display content such as images, icons, user interfaces, or text. For example, the display 230 may display an image based on image data acquired through the camera module 250 . As another example, the display 230 may display a UI indicating a focused object according to the AF function.

The distance sensor 240 may be used to calculate an object distance to an external object. The distance sensor 240 may output light (eg, IR) (hereinafter, transmitted light) of a specified wavelength and collect light reflected from an external object (hereinafter, received light). The control circuit of the processor 210 or the distance sensor 240 may calculate the object distance based on a time from an output time of the transmitted light to an arrival time of the received light. Additional information regarding the distance sensor 240 may be provided through FIG. 3 .

The camera module (or the camera device, the imaging device, or the imaging device) 250 (eg, the camera module 180 of FIG. 1 ) may acquire image data (eg, RGB data). The camera module 250 may include a lens unit, an image sensor, or an image processing unit. According to an embodiment, the camera module 250 may perform AF by moving the lens unit to a position corresponding to the object distance.

3 illustrates a distance sensor according to various embodiments.

Referring to FIG. 3 , the distance sensor 240 may include a light emitting unit 310 , a light receiving unit 320 , an auxiliary light receiving unit 325 , or a surface unit 330 .

Although not shown in FIG. 3 , the distance sensor 240 may further include a control circuit for controlling the light emitting unit 310 , the light receiving unit 320 , or the auxiliary light receiving unit 325 or processing data. The processor 210 or the control circuit of the distance sensor 240 calculates the distance (object distance) between the distance sensor 240 and the object 350 based on the time from the output time of the light to the arrival time of the reflected light. can

According to various embodiments, the light emitting unit 310 may output transmission light (eg, IR) of a specified wavelength. The light receiving unit 320 may detect the received light in which the transmitted light is reflected by the object 350 . The auxiliary light receiving unit 325 may be disposed in an area adjacent to the light emitting unit 310 . The auxiliary light receiving unit 325 may be used to detect the effect of crosstalk. The surface part 330 may be disposed on the front surface of the light emitting part 310 and the light receiving part 320 . The surface portion 330 may include glass or a polymer. For example, at least a portion of the surface portion 330 may be treated using a paint or film so that the light emitting unit 310 or the light receiving unit 320 is not visible. As another example, the surface part 330 may be included in a housing included in an electronic device (eg, the electronic device 201 of FIG. 2 ).

According to various embodiments, crosstalk may occur during operation of the distance sensor 240 . The crosstalk may be light flowing from the light emitting unit 310 to the light receiving unit 320 through a separate path other than the path reflected from the object 350 . For example, the structure to which the light emitting unit 310 and the light receiving unit 320 are fixed, the space between the light emitting unit 310 (or the light receiving unit 320 ) and the surface unit 330 , and the inner space of the surface unit 330 . Through this, crosstalk may occur (waveguide effect). When a paint or film that blocks visible light is applied to the light emitting unit 310 or the light receiving unit 320 , crosstalk may further increase.

The crosstalk may be an obstacle to recognizing the object distance with the object 350 , and may be excluded when calculating the object distance through pre-stored calibration data related to the crosstalk.

The graph 360 represents the number of photons flowing into the light receiving unit 320 over time. The light receiving unit 320 may obtain data on the first photons 361 arriving at the center of the first time t1 and the second photons 362 arriving at the center at the second time t2. have. The data for the first photons 361 may be generated by crosstalk, and the data for the second photons 362 may be generated by the received light Rx reflected through the object 350 . Data by the first photons 361 by crosstalk may be removed through calibration data.

When the distance at which the crosstalk calibration is performed is relatively close from the distance sensor 240 , the interval between the first time t1 and the second time t2 may be small, and therefore, the first photons due to the crosstalk It may be difficult to distinguish the 361 from the second photons 362 reflected by the object 350 . For this reason, it may be difficult to effectively remove the crosstalk.

According to various embodiments, the processor 210 or the control circuit of the distance sensor 240 may store calibration data by performing crosstalk calibration at a second distance greater than or equal to a specified reference distance (eg, about 30 cm).

Referring to FIG. 4 , in operation 410 , the processor 210 may start calibration of the distance sensor 240 . For example, in at least one of a manufacturing process of the distance sensor 240 , a setting process of the distance sensor 240 , and an application execution process related to the distance sensor 240 , the processor 210 performs calibration of the distance sensor 240 . can

In operation 420 , the processor 210 may determine whether calibration is performed at a distance within a reference distance (eg, 30 cm). For example, the reference distance may be set differently according to the level of occurrence of crosstalk.

In operation 430, if the calibration is within the reference distance, the processor 210 may perform the offset calibration at a first distance (eg, 10 cm) that is smaller than the set reference distance (eg, about 30 cm). Offset calibration may be an operation or process of correcting a distance measurement value according to manufacturing characteristics of the distance sensor 240 . For example, when the actual distance to the object is 100 cm and the object distance measured using the distance sensor 240 is 98 cm, the processor 210 may store the offset by offset calibration as 2 cm.

In operation 435 , the processor 210 may store first calibration data according to the offset calibration performed at the first distance in the memory 220 .

In operation 440 , if the calibration is equal to or greater than the reference distance, the processor 210 may perform crosstalk calibration at a second distance greater than or equal to the reference distance (eg, about 30 cm). When the distance at which the crosstalk calibration is performed is relatively close to the distance sensor 240 , it may be difficult to distinguish between a photon by the crosstalk and a photon reflected by the object. The processor 210 may perform crosstalk calibration at a second distance greater than or equal to a specified reference distance (eg, about 30 cm).

In operation 445 , the processor 210 may store second calibration data according to the crosstalk calibration performed at the second distance in the memory 220 .

Referring to FIG. 5 , in operation 510 , first calibration data by offset calibration may be stored in the memory 240 or a storage element included in the distance sensor 240 . For example, the first calibration data may be transmitted from a manufacturer of the distance sensor 240 to a manufacturer of the electronic device 201 . The first calibration data may be stored in the memory 240 during the manufacturing process of the electronic device 201 . The first calibration data may be stored by being performed at a first distance (10 cm) that is smaller than the reference distance (eg, 30 cm).

As another example, the first calibration data may be stored in a storage element included in the distance sensor 240 at the manufacturing time of the distance sensor 240 . The distance sensor 240 may be mounted on the electronic device 201 to manufacture the electronic device 201 . The electronic device 201 may load and use the first calibration data from a storage element included in the distance sensor 240 .

In operation 520 , the processor 210 may perform crosstalk calibration at a second distance greater than or equal to a reference distance (eg, about 30 cm). When the distance at which the crosstalk calibration is performed is relatively close to the distance sensor 240 , it may be difficult to distinguish between a photon by the crosstalk and a photon reflected by the object. The processor 210 may perform crosstalk calibration at a second distance greater than or equal to a specified reference distance (eg, about 30 cm).

In operation 530 , the processor 210 may store second calibration data according to the crosstalk calibration performed at the second distance in the memory 220 .

6 is a flowchart illustrating a method of performing auto focus (AF) according to various embodiments of the present disclosure;

Referring to FIG. 6 , in operation 610 , the processor 210 may execute a camera application using the camera module 180 . After the camera application is executed, the processor 210 may display a preview image on the display 230 based on the image data acquired through the camera module 180 .

In operation 620 , the processor 210 may primarily calculate a distance to an external object (hereinafter, a measurement distance) using the distance sensor 240 . The processor 210 outputs the transmitted light of the IR pulse through the light emitting unit of the distance sensor 240 (eg, the light emitting unit 310 of FIG. 3 ), and the light receiving unit of the distance sensor 240 (eg, the light receiving unit of FIG. 3 ) (320)), it is possible to collect data about the received light reflected by the object of the IR pulse. The processor 210 may calculate the measurement distance based on the speed of the IR pulse and the movement time (the output time of the transmitted light to the reception time of the received light).

According to various embodiments, the processor 210 may calculate the object distance by correcting the measurement distance based on the first calibration data or the second calibration data. The first calibration data may be correction data related to a distance offset according to a manufacturing characteristic of the distance sensor 240 . The second calibration data may be correction data related to crosstalk.

According to an embodiment, when the measurement distance is smaller than the reference distance (eg, 30 cm), the processor 210 may correct the measurement distance using the first calibration data and may not use the second calibration data.

According to another embodiment, when the measurement distance is greater than or equal to the reference distance (eg, 30 cm), the processor 210 corrects the measurement distance using the second calibration data, and does not use the first calibration data. can

According to an embodiment, before using the distance sensor 240 , the processor 210 first calculates the object distance using a phase detector, and the calculated distance to the object is within a specified range. In this case, the object distance may be secondarily calculated using the distance sensor 240 .

In operation 630 , the processor 210 may determine whether the object distance calculated using the distance sensor 240 is equal to or greater than (or greater than) a specified reliability level. The reliability may be stored in advance based on the shooting environment, the performance of the distance sensor 240, and data obtained through other sensors.

According to various embodiments, when the calculated object distance is less than (or less than) the specified reliability level, the processor 210 may not use the object distance calculated using the distance sensor 240 for the AF operation (operation 630 ). of NO).

In operation 640 , when the object distance calculated by using the distance sensor 240 is equal to or greater than the specified reliability level, the processor 210 may move the lens unit of the camera module 250 using the object distance.

According to an embodiment, the memory 220 may store a table in which the object distance and the position of the lens unit are matched. The processor 210 may determine a position of the lens unit corresponding to the calculated object distance with reference to the table.

According to various embodiments, the processor 210 may display the AF UI on the display 230 . According to an embodiment, the processor 210 may fine-tune the AF UI based on state information (movement, rotation) of the electronic device.

7 illustrates an error rate according to a distance according to various embodiments. 7 is illustrative and not limited thereto.

Referring to FIG. 7 , a first graph 701 shows an error rate for each distance of the distance sensor 240 that has performed both offset calibration and crosstalk calibration within a reference distance Ls (eg, 30 cm). In this case, the error rate of the object distance calculated from a distance within the reference distance (Ls) is relatively large (eg, about 20%), while the error rate of the object distance calculated from a distance greater than the reference distance (Ls) is relatively small. possible (eg about 5%).

The second graph 702 performs offset calibration at a first distance (eg, about 10 cm) within a reference distance (Ls) (eg, 30 cm), and a second graph equal to or greater than the reference distance (Ls) (eg, 30 cm). An error rate for each distance of the distance sensor 240 that has performed crosstalk calibration at distance is shown. According to an embodiment, when the measured distance calculated through the distance sensor 240 is less than (or less than) the reference distance Ls (eg, 30 cm), the processor 210 is configured to perform first calibration data by offset calibration can be used to correct the measurement distance. When the measurement distance calculated by the distance sensor 240 is equal to or greater than (or greater than) the reference distance Ls (eg, 30 cm), the processor 210 determines the measurement distance using the second calibration data by crosstalk calibration. can be corrected. In this case, both the error rate of the object distance calculated from a distance within the reference distance Ls and the error rate of the object distance calculated from a distance greater than the reference distance Ls may be relatively small (eg, about 5%).

Referring to FIG. 8 , the processor 210 may determine whether the crosstalk deviates from a reference value and an error in distance measurement occurs.

For example, when water droplets or contaminants are deposited on the surface portion of the distance sensor 240 (eg, the surface portion 330 of FIG. 3 ), the crosstalk may deviate from the reference value.

According to an embodiment, the reference value may be determined based on the number of photons detected by the distance sensor 240 . For example, the reference value may be one of about 100 kcps (kilo counts per second) to about 120 kcps.

When the crosstalk deviates from the reference value, the processor 210 may display the UI 810 on the display 230 . For example, the UI 810 may be a pop-up window that allows the user to remove foreign substances from the surface of the distance sensor 240 .

An electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ) according to various embodiments of the present disclosure includes a camera module (eg, the camera module 180 of FIG. 1 ) including a lens unit and an image sensor. , the camera module 250 of FIG. 2 ), a distance sensor including a light emitting unit and a light receiving unit (eg, the sensor module 176 of FIG. 1 , the distance sensor 240 of FIG. 2 ), a processor (eg, the processor of FIG. 1 ) 120, including the processor 210 of FIG. 2) and a memory (eg, the memory 130 of FIG. 1, the memory 220 of FIG. 2), the processor (eg, the processor 120 of FIG. 1); The processor 210 of FIG. 2) obtains first calibration data related to the offset at a first distance shorter than the reference distance, and stores it in the memory (eg, the memory 130 of FIG. 1 and the memory 220 of FIG. 2) and obtains second calibration data related to crosstalk at a second distance greater than or equal to the reference distance and stores it in the memory (eg, the memory 130 of FIG. 1 and the memory 220 of FIG. 2 ), and the camera A module (eg, the camera module 180 of FIG. 1 , the camera module 250 of FIG. 2 ) is executed, and an object distance with an external object is calculated based on the first calibration data or the second calibration data, The position of the lens unit for auto focus of the camera module (eg, the camera module 180 of FIG. 1 and the camera module 250 of FIG. 2 ) may be determined based on the object distance.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 , the processor 210 of FIG. 2 ) may include the distance sensor (eg, the sensor module 176 of FIG. 1 , the distance sensor 240 of FIG. 2 ) )), the distance calculated using the first calibration data or the second calibration data may be corrected to calculate the object distance.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 , the processor 210 of FIG. 2 ) corrects the distance based on the first calibration data when the distance is within the reference distance can do.

According to various embodiments, the processor (eg, the processor 120 of FIG. 1 , the processor 210 of FIG. 2 ) may correct the distance based on the second calibration data when the distance is greater than or equal to the reference distance. can

According to various embodiments, the reference distance may be determined based on a crosstalk characteristic of the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ).

According to various embodiments, the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ) may include a display (eg, the display device 160 of FIG. 1 , the display 230 of FIG. 2 ). )), wherein the processor (eg, the processor 120 of FIG. 1 , the processor 210 of FIG. 2 ) includes the display (eg, the display device 160 of FIG. 1 , the display 230 of FIG. 2 ) ), a user interface corresponding to the auto focus may be displayed.

According to various embodiments, the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ) may include a display (eg, the display device 160 of FIG. 1 , the display 230 of FIG. 2 ). )), wherein the processor (eg, the processor 120 of FIG. 1 , the processor 210 of FIG. 2 ) includes the distance sensor (eg, the sensor module 176 of FIG. 1 , the distance sensor of FIG. 2 ) 240)))) to remove foreign substances from the surface of the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ) when the data by the crosstalk exceeds the reference value. A user interface may be displayed on the display (eg, the display device 160 of FIG. 1 and the display 230 of FIG. 2 ).

According to various embodiments, the reference distance is 30 cm from the surface of the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ), and the first distance is the distance sensor (eg, FIG. It may be 10 cm from the surface of the sensor module 176 of 1, the distance sensor 240 of FIG. 2).

According to various embodiments, the first calibration data may be stored in a storage element included in the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ).

According to various embodiments, the first calibration data and the second calibration data are calculated in a manufacturing process of the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ), and the memory (eg, the memory 130 of FIG. 1 , the memory 220 of FIG. 2 ).

According to various embodiments, the auto focus execution method is performed in an electronic device (eg, the electronic device 101 of FIG. 1 , and the electronic device 201 of FIG. 2 ), and the electronic device (eg, FIG. 1 ) Using a distance sensor (eg, the sensor module 176 of FIG. 1 , and the distance sensor 240 of FIG. 2 ) of the electronic device 101 of FIG. 2 ), the first distance shorter than the reference distance An operation of acquiring and storing first calibration data related to an offset in a distance, using the distance sensor (eg, the sensor module 176 of FIG. 1 , and the distance sensor 240 of FIG. 2 ), greater than or equal to the reference distance An operation of acquiring and storing second calibration data related to crosstalk at a second distance, a camera module (eg, FIG. 1 ) of the electronic device (eg, the electronic device 101 of FIG. 1 , and the electronic device 201 of FIG. 2 ) An operation of executing the camera module 180 of FIG. 1 and the camera module 250 of FIG. 2 ), an operation of calculating an object distance with an external object based on the first calibration data or the second calibration data, and the object The camera module (eg, the camera module 180 of FIG. 1 ) for auto focus of the camera module (eg, the camera module 180 of FIG. 1 , the camera module 250 of FIG. 2 ) based on the distance ), and determining the position of the lens unit of the camera module 250 of FIG. 2 .

According to various embodiments, the determining of the position of the lens unit may include performing the first calibration of the distance calculated using the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ). and calculating the object distance by correcting it using data or the second calibration data.

According to various embodiments, the operation of determining the position of the lens unit may further include correcting the distance based on the first calibration data when the distance is within the reference distance.

According to various embodiments, the determining of the position of the lens unit may further include correcting the distance based on the second calibration data when the distance is equal to or greater than the reference distance.

According to various embodiments, the method includes a display (eg, the display device 160 of FIG. 1 , the electronic device 201 of FIG. 2 ) of the electronic device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ). The method may further include displaying a user interface corresponding to the auto focus on the display 230 .

According to various embodiments, the method may be performed when data by crosstalk measured by the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ) exceeds a reference value. A user interface for removing foreign substances from the surface of the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ) is provided with the electronic device (eg, the electronic device 101 of FIG. 1 , FIG. The method may further include displaying on a display (eg, the display device 160 of FIG. 1 , and the display 230 of FIG. 2 ) of the electronic device 201 of FIG. 2 .

According to various embodiments, the operation of acquiring and storing the 1 calibration data is the 1 stored in a storage element included in the distance sensor (eg, the sensor module 176 of FIG. 1 and the distance sensor 240 of FIG. 2 ). It may include an operation of loading calibration data.

According to various embodiments, the first calibration data and the second calibration data are calculated in a manufacturing step of the electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 201 of FIG. 2 ), and the electronic device It may be stored in a memory (eg, the memory 130 of FIG. 1 , the memory 220 of FIG. 2 ) of the device (eg, the electronic device 101 of FIG. 1 , the electronic device 201 of FIG. 2 ).

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item unless the context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the element from other elements in question, and may refer to elements in other aspects (e.g., importance or order) is not limited. that one (eg first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively” When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

As used herein, the term “module” may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 801 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed online (eg download or upload), directly between smartphones (eg smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

Claims

In an electronic device,

a camera module including a lens unit and an image sensor;

a distance sensor including a light emitting unit and a light receiving unit;

processor; and

memory; including;

the processor is

At a first distance shorter than the reference distance, first calibration data related to the distance measurement characteristic of the distance sensor is acquired and stored in the memory;

At a second distance greater than or equal to the reference distance, obtain second calibration data related to crosstalk and store it in the memory;

Execute the camera module,

calculating an object distance with an external object based on the first calibration data or the second calibration data,

An electronic device for determining a position of the lens unit for auto focus of the camera module based on the object distance.
The method of claim 1, wherein the processor is

calculating the distance to the external object using the distance sensor,

The electronic device calculates the object distance by correcting the calculated distance using the first calibration data or the second calibration data.
3. The method of claim 2, wherein the processor

When the calculated distance is within the reference distance, the electronic device calculates the object distance by correcting the distance based on the first calibration data.
3. The method of claim 2, wherein the processor

When the calculated distance is equal to or greater than the reference distance, the electronic device calculates the object distance by correcting the distance based on the second calibration data.
The method of claim 1, wherein the reference distance

An electronic device determined based on a crosstalk characteristic of the distance sensor.
According to claim 1,

further comprising a display;

the processor is

An electronic device that displays a user interface corresponding to the auto focus on the display.
According to claim 1,

further comprising a display;

the processor is

When the crosstalk data measured by the distance sensor exceeds a reference value, the electronic device displays a user interface for removing foreign substances from the surface of the distance sensor on the display.
The method of claim 1, wherein the reference distance

30 cm from the surface of the distance sensor,

the first distance

An electronic device 10 cm from the surface of the distance sensor.
The method of claim 1, wherein the first calibration data is

An electronic device stored in a storage element included in the distance sensor.
The method of claim 9, wherein the first calibration data

An electronic device stored in the storage element during the manufacturing process of the distance sensor.
10. The method of claim 9, wherein the processor

An electronic device for loading the first calibration data stored in the storage element and storing the load in the memory.
The method of claim 1 , wherein the first calibration data and the second calibration data are

An electronic device that is calculated during a manufacturing process of the electronic device and stored in the memory.
The method of claim 1 , wherein the characteristic is

The electronic device including offset information between the distance measured by the distance sensor and the actual distance.
A method for performing auto focus performed in an electronic device, the method comprising:

obtaining and storing first calibration data related to an offset at a first distance shorter than a reference distance by using a distance sensor of the electronic device;

acquiring and storing second calibration data related to crosstalk at a second distance greater than or equal to a reference distance by using the distance sensor;

executing a camera module of the electronic device;

calculating an object distance to an external object based on the first calibration data or the second calibration data; and

and determining a position of a lens unit of the camera module for auto focus of the camera module based on the object distance.
The method of claim 14, wherein the determining of the position of the lens unit comprises:

calculating the object distance by correcting the distance calculated using the distance sensor using the first calibration data or the second calibration data; How to include.