KR20240014988A - Electronic device and method for connecting with external display device based on biometric information - Google Patents

Abstract

One embodiment of the present invention may include a camera module, a communication module, a display, a memory, and a processor operatively connected to at least one of the camera module, the communication module, the display, or the memory. The processor acquires a face image of the user through the camera module while wearing the external display device based on a connection request with the external display device, and compares the acquired face image with face information stored in the memory. Face authentication is performed, and when the face authentication is completed, an authentication code associated with an external display device associated with the stored face information is displayed on the display, and the external display device uses the communication module based on the authentication result of the authentication code. A request for communication connection with the electronic device may be received from the external display device and may be set to connect with the external display device according to the request. Various embodiments are possible.

Description

Method and electronic device for connecting with external display device based on biometric information {ELECTRONIC DEVICE AND METHOD FOR CONNECTING WITH EXTERNAL DISPLAY DEVICE BASED ON BIOMETRIC INFORMATION}

Embodiments of the present invention disclose a method and electronic device for connecting to an external display device based on biometric information.

With the development of digital technology, various types of electronic devices such as mobile communication terminals, personal digital assistants (PDAs), electronic notebooks, smart phones, personal computers (tablet PCs), and wearable devices are widely used. In order to support and increase functionality, the hardware and/or software aspects of electronic devices are continuously being improved.

For example, an electronic device (e.g., a smart phone) can connect with a wearable display device (e.g., AR glasses) to provide virtual reality (VR), augmented reality (AR), and/or mixed reality. It is possible to provide expanded reality (XR) content such as ; MR). Electronic devices include a tethered AR method that provides virtual content generated by an electronic device through the display of a wearable display device, and a stand-alone method in which a wearable display device independently generates virtual content and provides it through the display without being connected to an electronic device. We are implementing an AR environment similar to the AR method.

The electronic device executes an application for connecting to an external display device (or wearable display device) (e.g., AR glasses), and can check the ID of the external display device through the executed application. When the ID of the confirmed external display device is selected, the electronic device can receive the pass code displayed on the external display device from the user. The electronic device can be connected to an external display device once the entered pass code is confirmed. Afterwards, the user may need to confirm and select an ID and enter a pass code every time the electronic device is connected to an external display device.

In one embodiment, a method and device for relatively easily connecting an electronic device and an external display device while a user is wearing an external display device using biometric information may be disclosed.

The technical problem to be achieved in this document is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An electronic device according to an embodiment of the present disclosure includes a camera module, a communication module, a display, a memory, and a processor operatively connected to at least one of the camera module, the communication module, the display, or the memory, Based on a connection request with an external display device, the processor acquires a face image of the user through the camera module while wearing the external display device, compares the acquired face image with face information stored in the memory, and When authentication is completed, an authentication code associated with an external display device associated with the stored face information is displayed on the display, and the external display device transmits the authentication code to the communication module based on an authentication result of the authentication code. A communication connection with the electronic device may be requested from an external display device, and may be set to connect with the external display device according to the request.

An electronic device according to an embodiment of the present disclosure includes a camera module, a communication module, a display, a memory, and a processor operatively connected to at least one of the camera module, the communication module, the display, or the memory, The processor detects whether the electronic device is worn by the user, photographs the authentication code displayed on the display of the external electronic device through the camera module while the electronic device is worn by the user, and Authenticates and extracts device information of the external electronic device from the authentication code, transmits a connection request to the external electronic device through the communication module based on the extracted device information, and the external electronic device responds to the request. It can be set up to connect with the device.

A method of operating an electronic device according to an embodiment of the present disclosure includes acquiring a face image of a user through a camera module of the electronic device while wearing the external display device based on a connection request with an external display device; An operation of authenticating a face by comparing the obtained facial image with facial information stored in the memory of the electronic device. When the face authentication is completed, an authentication code associated with an external display device associated with the stored facial information is displayed on the display of the electronic device. An operation of displaying, an operation of receiving a request for a communication connection with the electronic device from the external display device through a communication module of the electronic device based on the authentication result of the authentication code on the external display device, and responding to the authenticated face information This may include an operation of performing iris authentication by comparing iris information stored in the security area of the memory, and an operation of connecting to the external display device when the iris authentication is completed.

According to one embodiment, the user's face is authenticated through an electronic device while the user is wearing an external display device, and when face authentication is completed, an authentication code (e.g., barcode or QR code) is sent to authenticate the external display device. By displaying the authentication code on the display of the electronic device and recognizing the authentication code with the external display device, the connection operation between the electronic device and the external display device can be relatively simplified.

According to one embodiment, iris information acquired through iris authentication in an external display device and face information acquired through face authentication in an electronic device are matched with each other using device information of the external display device, so that the electronic device and the external display device Security can be improved when connecting between devices.

According to one embodiment, the user's face is authenticated through an electronic device, and the electronic device and the external display device are connected only when iris authentication is completed through an external display device (e.g., composite authentication), thereby increasing user convenience. Relatively high security can be maintained.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a block diagram of an electronic device in a network environment according to an embodiment.
FIG. 2 is a diagram illustrating an example of an external display device according to an embodiment.
FIG. 3 is a block diagram of an external display device according to an exemplary embodiment.
FIG. 4 is a flowchart illustrating a method of matching biometric information between an electronic device and an external display device according to an embodiment.
FIG. 5 is a flowchart illustrating a method of connecting an electronic device and an external display device according to an embodiment.
FIG. 6 is a diagram illustrating an example of recognizing an authentication code displayed on an electronic device by an external display device, according to an embodiment.
Figure 7 is a flowchart illustrating a method of operating an electronic device according to an embodiment.
FIG. 8 is a flowchart illustrating a method of operating an external display device according to an embodiment.
FIG. 9 is a flowchart illustrating an iris authentication method before connection between an electronic device and an external display device, according to an embodiment.
FIG. 10 is a flowchart illustrating an iris authentication method before connection between an electronic device and an external display device, according to an embodiment.
Figure 11 is a flowchart illustrating an iris authentication method of an electronic device according to an embodiment.
Figure 12 is a flowchart illustrating an iris authentication method of an external display device according to an embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 to communicate within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 198 or the second network 199, is connected to the plurality of antennas by, for example, the communication module 190. can be selected Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

FIG. 2 is a diagram illustrating an example of an external display device according to an embodiment.

Referring to FIG. 2, an external display device 201 (or wearable display device) (e.g., electronic device 102 of FIG. 1) according to an embodiment includes a display in the form of glasses and displays a part of the user's body (e.g. : worn on the face or head) and may include various types of devices that provide augmented reality (AR), mixed reality (MR), and/or virtual reality (VR) services. there is. For example, the external display device 201 may be implemented in the form of at least one of glasses, goggles, a helmet, or a hat, but is not limited thereto. The external display device 201 described below may be a device that includes at least some of the components included in the electronic device 101 as previously described with reference to FIG. 1 . Even if not mentioned in the following description, the external display device 201 according to the present disclosure can be interpreted as including various components as described with reference to FIG. 1.

According to one embodiment, the external display device 201 may provide an AR service that adds virtual information (or virtual objects) to at least a portion of the actual space (or environment). For example, the external display device 201 may provide the user with virtual information by superimposing it on the actual space corresponding to the wearer's field of view (FOV).

This external display device 201 includes a glass member (or window member) 210 disposed at a position corresponding to both eyes of the user (e.g., left eye and right eye), and a main frame (or body) that fixes the glass member 210. unit) 240, a support frame (or support member) 250 connected at both ends of the main frame 240 and mounted on at least a portion of the user's ear, and a camera module 280 (e.g., a camera for filming). You can.

According to one embodiment, the glass member 210 may include a first glass 220 corresponding to the user's left eye and a second glass 230 corresponding to the user's right eye. According to one embodiment, the glass member 210 may be supported by the main frame 240. For example, the glass member 210 may be fitted into the opening formed in the main frame 240. According to one embodiment, an AR image emitted from a display module (eg, display module 320 of FIG. 3) may be projected on the glass member 210.

According to one embodiment, a waveguide (eg, an optical waveguide or a transparent waveguide) may be formed in at least some areas of the glass member 210. In one embodiment, the waveguide may serve to guide the AR image emitted from the display module to the user's eyes. According to one embodiment, the glass member 210 is implemented in a form that is separated into a first glass 220 and a second glass 230 to correspond to the user's left eye and right eye, respectively, as illustrated in FIG. 2. Although this is used as an example, according to some embodiments, the glass member 210 may be implemented in the form of a single glass without being divided into the first glass 220 and the second glass 230.

According to one embodiment, the main frame 240 and the support frame 250 may be implemented in the form of glasses.

In one embodiment, the main frame 240 may be a structure that is at least partially mounted on the user's nose. According to one embodiment, the main frame 240 may support the glass member 210. According to one embodiment, the main frame 240 may be formed of synthetic resin material. According to one embodiment, the glass member 210 is inserted into the opening formed in the main frame 240, so that the main frame 240 can support the glass member 210.

In one embodiment, the support frame 250 includes a first support frame 260 that is at least partially mounted on the ear in the first direction (e.g., the left ear) and a second support frame 260 that is at least partially mounted on the ear in the second direction (e.g., the right ear). 2 may include a support frame 270. For example, the main frame 240 and the support frame 250 (e.g., the first support frame 260 and the second support frame 270) may be connected through a hinge portion (not shown) and may be connected by folding (folding). ) can be combined to make it possible.

In one embodiment, the support frame 250 may be rotatably connected to the main frame 240. According to one embodiment, the support frame 250 may include a first support frame 260 and a second support frame 270. The first support frame 260 may be connected to the main frame 240 from the left side (eg, the first direction) with respect to the main frame 240 when viewed in direction ‘A’. The second support frame 270 may be connected to the main frame 240 from the right side (eg, the second direction) with respect to the main frame 240 when viewed from direction ‘A’.

In one embodiment, the support frame 250 may be fixedly installed on a portion of the main frame 240. For example, the first support frame 260 connected to the left side of the main frame 240 and the second support frame 270 connected to the right side of the main frame 240 may be formed to be connected to each other. According to one embodiment, the support frame 250 connected to both sides of the main frame 240 forms a ring shape and can be worn by fitting on the user's head. In addition, the support frame 250 can be modified into various forms so that the external display device 201 can be worn on the user's face.

According to one embodiment, the support frame 250 may be formed to cover at least part of the user's ears. For example, the external display device 201 may be worn on the user's face in such a way that the support frame 250 connected to the main frame 240 covers at least part of the user's ears. According to one embodiment, the support frame 250 may rotate relative to the main frame 240. According to one embodiment, the support frame 250 is rotated to approach the main frame 240, thereby reducing the volume of the external display device 201.

According to one embodiment, the display module (e.g., the display module 320 of FIG. 3) is the processor 120 of the electronic device 101 or the processor of the external display device 201 (e.g., the processor 390 of FIG. 3). ), you can output the AR image created. When an AR image is generated in the display module and projected onto the glass member 210, the visible light (L) incident from the front (e.g., the direction the user is looking) through the glass member 210 causes an object included in the AR image to appear. By combining them, AR can be implemented. The display module may be a projector (e.g., micro projector, pico projector) that is relatively small in size. For example, the display module may be a laser scanning display (LSD), a digital micro-mirror display (DMD), and/or liquid crystal on silicon (LCoS). According to one embodiment, the display module may be a transparent display. In this case, the light emitting element included in the display module may be directly disposed on the glass member 210. In addition, the display module may be a variety of display devices for implementing AR.

In one embodiment, the glass member 210, the support frame 250, and/or the display module (e.g., the display module 320 in FIG. 3) may be provided as a pair to correspond to the user's left eye and right eye. . For example, the glass member 210 may include a first glass 220 and a second glass 230, and the support frame 250 may include a first support frame 260 and a second support frame 270. may include. According to some embodiments, at least some of the components described above may have different configurations corresponding to the left eye and those corresponding to the right eye.

According to one embodiment, the camera module 280 may represent, for example, a photographing camera (eg, a front photographing camera). For example, the camera module 280 may be implemented by being placed on the main frame 240 to capture a subject in front of the external display device 201 (or in front of the user's view). For example, the camera module 280 is positioned approximately at the center (or center point) between the first glass 220 and the second glass 230 in the main frame 240 to capture the front of the main frame 240. can be placed. In one embodiment, the front of the main frame 240 may refer to the direction the user faces when the user wears the external display device 201. In one embodiment, the external display device 201 may include a camera module 280 and a plurality of other cameras.

The camera module 280 may be a photographing camera (eg, an RGB camera) for capturing an image corresponding to the user's field of view (FoV) and/or measuring the distance to an object. The camera module 280 may further include an eye tracking camera module to check the direction of the user's gaze. The camera module 280 may further include a gesture camera module for recognizing a certain space.

The photographing camera may photograph a front direction of the external display device 201, and the gaze tracking camera may photograph a direction opposite to the photographing direction of the photographing camera. For example, the gaze tracking camera may include a first gaze tracking camera that partially captures the user's left eye, and a second gaze tracking camera that partially captures the user's right eye. According to one embodiment, the external display device 201 may display virtual objects (or virtual information) related to the AR service based on image information related to the actual space obtained through the shooting camera. According to one embodiment, the external display device 201 is based on display modules arranged corresponding to both eyes of the user (e.g., a first display module corresponding to the left eye and/or a second display module corresponding to the right eye). Virtual objects can be displayed. According to one embodiment, the external display device 201 may display a virtual object based on preset setting information (e.g., resolution, frame rate, brightness, and/or display area). .

According to one embodiment, the photographing camera may include a high resolution camera such as a high resolution (HR) camera and/or a photo video (PV) camera. For example, the shooting camera can be used to obtain high-definition images, such as an autofocus function and image stabilization (OIS). The photographing camera may be implemented as a GS (global shutter) camera and an RS (rolling shutter) camera in addition to a color camera.

According to one embodiment, the gaze tracking camera may detect the user's gaze direction (eg, eye movement). For example, the gaze tracking camera can detect the user's eyes and track the gaze direction. In one embodiment, the tracked gaze direction can be used to move the center of a virtual image including a virtual object in response to the gaze direction. For example, the eye tracking camera may be a GS (global shutter) camera to detect the pupil and track fast eye movement without screen drag, and the performance and specifications of each eye tracking camera are practical. can be the same.

According to one embodiment, the recognition camera may detect a user gesture and/or a certain space within a preset distance (eg, a certain space). According to one embodiment, the recognition camera may be used for 3DoF (depth of field), 6DoF head tracking, hand detection, and/or hand tracking. For example, the recognition camera can be used to perform simultaneous localization and mapping (SLAM) functions through spatial recognition for 6DoF and depth shooting. According to some embodiments, the recognition camera may be used for a gesture recognition function to recognize user gestures. The recognition camera may include a camera including GS. For example, the recognition camera is a GS that has relatively little screen drag (or the RS phenomenon can be reduced), such as a rolling shutter (RS) camera, in order to detect and track fast hand movements and/or fine movements of fingers. It may include a camera including.

According to one embodiment, the external display device 201 may use at least one camera to detect the eye corresponding to the main eye and/or the secondary eye among the user's left eye and/or right eye. For example, the external display device 201 may detect the eye corresponding to the primary eye and/or the secondary eye based on the user's gaze direction with respect to the external object or virtual object.

The number or location of the camera modules 280 (e.g., the photographing camera, the eye tracking camera, and/or the recognition camera) may not be limited. For example, the number and location of camera modules 280 (e.g., the photographing camera, the eye tracking camera, and/or the recognition camera) based on the form (e.g., shape or size) of the external display device 201. can be changed in various ways.

According to one embodiment, the external display device 201 includes at least one lighting member (or light emitting member) to increase the accuracy of the camera module 280 (e.g., the photographing camera, the eye tracking camera, and/or the recognition camera). A device (illumination LED) may be included. For example, the first lighting member may be disposed in a portion corresponding to the user's left eye, and the second lighting member may be disposed in a portion corresponding to the user's right eye.

According to one embodiment, the external display device 201 may further include a microphone (eg, a first microphone, a second microphone) for receiving the user's voice and surrounding sounds. According to one embodiment, the external display device 201 may further include a first speaker for transmitting an audio signal to the user's left ear and a second speaker for transmitting an audio signal to the user's right ear.

FIG. 3 is a block diagram of an external display device according to an exemplary embodiment.

Referring to FIG. 3, the external display device 201 according to an embodiment includes glass 310 (e.g., glass member 210 of FIG. 2), a display module 320, a sensor module 330, and a communication module ( 340), a memory 350, an audio module 360, a camera module 370 (e.g., the camera module 280 of FIG. 2), a battery 380, and a processor 390. According to one embodiment, components included in the external display device 201 may be understood as, for example, hardware modules (e.g., circuitry).

According to one embodiment, the glass 310 (e.g., the first glass 220 and/or the second glass 230 in FIG. 2) includes a condensing lens and/or a transparent waveguide. can do. For example, the transparent waveguide may be at least partially located in a portion of the glass 310 . According to one embodiment, the light emitted from the display module 320 may be incident on one end of the glass 310, and the incident light may be transmitted through a waveguide and/or a waveguide (e.g., waveguide) formed within the glass 310. It can be delivered to the user. The waveguide may be made of glass or polymer, and may include a nanopattern formed on one of the inner or outer surfaces, for example, a polygonal or curved grating structure.

According to one embodiment, the display module 320 may include a plurality of panels (or display areas), and the plurality of panels may be located on the glass 310. According to one embodiment, at least a portion of the display module 320 may be composed of a transparent element, and the user may perceive the actual space behind the display module 320 through the display module 320. . According to one embodiment, the display module 320 may display a virtual object on at least a portion of the transparent element so that the user sees the virtual object (or virtual information) as being added to at least a portion of the actual space.

According to one embodiment, the sensor module 330 may include a proximity sensor 331, an illumination sensor 332, and/or a gyro sensor 333. The proximity sensor 331 can detect objects adjacent to the external display device 201. The illuminance sensor 332 can measure the brightness level around the external display device 201. According to one embodiment, the processor 390 uses the illuminance sensor 332 to check the brightness level around the external display device 201 and changes the brightness-related setting information of the display module 320 based on the brightness level. You can. For example, if the surrounding brightness is brighter than the preset brightness, the processor 390 may set the brightness level of the display module 320 higher to increase user visibility. The gyro sensor 333 can detect the status and location of the external display device 201. For example, the gyro sensor 333 may detect whether the external display device 201 is properly worn on the user's head. For another example, the gyro sensor may detect the external display device 201 or the movement of a user wearing the external display device 201.

The communication module 340 may correspond to the wireless communication module 192 as illustrated in FIG. 1 . According to one embodiment, the external display device 201 uses the communication module 340 to communicate with an external device (e.g., the server 201 of FIG. 1 and/or other electronic devices 101, 102, and 104) through a network. You can perform wireless communication with. For example, the external display device 201 may perform wireless communication with the electronic device 101 and exchange commands and/or data with each other. According to some embodiments, the external display device 201 may be at least partially controlled by an external device (eg, the electronic device 101) through the communication module 340. For example, the external display device 201 may perform at least one function under the control of another external electronic device. According to one embodiment, the external display device 201 displays object information (e.g., image) located in real space, distance information to the object, user's gaze information, and/or user's gesture through the camera module 370. Information may be transmitted to an external device (eg, the server 201 of FIG. 1 and/or other electronic devices 101, 102, and 104) through the communication module 340.

According to one embodiment, the communication module 340 may include an antenna module 345. For example, the communication module 340 may support various technologies (e.g., beamforming, multiple input/output (MIMO), and/or array antenna) to secure performance in a designated frequency band. there is. According to one embodiment, the antenna module 345 may transmit a signal or power to or receive a signal or power from the outside (eg, a peripheral device and/or server of the external display device 201). According to some embodiments, the antenna module 345 may include a plurality of antennas (eg, an array antenna). According to one embodiment, signals or power may be transmitted or received between the communication module 340 and the outside through the antenna module 345.

The memory 350 may correspond to the memory 130 as described in the description with reference to FIG. 1 . According to one embodiment, the memory 350 may store various data used by the external display device 201 when the external display device 201 provides an AR service. Data may include, for example, input data or output data for software (eg, program 140 of FIG. 1) and commands related thereto. The memory 350 may store instructions that enable the processor 390 to operate. Instructions may be stored as software on memory 350 and may be executable by processor 390.

The audio module 360 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound based on the control of the processor 390.

Camera module 370 may include at least one camera. According to one embodiment, the camera module 370 may include at least one of a photographing camera, an eye tracking camera, or a recognition camera.

The battery 380 may supply power to at least one component of the external display device 201. Battery 380 may include a plurality of batteries. For example, battery 380 may include a first battery supplying power from a first support frame (e.g., first support frame 260 in FIG. 2) and a second support frame (e.g., second support frame 260 in FIG. 2). (270)) may include a second battery that supplies power.

The processor 390 may execute a program (e.g., program 140 of FIG. 1) stored in the memory 350 to control at least one other component (e.g., a hardware or software component) and store various data. Processing or calculations can be performed. The processor 390 may operate to provide an AR service to the user based on the external display device 201 alone or in connection with the electronic device 101. Here, connection may mean that the external display device 201 is paired with the electronic device 101 through short-range wireless communication (eg, Bluetooth). The processor 390 displays at least one virtual object in the actual space corresponding to the line of sight (e.g., FOV, angle of view (AOV)) of the user wearing the external display device 201 through the display module 320. At least one virtual object may be output so that (or virtual information) is added.

According to one embodiment, the processor 390 may correspond to the processor 120 as described in the description with reference to FIG. 1. According to one embodiment, the processor 390 executes an application (e.g., an AR application) (e.g., the application 146 of FIG. 1) and displays a peripheral device connected to the external display device 201 through a data communication link on an AR screen. It can be controlled from the top, and various data processing or calculations related to AR services can be performed. According to one embodiment, as at least part of the data processing or computation, the processor 390 stores data received through the communication module 340 in the memory 350, processes the data stored in the memory 350, and The resulting data may be stored in the memory 350 and/or transmitted to a peripheral device (eg, electronic device 101). According to one embodiment, the processor 390 uses the display module 320 to display one image (e.g., an AR screen) by overlapping various digital contents (e.g., an AR image) on the real world provided through the display module 320. ) can be controlled.

According to one embodiment, the processor 390 may operate while being included in either the first support frame 260 or the second support frame 270 of the external display device 201. According to some embodiments, the processor 390 may include a first processor (e.g., a main processor) and a second processor (e.g., a main processor) in each of the first support frame 260 and the second support frame 270 of the external display device 201. : auxiliary processor), and may include a dual system structure that operates in synchronization with the first processor and the second processor.

An electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment of the present invention includes a camera module (e.g., the camera module 180 of FIG. 1) and a communication module (e.g., the communication module of FIG. 1). 190)), a display (e.g., display module 160 of FIG. 1), a memory (e.g., memory 130 of FIG. 1), and operating with at least one of the camera module, the communication module, the display, or the memory. It includes a processor (e.g., processor 120 of FIG. 1) connected to the external display device, and the processor wears the external display device based on a connection request with an external display device (e.g., external display device 201 of FIG. 2). In one state, a face image of the user is acquired through the camera module, the acquired face image is compared with the face information stored in the memory to authenticate the face, and when the face authentication is completed, an external display associated with the stored face information is performed. An authentication code associated with the device is displayed on the display, and a communication connection is established with the electronic device from the external display device 201 through the communication module based on the authentication result of the authentication code in the external display device 201. A request may be received and set to connect to the external display device 201 according to the request.

In the security area of the memory, the user's face information and the user's iris information may be stored in correspondence with the device information of the external display device.

The processor may be configured to extract facial feature information by analyzing the acquired facial image and determine whether the extracted facial feature information substantially matches facial information stored in the security area.

The processor determines that the face authentication is successful when the extracted facial feature information substantially matches the facial information stored in the secure area, and the extracted facial feature information substantially matches the facial information stored in the secure area. If it does not match, it may be set to determine that the face authentication has failed.

The processor may be set to generate the authentication code based on device information of the external display device stored in a secure area of the memory in response to the stored face information.

The authentication code may be obtained by encrypting the service set identifier (SSID) of the external display device and the SSID of the electronic device corresponding to the stored face information with the public key of the external display device.

The processor may be set to allow the external display device to authenticate the displayed authentication code and receive a communication connection request from the external display device with the electronic device through the SSID of the electronic device included in the authentication code.

The processor identifies iris information stored in a secure area of the memory in response to the stored face information, receives an iris image from the external display device through a communication channel established with the external display device, and receives the identified iris information. It may be set to perform iris authentication by comparing the received iris image.

The processor determines that iris authentication of the user is successful when the iris characteristic information extracted from the iris image and the identified iris information substantially match, and when the iris authentication is successful, the processor connects to the external display device. can be set to complete.

The processor determines that iris authentication of the user has failed when the iris feature information extracted from the iris image and the identified iris information do not substantially match, and when the iris authentication fails, the processor determines that the user's iris authentication has failed. It may be set to request an iris image obtained by re-photographing the user's iris, and receive an iris image obtained by re-photographing the user's iris according to the request to re-perform iris authentication.

An electronic device (e.g., the external display device 201 of FIG. 2) according to an embodiment of the present invention includes a camera module (e.g., the camera module 370 of FIG. 3) and a communication module (e.g., the communication module of FIG. 3). (340)), a display (e.g., display module 320 in FIG. 3), a memory (e.g., memory 350 in FIG. 3), and at least one of the camera module, the communication module, the display, or the memory and an operatively connected processor (e.g., processor 390 of FIG. 3), wherein the processor detects whether the electronic device is worn by a user, and detects whether the electronic device is worn by the user. Photograph the authentication code displayed on the display (e.g., display module 160 of FIG. 1) of an external electronic device (e.g., electronic device 101 of FIG. 1) through a camera module, authenticate the authentication code, and obtain the authentication code. Extract device information of the external electronic device from the device, transmit a connection request to the external electronic device through the communication module based on the extracted device information, and configure the device to connect to the external electronic device that responds to the request. It can be.

The processor may be set to photograph the authentication code when the user's face is successfully authenticated by the external electronic device while the user is wearing the electronic device.

The authentication code may be obtained by encrypting the SSID of the electronic device and the SSID of the external electronic device corresponding to the face information of the authenticated user with the public key of the electronic device.

The processor may be set to decrypt the authentication code with the private key of the electronic device and extract the SSID of the electronic device and the SSID of the external electronic device from the authentication code.

When the SSID of the electronic device extracted from the authentication code matches the SSID stored in the memory, the processor requests a communication connection to the external electronic device through the SSID of the external electronic device extracted from the authentication code. can be set.

The processor acquires an iris image by photographing the user's iris through the camera module, receives iris information corresponding to the face information of the authenticated user from the external electronic device, and combines the acquired iris image with the It may be set to perform iris authentication by comparing received iris information.

The processor determines that iris authentication of the user is successful when the iris feature information extracted from the iris image and the received iris information substantially match, and when the iris authentication is successful, the processor connects to the external electronic device. can be set to complete.

If the iris feature information extracted from the iris image and the received iris information do not substantially match, the processor determines that iris authentication of the user has failed, and if the iris authentication fails, re-authenticates the user's iris. It may be set to acquire an iris image by taking a photo, and re-perform iris authentication by comparing the obtained iris image with the received iris information.

FIG. 4 is a flowchart illustrating a method of matching biometric information between an electronic device and an external display device according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 4, in operation 401, an electronic device (e.g., electronic device 101 of FIG. 1) according to an embodiment may be connected to an external display device (e.g., external display device 201 of FIG. 2). there is. Here, connection may mean that the electronic device 101 is paired with the external display device 201 through short-range wireless communication. The external display device 201 (or wearable display device) may include a display in the form of glasses and various types of devices that are worn on a part of the user's body and display content. For example, when the electronic device 101 and the external display device 201 are connected for the first time, they may be connected to each other using a conventional connection method. For example, in a conventional connection method, the electronic device 101 executes an application (e.g., the application 146 in FIG. 1) for connection to the external display device 201 based on a user input, and the executed The ID of the external display device 201 can be confirmed by the application. When the confirmed ID of the external display device 201 is selected, the electronic device 101 can receive the pass code displayed on the external display device 201 from the user. The electronic device 101 can connect to the external display device 101 when the input pass code is confirmed.

Hereinafter, operations 403 to 409 may be performed through an application for connecting the electronic device 101 and the external display device 201.

In operation 403, the electronic device 101 may photograph the user's face while wearing the external display device 201. Conventionally, in order to authenticate the user's face, the user's face had to be photographed without wearing the external display device 201 (e.g., with the external display device 201 removed), but in the present invention, the user's face must be photographed while wearing the external display device 201. The user's face can be authenticated in this state. The electronic device 101 may capture the user's face by driving a camera (eg, the camera module 180 of FIG. 1 ) while the user is wearing the external display device 201 .

In operation 405, the electronic device 101 may obtain facial information based on the captured facial image. The facial information may include the user's facial feature information (eg, facial feature value). The electronic device 101 may acquire the facial information by analyzing the captured facial image.

In operation 404, the external display device 201 may capture the user's iris. The external display device 201 may capture the user's iris by executing a camera module (eg, the camera module 370 in FIG. 3).

In operation 406, the external display device 201 may obtain iris information based on the captured iris image. The iris information may include the user's iris feature information (eg, iris feature value). The external display device 201 may obtain the iris information by analyzing the captured iris image. According to one embodiment, the external display device 201 transmits the captured iris image to the electronic device 101 (not shown), and the electronic device 101 may obtain iris information based on the iris image. there is.

Operations 403 and 405 may be performed in parallel with operations 404 and 406, or may be performed sequentially. For example, operations 403 and 405 may be performed before operations 404 and 406, or may be performed after operations 404 and 406. This is only an implementation issue and does not limit the present invention.

In operation 407, the external display device 201 may transmit the device information of the external display device 407 and the iris information obtained in operation 406 to the electronic device 101. The device information may include a public key of the external display device 407 and a service set identifier (SSID) of the external display device 407. The device information may be necessary to match the external display device 407 to the iris information and the face information. Since the external display device 201 is connected to the electronic device 101, the device information and the iris information can be transmitted to the electronic device 101 through an established communication channel (or communication session).

In operation 409, the electronic device 101 may match the iris information and the face information to the device information and store them. The electronic device 101 may receive the device information and the iris information from the external display device 201 through a communication module (eg, the communication module 190 of FIG. 1). The electronic device 101 may pair iris information and face information acquired for one user with device information of the external display device 201 and store them in memory (eg, memory 130 in FIG. 1). According to one embodiment, the electronic device 101 may store one or more iris information and face information for one user. The electronic device 101 may store the iris information and the face information paired with the device information in a secure area (eg, secure world) of the memory 130.

FIG. 5 is a flowchart illustrating a method of connecting an electronic device and an external display device according to an embodiment. 5 may be performed when the electronic device and the external display device are connected to each other after performing the flowchart of FIG. 4 at least once.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 5, in operation 501, an electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment is connected to an external display device (e.g., the external display device 201 of FIG. 2). An application (e.g., application 146 in FIG. 1) can be executed. The electronic device 101 may receive a request from a user to execute the application and execute the application.

In operation 503, the electronic device 101 may photograph the user's face while the user is wearing the external display device 201. For example, when a connection to the external display device 201 is requested through the executed application, the electronic device 101 operates a camera (e.g., the camera module 180 in FIG. 1) to capture the user's face. You can. When photographing the user's face, the user may be wearing the external display device 201.

In operation 505, the electronic device 101 may analyze the captured facial image. The electronic device 101 may extract facial feature information by analyzing the captured facial image. The electronic device 101 may determine whether the extracted facial feature information corresponds to facial information stored in a secure area of a memory (eg, memory 130 in FIG. 1). In the security area of the memory 130, iris information and face information may be stored paired with device information of the external display device 201. The electronic device 101 may determine whether the extracted facial feature information substantially matches facial information stored in the security area.

In operation 507, the electronic device 101 may display the authentication code on a display (eg, the display module 160 of FIG. 1). If the extracted facial feature information substantially matches the facial information stored in the security area, the electronic device 101 may generate an authentication code. The authentication code may be a code needed for the electronic device 101 to authenticate the external display device 201. The authentication code may be generated as an encryption code (eg, a quick response (QR) code or a barcode). The authentication code may be obtained by encrypting the SSID of the external display device 201 and the SSID of the electronic device 101 corresponding to the facial information with the public key of the external display device 201. The electronic device 101 may display the generated authentication code on a display (eg, the display module 160 of FIG. 1).

In operation 509, the external display device 201 may photograph the authentication code. The user may photograph the authentication code displayed on the electronic device 101 using the external display device 201 while wearing the external display device 201 . For example, the external display device 201 may execute an application for connection to the electronic device 101 and execute a camera module (e.g., the camera module 370 in FIG. 3) by the executed application. . The camera module 370 may be a photographing camera that photographs the front of the external display device 201.

In operation 511, the external display device 201 may extract device information of the electronic device 101 included in the authentication code. The external display device 201 may decrypt the captured authentication code using the private key of the external display device 201. Since the authentication code is encrypted with the public key of the external display device 201, it can be decrypted with the private key of the external display device 201. The external display device 201 may decrypt the authentication code and extract device information of the electronic device 101 included in the authentication code. The external display device 201 can decrypt the authentication code and extract the SSID of the external display device 201 and the SSID of the electronic device 101. Device information of the electronic device 101 may include the SSID of the electronic device 101.

In operation 513, the external display device 201 may attempt to connect to the electronic device 101 using device information of the electronic device 101. The external display device 201 may determine whether its own SSID matches the SSID of the external display device 201 extracted from the authentication code. If its own SSID and the SSID of the external display device 201 extracted from the authentication code match, the external display device 201 may request the establishment of a communication channel with the SSID of the electronic device 101.

In operation 515, the electronic device 101 can check device information of the external display device 201. When the electronic device 101 receives a request to establish a communication channel from the external display device 201, the electronic device 101 may identify the SSID of the external display device 201 corresponding to the face information. The electronic device 101 may determine whether the SSID of the external display device 201 that requested establishment of the communication channel matches the SSID of the external display device 201 corresponding to the face information. When the SSID of the external display device 201 that has requested the communication channel formation matches the SSID of the external display device 201 corresponding to the face information, the electronic device 101 is configured to: 201) can be allowed.

According to one embodiment, after displaying the authentication code, the electronic device 101 may wait for connection with the external display device 201 for a set time (eg, 1 minute, 5 minutes). The electronic device 101 may determine whether establishment of a communication channel is requested from the external display device 201 within the set time. When establishment of a communication channel is requested from the external display device 201 within the set time, the electronic device 101 generates an external display device ( 201) can be determined whether the SSID matches. If the electronic device 101 requests establishment of a communication channel from the external display device 201 after the set time has elapsed, the electronic device 101 may disallow connection with the external display device 201 that requested the communication channel.

In operation 517, the electronic device 101 can connect to the external display device 201. The electronic device 101 performs face authentication while the user is wearing the external display device 201, and displays (or outputs) an authentication code when face authentication is successful (or complete), thereby displaying (or outputting) an authentication code on the external display device 201. ) can perform a connection operation. According to one embodiment, the electronic device 101 may authenticate the iris of a user wearing the external display device 201 before connecting to the external display device 201 (eg, after operation 515). The electronic device 101 may connect to the external display device 201 when both face authentication and iris authentication are successful (or complete) while the user is wearing the external display device 201. The iris authentication method will be described in detail with reference to FIGS. 9 to 12.

FIG. 6 is a diagram illustrating an example of recognizing an authentication code displayed on an electronic device by an external display device, according to an embodiment.

Referring to FIG. 6, the user can photograph the authentication code displayed on the electronic device (e.g., the electronic device 101 of FIG. 1) while wearing an external display device (e.g., the external display device 201 of FIG. 2). there is. Reference numeral 601 may represent the line of sight of a user wearing the external display device 201, or may be taken from a camera of the external display device 201 (e.g., camera module 370 in FIG. 3). . The external display device 201 can capture and recognize the authentication code displayed on the electronic device 101 in order to connect to the electronic device 101.

FIG. 7 is a flowchart 700 illustrating a method of operating an electronic device according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 701 to 713 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1).

Referring to FIG. 7, in operation 701, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1) according to an embodiment operates an external display device (e.g., FIG. An application (e.g., the application 146 of FIG. 1) for connection to the external display device 201 of 2 may be executed. The external display device 201 (or wearable display device) may include a display in the form of glasses and various types of devices that are worn on a part of the user's body and display content. Operation 701 may be performed after the electronic device 101 and the external display device 201 are connected at least once. The processor 120 may receive a request to execute the application from the user and execute the application.

In operation 703, the processor 120 may photograph the user's face using a camera (eg, the camera module 180 of FIG. 1). The processor 120 may capture the user's face through the camera module 180 while the user is wearing the external display device 201. The processor 120 may acquire a facial image of the user while the user is wearing the external display device 201.

At operation 705, processor 120 may authenticate the face. The processor 120 may compare the acquired facial image of the user with facial information stored in a secure area of memory (eg, memory 130 of FIG. 1). The processor 120 may extract facial feature information from the acquired facial image of the user. The processor 120 may determine whether the extracted facial feature information corresponds to facial information stored in the security area. In the security area, iris information and face information may be paired and stored in device information (e.g., public key, SSID) of the external display device 201. The processor 120 may determine whether the extracted facial feature information substantially matches facial information stored in the security area.

If the extracted facial feature information substantially matches the facial information stored in the security area, the processor 120 may determine that face authentication is successful and perform operation 707. If the extracted facial feature information does not substantially match the facial information stored in the secure area, the processor 120 may determine that face authentication has failed and return to operation 703. Returning to operation 703, processor 120 may re-photograph the user's face. According to one embodiment, the processor 120 may terminate if face authentication fails even after performing face authentication a set number of times (e.g., 3 times, 5 times, etc.).

If face authentication is successful, in operation 707, the processor 120 may generate an authentication code corresponding to the face information. The authentication code may be a code needed for the electronic device 101 to authenticate the external display device 101. The authentication code may be generated as an encryption code (eg, QR code or barcode). The authentication code may be obtained by encrypting the SSID of the external display device 201 and the SSID of the electronic device 101 corresponding to the facial information with the public key of the external display device 201.

According to one embodiment, the processor 120 may generate the authentication code in advance and store it in a secure area of the memory 130. The processor 120 may store the generated authentication code in response to the facial information. If the authentication code is stored in the secure area corresponding to the face information, the processor 120 may omit operation 707. Alternatively, if the authentication code is stored in the security area in response to the face information, the processor 120 may perform an operation of identifying the authentication code stored in the security area instead of generating the authentication code. there is. This is only an implementation issue and is not limited by the description of the present invention.

In operation 709, the processor 120 may display the generated authentication code on a display (eg, display module 160 of FIG. 1). The processor 120 may display the authentication code so that the external display device 201 can recognize the authentication code. The external display device 201 may photograph the authentication code and extract device information of the electronic device 101 included in the authentication code. Since the authentication code is encrypted with the public key of the external display device 201, it can be decrypted with the private key of the external display device 201. The external display device 201 may decrypt the authentication code and extract device information of the electronic device 101 included in the authentication code. The external display device 201 can decrypt the authentication code and extract the SSID of the external display device 201 and the SSID of the electronic device 101. Device information of the electronic device 101 may include the SSID of the electronic device 101. The external display device 201 may attempt to connect to the electronic device 101 using the device information of the electronic device 101. The external display device 201 may request the establishment of a communication channel using the SSID of the electronic device 101.

In operation 711, the processor 120 may receive a connection request from the external display device 201. The processor 120 may receive a connection request from the external display device 201 through a communication module (eg, the communication module 190 of FIG. 1). When receiving a request to establish a communication channel from the external display device 201, the processor 120 may identify the SSID of the external display device 201 corresponding to the face information. The processor 120 may determine whether the SSID of the external display device 201 that requested establishment of the communication channel matches the SSID of the external display device 201 corresponding to the face information. If the SSID of the external display device 201 that requested the establishment of the communication channel matches the SSID of the external display device 201 corresponding to the face information, the processor 120 controls the external display device 201 that requested the establishment of the communication channel. ) can be allowed to connect.

According to one embodiment, the processor 120 may wait for connection with the external display device 201 for a set time (eg, 1 minute, 5 minutes) after displaying the authentication code. The processor 120 may determine whether establishment of a communication channel is requested from the external display device 201 within the set time. When a communication channel establishment request is made from the external display device 201 within the set time, the processor 120 generates an external display device 201 corresponding to the SSID of the external display device 201 requesting the communication channel formation and the face information. ) can be determined whether the SSID matches. If the external display device 201 requests establishment of a communication channel after the set time has elapsed, the processor 120 may disallow a connection with the external display device 201 that requested the communication channel.

In operation 713, the processor 120 may connect to the external display device 201. The processor 120 performs face authentication while the user is wearing the external display device 201, and displays (or outputs) an authentication code when face authentication is successful (or complete), thereby displaying the external display device 201. You can perform connection operations with . According to one embodiment, the processor 120 may authenticate the iris of a user wearing the external display device 201 before connecting to the external display device 201 (eg, after operation 711). The processor 120 may connect to the external display device 201 when both face authentication and iris authentication are successful (or complete) while the user is wearing the external display device 201. The iris authentication method will be described in detail with reference to FIGS. 9 to 12.

FIG. 8 is a flowchart 800 illustrating a method of operating an external display device according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 801 to 711 may be understood as being performed by a processor (e.g., processor 390 of FIG. 3) of an electronic device (e.g., external display device 201 of FIG. 2).

Referring to FIG. 8, in operation 801, a processor (e.g., processor 390 of FIG. 3) of an external display device (e.g., external display device 201 of FIG. 2) according to an embodiment allows the user to wear the device. You can check it. For example, the processor 390 may determine whether the user is wearing the external display device 201 through a sensor module (eg, the sensor module 330 in FIG. 3). The processor 390 may determine whether the user is wearing the external display device 201 based on the sensing value obtained from the sensor module 330.

In operation 803, the processor 390 may capture the authentication code displayed on the electronic device (e.g., the electronic device 101 of FIG. 1). The user may photograph the authentication code displayed on the electronic device 101 using the external display device 201 while wearing the external display device 201 . For example, the processor 390 executes an application to connect to the electronic device 101 based on user input, and executes a camera module (e.g., camera module 370 in FIG. 3) by the executed application. You can do it. The camera module 370 may be a photographing camera that photographs the front of the external display device 201.

In operation 805, the processor 390 may determine whether authentication of the authentication code was successful. Since the authentication code is encrypted with the public key of the external display device 201, it can be decrypted with the private key of the external display device 201. The processor 390 can decrypt the authentication code using the private key of the external display device 201. If the authentication code is decrypted with the private key of the external display device 201, the processor 390 may determine that authentication is successful and perform operation 807. Alternatively, the processor 390 may determine whether its own SSID matches the SSID of the external display device 201 extracted from the authentication code. If the processor 390 matches its own SSID and the SSID of the external display device 201 extracted from the authentication code, the processor 390 may determine that authentication has succeeded.

If the authentication code is not decrypted with the private key of the external display device 201, the processor 390 may determine that authentication has failed and return to operation 803. Alternatively, the processor 390 determines that authentication has failed when its own SSID (e.g., SSID stored in the memory of the external display device 201) and the SSID of the external display device 201 extracted from the authentication code do not match. You can judge. When the processor 390 returns to operation 803, the processor 390 may re-photograph the authentication code displayed on the electronic device 101. According to one embodiment, if the processor 390 fails to authenticate the authentication code displayed on the electronic device 101 within a set number of times, the processor 390 may terminate the connection operation with the electronic device 101.

If authentication of the authentication code is successful, in operation 807, the processor 390 may extract device information of the electronic device 101 included in the authentication code. The processor 390 can decrypt the authentication code and extract the SSID of the external display device 201 and the SSID of the electronic device 101. Device information of the electronic device 101 may include the SSID of the electronic device 101.

In operation 809, the processor 390 may transmit a connection request to the electronic device 101 based on device information of the electronic device 101. The processor 390 requests (or attempts to) connect (e.g., establish a communication channel) with the electronic device 101 using the SSID of the electronic device 101 through a communication module (e.g., the communication module 340 of FIG. 3). You can. After displaying the authentication code, the electronic device 101 may wait for connection with the external display device 201 for a set time (eg, 1 minute, 5 minutes). The electronic device 101 may determine whether a connection request is received from the external display device 201 within the set time. When a connection request is received from the external display device 201 within the set time, the electronic device 101 displays the SSID of the external display device 201 requesting the connection and the external display device 201 corresponding to the face information. You can determine whether the SSID matches. If the SSID of the external display device 201 that requested the connection matches the SSID of the external display device 201 corresponding to the face information, the electronic device 101 connects to the external display device 201 that requested the connection. can be allowed. If the electronic device 101 requests establishment of a communication channel from the external display device 201 after the set time has elapsed, the electronic device 101 may disallow connection with the external display device 201 that requested the communication channel.

In operation 811, the processor 390 may connect to the electronic device 101. The processor 390 can connect to the electronic device 101 when the electronic device 101 allows the connection request. According to one embodiment, the processor 390 may authenticate the iris of a user wearing the external display device 201 before connecting to the electronic device 101 (eg, before operation 811). The processor 390 may connect to the electronic device 101 when both face authentication and iris authentication are successful (or complete) while the user is wearing the external display device 201. The iris authentication method will be described in detail with reference to FIGS. 9 to 12.

FIG. 9 is a flowchart illustrating an iris authentication method before connection between an electronic device and an external display device, according to an embodiment. 9 may be performed before operation 517 of FIG. 5 or may be performed included in operation 517.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 9, in operation 901, an electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment forms a communication channel with an external display device (e.g., the external display device 201 of FIG. 2). The electronic device 101 may form a communication channel with the external display device 201 before connection (e.g., pairing) with the external display device 201. The communication channel formation may be performed using iris information for iris authentication. It may be for exchanging.

In operation 903, the electronic device 101 may identify iris information stored in a secure area of the memory (e.g., memory 130 of FIG. 1). The electronic device 101 may identify iris information corresponding to face information that successfully face authentication while the user is wearing the external display device 201.

In operation 905, the external display device 201 may capture the user's iris. Since the user is wearing the external display device 201, the external display device 201 uses a gaze tracking camera (e.g., the camera module 370 in FIG. 3) to check the direction of the user's gaze. Iris can be photographed. The external display device 201 may acquire the user's iris image by photographing the user's iris.

In the drawing, operation 903 is shown as being performed before operation 905, but operations 903 and 905 may be performed in parallel, or operation 903 may be performed after operation 905 is performed first. This is only an implementation issue, and the present invention is not limited by the description.

In operation 907, the electronic device 101 may transmit the identified iris information through a communication module (eg, the communication module 190 of FIG. 1). The electronic device 101 may transmit the iris information to the external display device 201 through a communication channel established with the external display device 201. Operation 907 may be performed while operation 905 is performed.

In operation 909, the external display device 201 may authenticate the iris. For example, the external display device 201 may receive iris information from the electronic device 101 through a communication module (eg, the communication module 340 of FIG. 3). The external display device 201 may analyze the acquired iris image and extract iris feature information. The external display device 201 may authenticate the user's iris by comparing iris feature information extracted from the iris image with the received iris information.

In operation 911, when the user's iris authentication is successfully completed, the electronic device 101 and the external display device 201 may be connected to each other. The external display device 201 may transmit an iris authentication result (e.g., authentication success) to the electronic device 101, and the electronic device 101 may allow connection with the external display device 201 upon receiving the iris authentication result. there is.

FIG. 10 is a flowchart illustrating an iris authentication method before connection between an electronic device and an external display device, according to an embodiment. 10 may be performed before operation 517 of FIG. 5 or may be performed included in operation 517.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 10, in operation 1001, an electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment forms a communication channel with an external display device (e.g., the external display device 201 of FIG. 2). The electronic device 101 may form a communication channel with the external display device 201 before connection (e.g., pairing) with the external display device 201. The communication channel formation may be performed using iris information for iris authentication. It may be for exchanging. Operation 1001 may be the same or similar to operation 901 of FIG. 9 .

In operation 1003, the electronic device 101 may identify iris information stored in a secure area of a memory (eg, memory 130 of FIG. 1). The electronic device 101 may identify iris information corresponding to face information that successfully face authentication while the user is wearing the external display device 201. Operation 1003 may be the same or similar to operation 903 of FIG. 9 .

In operation 1005, the external display device 201 may capture the user's iris. Since the user is wearing the external display device 201, the external display device 201 uses a gaze tracking camera (e.g., the camera module 370 in FIG. 3) to check the direction of the user's gaze. Iris can be photographed. The external display device 201 may obtain an iris image of the user by photographing the user's iris. Operation 1005 may be the same or similar to operation 905 of FIG. 9 .

In the drawing, operation 1003 is shown as being performed before operation 1005, but operations 1003 and 1005 may be performed in parallel, or operation 1003 may be performed after operation 1005 is performed first. This is only an implementation issue, and the present invention is not limited by the description.

In operation 1007, the external display device 201 may transmit the acquired iris image through a communication module (eg, the communication module 370 of FIG. 3). The external display device 201 may transmit the iris image to the electronic device 101 through a communication channel established with the electronic device 101.

In operation 1009, the electronic device 101 may authenticate the iris. For example, the electronic device 101 may receive an iris image from the external display device 201 through a communication module (eg, the communication module 190 of FIG. 1). The electronic device 101 may extract iris feature information by analyzing the acquired iris image. The electronic device 101 may authenticate the user's iris by comparing iris characteristic information extracted from the iris image with iris information stored in the secure area of the memory 130.

In operation 1011, when the user's iris authentication is successfully completed, the electronic device 101 and the external display device 201 may be connected to each other. The electronic device 101 may transmit the iris authentication result (e.g., authentication success) to the external display device 201, and the external display device 201 may allow connection with the electronic device 101 upon receiving the iris authentication result. there is.

FIG. 11 is a flowchart 1100 illustrating an iris authentication method of an electronic device according to an embodiment. 11 may be performed before operation 713 of FIG. 7 or may be performed included in operation 713 of FIG. 7 .

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 1101 to 1109 may be understood as being performed by a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1).

Referring to FIG. 11, in operation 1101, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1) according to an embodiment operates an external display device (e.g., FIG. A communication channel with the external display device 201 of 2 can be formed. The processor 120 may form a communication channel with the external display device 201 through a communication module (e.g., the communication module 190 of FIG. 1) before connection (e.g., pairing) with the external display device 201. The communication channel may be formed to exchange iris information for iris authentication.

In operation 1103, the processor 120 may identify iris information corresponding to face information. The processor 120 may identify iris information stored in a secure area of a memory (eg, memory 130 of FIG. 1). The processor 120 may identify iris information corresponding to face information that successfully face authentication while the user is wearing the external display device 201. The processor 120 may request an iris image obtained by photographing the user's iris from the external display device 201.

In operation 1105, the processor 120 may receive an iris image from the external display device 201 through the communication module 190. The processor 120 may receive an iris image obtained by photographing the user's iris according to the request.

At operation 1107, processor 120 may authenticate the iris. The processor 120 may analyze the received iris image and extract iris feature information. The processor 120 may authenticate the user's iris by comparing iris characteristic information extracted from the iris image with iris information stored in the secure area of the memory 130. If the iris characteristic information substantially matches the stored iris information, the processor 120 may determine that the user's iris authentication has succeeded. If the iris characteristic information does not substantially match the stored iris information, the processor 120 may determine that the user's iris authentication has failed.

If the iris authentication succeeds, the processor 120 may perform operation 1109, and if the iris authentication fails, the processor 120 may return to operation 1105. The processor 120 returns to operation 1105 to request an iris image obtained by re-photographing the user's iris from the external display device 201, and may receive an iris image obtained by re-photographing the user's iris according to the request.

If the iris authentication is successful, the processor 120 may complete the connection with the external display device 201 in operation 1109. If both face authentication (eg, operation 705 of FIG. 7 ) and iris authentication (eg, operation 1107 ) are successful, the processor 120 may complete the connection with the external display device 201 .

FIG. 12 is a flowchart 1200 illustrating an iris authentication method of an external display device according to an embodiment. 12 may be performed before operation 811 of FIG. 8 or may be performed included in operation 811 of FIG. 8.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 1201 to 1209 may be understood as being performed by a processor (e.g., processor 390 of FIG. 3) of an electronic device (e.g., external display device 201 of FIG. 2).

Referring to FIG. 12, in operation 1201, a processor (e.g., processor 390 of FIG. 3) of an external display device (e.g., external display device 201 of FIG. 2) according to an embodiment operates an electronic device (e.g., processor 390 of FIG. 3). A communication channel may be formed with the electronic device 101 of FIG. 1. The processor 390 may request a connection (e.g., establish a communication channel) with the electronic device 101 using the SSID of the electronic device 101 through a communication module (e.g., the communication module 340 of FIG. 3). The processor 390 may form a communication channel with the electronic device 101 that responds to the request before connection (e.g., pairing) with the electronic device 101. The communication channel may be formed to exchange iris information for iris authentication.

In operation 1203, the processor 390 may capture the user's iris. The processor 390 may photograph the user's iris using a gaze tracking camera (e.g., the camera module 370 of FIG. 3) to check the direction of the user's gaze. The processor 390 may obtain an iris image of the user by photographing the user's iris.

In operation 1205, the processor 390 may receive iris information from the electronic device 101 through a communication module (eg, the communication module 370 of FIG. 3). The iris information is stored in a secure area of the electronic device 101 and may be identified in correspondence with face information.

At operation 1207, processor 390 may authenticate the iris. The processor 390 may analyze the acquired iris image and extract iris feature information. The processor 390 may authenticate the user's iris by comparing iris feature information extracted from the iris image with iris information received at all times. If the iris characteristic information substantially matches the received iris information, the processor 390 may determine that the user's iris authentication has succeeded. If the iris characteristic information does not substantially match the received iris information, the processor 390 may determine that the user's iris authentication has failed.

If the iris authentication succeeds, the processor 390 may perform operation 1209, and if the iris authentication fails, the processor 390 may return to operation 1203. The processor 390 may return to operation 1203 to re-photograph the user's iris and obtain the captured iris image.

If the iris authentication is successful, the processor 390 may complete the connection with the electronic device 101 in operation 1209. If face authentication (e.g., operation 705 of FIG. 7), authentication code authentication (e.g., operation 805 of FIG. 8), and iris authentication (e.g., operation 1207) are all successful, the processor 390 The connection can be completed.

A method of operating an electronic device according to embodiments of the present disclosure includes acquiring a face image of a user through a camera module of the electronic device while wearing the external display device based on a connection request with the external display device; An operation of authenticating a face by comparing the obtained facial image with facial information stored in the memory of the electronic device. When the face authentication is completed, an authentication code associated with an external display device associated with the stored facial information is displayed on the display of the electronic device. An operation of displaying, an operation of receiving a request for a communication connection with the electronic device from the external display device through a communication module of the electronic device based on the authentication result of the authentication code on the external display device, and responding to the authenticated face information This may include an operation of performing iris authentication by comparing iris information stored in the security area of the memory, and an operation of connecting to the external display device when the iris authentication is completed.

In the security area of the memory, the user's face information and the user's iris information may be stored in correspondence with the device information of the external display device.

The authentication code may be obtained by encrypting the SSID of the external display device and the SSID of the electronic device corresponding to the stored face information with the public key of the external display device.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "secondary", or "first" or "second" may be used simply to distinguish one element from another and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be “coupled” or “connected” to another (e.g. second) component, with or without the terms “functionally” or “communicatively”. Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play Store ^TM ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

The various embodiments of the present invention disclosed in the specification and drawings are merely provided as specific examples to easily explain the technical content of the present invention and to facilitate understanding of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be construed as including all changes or modified forms derived based on the technical idea of the present invention in addition to the embodiments disclosed herein.

101: Electronic devices
120: processor
130: memory
201: external display device

Claims (20)

In electronic devices,
camera module;
communication module;
display;
Memory; and
A processor operatively connected to at least one of the camera module, the communication module, the display, and the memory, the processor comprising:
Based on a connection request with an external display device, obtain an image of the user's face through the camera module while wearing the external display device,
Face authentication is performed by comparing the acquired facial image with facial information stored in the memory,
When the face authentication is completed, displaying an authentication code associated with an external display device associated with the stored face information on the display,
Receive a request for a communication connection with the electronic device from the external display device through the communication module based on the authentication result of the authentication code in the external display device,
An electronic device configured to connect with the external display device according to the request.
The method of claim 1, wherein in the security area of the memory,
An electronic device wherein user's face information and user's iris information are stored in response to device information of the external display device.
The method of claim 2, wherein the processor:
Analyzing the acquired facial image to extract facial feature information,
An electronic device configured to determine whether the extracted facial feature information substantially matches facial information stored in the security area.
The method of claim 3, wherein the processor:
If the extracted facial feature information substantially matches the facial information stored in the security area, it is determined that the face authentication is successful,
An electronic device configured to determine that face authentication has failed when the extracted facial feature information does not substantially match face information stored in the secure area.
The method of claim 1, wherein the processor:
An electronic device configured to generate the authentication code based on device information of the external display device stored in a secure area of the memory in response to the stored face information.
The method of claim 1, wherein the authentication code is:
An electronic device, wherein the service set identifier (SSID) of the external display device and the SSID of the electronic device corresponding to the stored face information are encrypted with the public key of the external display device.
The method of claim 1, wherein the processor:
An electronic device configured so that the external display device authenticates the displayed authentication code and receives a communication connection request from the external display device with the electronic device through the SSID of the electronic device included in the authentication code.
The method of claim 1, wherein the processor:
Identifying iris information stored in a secure area of the memory in response to the stored face information,
Receiving an iris image from the external display device through a communication channel established with the external display device,
An electronic device configured to perform iris authentication by comparing the identified iris information with the received iris image.
The method of claim 8, wherein the processor:
If the iris feature information extracted from the iris image and the identified iris information substantially match, it is determined that the user's iris authentication was successful,
An electronic device configured to complete connection with the external display device when the iris authentication is successful.
The method of claim 8, wherein the processor:
If the iris feature information extracted from the iris image and the identified iris information do not substantially match, it is determined that the user's iris authentication has failed,
If the iris authentication fails, request an iris image obtained by rephotographing the user's iris from the external display device,
An electronic device configured to receive an iris image obtained by rephotographing the user's iris according to the request and re-perform iris authentication.
In electronic devices,
camera module
communication module;
display;
Memory; and
A processor operatively connected to at least one of the camera module, the communication module, the display, and the memory, the processor comprising:
Detect whether the electronic device is worn by the user,
Photographing an authentication code displayed on a display of an external electronic device through the camera module while the electronic device is worn by the user,
Authenticate the authentication code and extract device information of the external electronic device from the authentication code,
Transmitting a connection request with the external electronic device through the communication module based on the extracted device information,
An electronic device configured to connect with the external electronic device that responds to the request.
The method of claim 11, wherein the processor:
An electronic device configured to photograph the authentication code when the user's face is successfully authenticated by the external electronic device while the user is wearing the electronic device.
The method of claim 12, wherein the authentication code is:
An electronic device wherein the service set identifier (SSID) of the electronic device and the SSID of the external electronic device corresponding to the face information of the authenticated user are encrypted with the public key of the electronic device.
The method of claim 13, wherein the processor:
An electronic device configured to decrypt the authentication code with the private key of the electronic device and extract the SSID of the electronic device and the SSID of the external electronic device from the authentication code.
15. The method of claim 14, wherein the processor:
When the SSID of the electronic device extracted from the authentication code matches the SSID stored in the memory, an electronic device configured to request a communication connection to the external electronic device through the SSID of the external electronic device extracted from the authentication code.
The method of claim 11, wherein the processor:
Obtain an iris image by photographing the user's iris through the camera module,
Receive iris information corresponding to the face information of the authenticated user from the external electronic device,
An electronic device configured to perform iris authentication by comparing the acquired iris image and the received iris information.
17. The method of claim 16, wherein the processor:
If the iris feature information extracted from the iris image and the received iris information substantially match, it is determined that the user's iris authentication was successful,
An electronic device configured to complete connection with the external electronic device when the iris authentication is successful.
17. The method of claim 16, wherein the processor:
If the iris feature information extracted from the iris image and the received iris information do not substantially match, it is determined that the user's iris authentication has failed,
If the iris authentication fails, obtain an iris image by rephotographing the user's iris,
An electronic device configured to re-perform iris authentication by comparing the acquired iris image and the received iris information.
In a method of operating an electronic device,
An operation of acquiring a face image of a user through a camera module of the electronic device while wearing the external display device based on a connection request with the external display device;
face authentication by comparing the acquired facial image with facial information stored in the memory of the electronic device;
When the face authentication is completed, displaying an authentication code associated with an external display device associated with the stored face information on a display of the electronic device;
An operation of receiving a request for a communication connection with the electronic device from the external display device through a communication module of the electronic device based on an authentication result of the authentication code in the external display device;
An operation of performing iris authentication by comparing iris information stored in a secure area of the memory in response to the authenticated face information; and
When the iris authentication is completed, the method includes connecting to the external display device.
According to clause 19,
In the security area of the memory, the user's face information and the user's iris information are stored in correspondence with the device information of the external display device,
The authentication code is a service set identifier (SSID) of the external display device corresponding to the stored face information and the SSID of the electronic device are encrypted with a public key of the external display device.

Images