KR20240043634A - Wearable electronic devices providing video call and operating method of wearable electronic device - Google Patents

Abstract

A wearable electronic device according to an embodiment may include a camera module that acquires real-world images around a user wearing the wearable electronic device in response to execution of an application for a video call. A wearable electronic device detects a reflector from a real-world image, calculates the location information of the reflector including the distance and direction from the image of the area corresponding to the reflector in the real-world image, and moves the user to the location of the reflector according to the location information. By controlling the output device to provide guidance, it may include a processor that generates a video for the user's call to be transmitted to the other party of the video call. A wearable electronic device may include a communication module that transmits video for a user's call.

Description

Wearable electronic device providing video call and operating method of wearable electronic device {WEARABLE ELECTRONIC DEVICES PROVIDING VIDEO CALL AND OPERATING METHOD OF WEARABLE ELECTRONIC DEVICE}

The disclosure below relates to a wearable electronic device that provides video calling and a method of operating the wearable electronic device.

Various wearable electronic devices, such as virtual reality (VR) devices, augmented reality devices, and/or mixed reality (MR) devices, are being commercialized. Wearable electronic devices are worn on the user's head or eyes, for example, a head mounted display (HMD), and therefore include an external lens that allows the user to view objects nearby or capture objects facing them. It may take the form of sunglasses or goggles.

According to one embodiment, in response to execution of an application for a video call on the wearable electronic device, the wearable electronic device includes a camera module for acquiring a real-world image around the user, detecting a reflector from the real-world image, and By calculating location information of the reflector including distance and direction from the image of the area corresponding to the reflector, and controlling the output device to provide guidance for the user to move to the location of the reflector according to the location information, It may include a processor that generates a video for the user's call to be transmitted to the other party of the video call, and a communication module that transmits the video for the user's call.

According to one embodiment, a method of operating a wearable electronic device includes acquiring a real-world image around the user in response to execution of an application for video calling on the wearable electronic device, detecting a reflector from the real-world image, and detecting a reflector from the real-world image. An operation of calculating location information of the reflector including distance and direction from an image of an area corresponding to the reflector in an image, and an output device configured to provide guidance for the user to move to the location of the reflector according to the location information. By controlling , it may include an operation of generating a video for the user's call to be transmitted to the other party of the video call.

FIG. 1 is a block diagram illustrating the configuration of an electronic device in a network environment according to an embodiment.
Figure 2 is a block diagram illustrating a program according to one embodiment.
Figure 3 is a block diagram of a wearable electronic device according to an embodiment.
FIG. 4 is a diagram illustrating the configuration of a video call application running on a wearable electronic device according to an embodiment.
FIG. 5 is a diagram illustrating a process for detecting a reflector from a real-world image according to an embodiment.
Figure 6 is a diagram for explaining the operation of a video stabilization module of a video call application according to an embodiment.
Figure 7 is a diagram for explaining the operation of a face reproduction module of a video call application according to an embodiment.
FIG. 8 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment.
FIG. 9 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment.
FIG. 10 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment.
Figure 11 is a flowchart showing a method of operating a wearable electronic device according to an embodiment.
Figure 12 is a flowchart showing a method of calculating location information from an image of an area corresponding to a reflector reflected on a reflector according to an embodiment.
Figure 13 is a flowchart showing a method of generating a video for a user's call sent to the other party of a video call, according to an embodiment.
Figure 14 is a flowchart showing a method of operating a wearable electronic device according to an embodiment.
FIG. 15 is a diagram illustrating a method of performing a video call between a sender terminal and a recipient terminal using a video call application according to an embodiment.
FIG. 16 is a diagram illustrating a method of performing a video call between a sender terminal and a recipient terminal using a video call application according to an embodiment.
FIG. 17 is a diagram illustrating a method of performing a video call between a sender terminal on which a video call application is installed and a recipient terminal on which another video call application is installed, according to an embodiment.
FIG. 18 is a diagram illustrating a method of performing a video call between a recipient terminal on which a video call application is installed and a sender terminal on which a voice call application is installed, according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

FIG. 1 is a block diagram illustrating the configuration of an electronic device in a network environment according to an embodiment. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted, or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Figure 2 is a block diagram 200 illustrating program 140 according to one embodiment. According to one embodiment, the program 140 includes an operating system 142, middleware 144, or an application 146 executable on the operating system 142 for controlling one or more resources of the electronic device 101. It can be included. Operating system 142 may include, for example, AndroidTM, iOSTM, WindowsTM, SymbianTM, TizenTM, or BadaTM. At least some of the programs 140 are preloaded into the electronic device 101, for example, at the time of manufacture, or are stored in an external electronic device (e.g., the electronic device 102 or 104, or a server) when used by a user. It can be downloaded or updated from 108)).

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, process, memory, or power) of the electronic device 101 . Operating system 142 may additionally or alternatively operate on other hardware devices of electronic device 101, such as input module 150, audio output module 155, display module 160, and audio module 170. , sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or It may include one or more driver programs for driving the antenna module 197.

The middleware 144 may provide various functions to the application 146 so that functions or information provided from one or more resources of the electronic device 101 can be used by the application 146. The middleware 144 includes, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213. ), connectivity manager (215), notification manager (217), location manager (219), graphics manager (221), security manager (223), call manager (225), or voice recognition manager (227). You can.

The application manager 201 may, for example, manage the life cycle of the application 146. The window manager 203 may, for example, manage one or more GUI resources used on the screen. For example, the multimedia manager 205 identifies one or more formats required for playing media files, and encodes or decodes the corresponding media file using a codec suitable for the selected format. It can be done. The resource manager 207 may, for example, manage the source code of the application 146 or the memory space of the memory 130. The power manager 209 manages, for example, the capacity, temperature, or power of the battery 189, and may use this information to determine or provide related information necessary for the operation of the electronic device 101. . According to one embodiment, the power manager 209 may interface with a basic input/output system (BIOS) (not shown) of the electronic device 101.

Database manager 211 may create, search, or change a database to be used by application 146, for example. The package manager 213 may, for example, manage the installation or update of applications distributed in the form of package files. The connectivity manager 215 may manage, for example, a wireless connection or direct connection between the electronic device 101 and an external electronic device. For example, the notification manager 217 may provide a function for notifying the user of the occurrence of a designated event (eg, an incoming call, message, or alarm). The location manager 219 may, for example, manage location information of the electronic device 101. The graphics manager 221 may, for example, manage one or more graphic effects to be provided to the user or a user interface related thereto.

Security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101. The call manager 225 can manage functions related to a video call application, which will be described later.

For example, the voice recognition manager 227 transmits the user's voice information to the server 108 and provides a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice information, Alternatively, text data converted based at least in part on the voice information may be received from the server 108. According to one embodiment, the middleware 244 may dynamically delete some existing components or add new components. According to one embodiment, at least a portion of the middleware 144 may be included as part of the operating system 142 or may be implemented as separate software different from the operating system 142.

The application 146 includes, for example, home 251, dialer 253, SMS/MMS (255), instant message (IM) 257, browser 259, camera 261, and alarm 263. , Contacts (265), Voice Recognition (267), Email (269), Calendar (271), Media Player (273), Album (275), Watch (277), Health (279) (such as exercise amount or blood sugar) It may include applications that measure biometric information) or environmental information 281 (e.g., measure atmospheric pressure, humidity, or temperature information). According to one embodiment, the application 146 may further include an information exchange application (not shown) that can support information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to deliver designated information (e.g., calls, messages, or alarms) to an external electronic device, or a device management application configured to manage the external electronic device. there is. The notification relay application, for example, transmits notification information corresponding to a specified event (e.g., mail reception) generated in another application (e.g., email application 269) of the electronic device 101 to an external electronic device. You can. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide it to the user of the electronic device 101.

The device management application, for example, controls the power (e.g., turn-on or turn-off) of an external electronic device or some component thereof (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device 101. ) or functions (such as brightness, resolution, or focus). A device management application may additionally or alternatively support installation, deletion, or update of applications running on external electronic devices. In addition, the application 146 may further include a video call application, which will be described later.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable electronic devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be "coupled" or "connected" to another (e.g. second) component, with or without the terms "functionally" or "communicatively". Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document are stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101 of FIG. 1). It may be implemented as software (e.g., program 140) including one or more instructions. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Figure 3 is a block diagram of a wearable electronic device according to an embodiment. Referring to FIG. 3, a wearable electronic device 300 (e.g., electronic device 101 of FIG. 1) according to an embodiment includes a camera module 310 (e.g., camera module 180 of FIG. 1) and a processor ( 330) (e.g., the processor 120 of FIG. 1), and a communication module 350 (e.g., the communication module 190 of FIG. 1). The wearable electronic device 300 may further include an input module 370 (e.g., input module 150 of FIG. 1) and a memory 390 (e.g., memory 130 of FIG. 1).

The wearable electronic device 300 may correspond to an electronic device that a user can wear on the head or face, such as a head mounted display (HMD), but is not necessarily limited thereto. Head-mounted displays can make users feel virtual images as if they were real through vivid images, videos, and voices.

In response to the execution of an application for a video call (e.g., the video call application 400 of FIG. 4) on the wearable electronic device 300, the camera module 310 captures real-world images around the user (e.g., the real-world image of FIG. 5). Obtain (510, 530, 550)). For example, the user may be a user wearing a wearable electronic device 300 on which the video call application 400 is installed. Real-world images 510, 530, and 550 may include, for example, various objects (eg, a user and/or the environment around the user) included within the range of the lens field of view of the camera module 310.

The processor 330 may detect reflectors from real-world images 510, 530, and 550 acquired through the camera module 310. A 'reflector' is an object that receives and reflects light, and may correspond to an object that has a reflectivity above a certain level that can reflect the image of the object reflected on the reflective surface. Reflectors are, for example, images of surrounding objects, such as surrounding objects or scenery, such as reflectors (e.g., helmets, mirrors, and/or glass) included in the real-world images 510, 530, and 550 of FIG. 5 below. It may correspond to a smooth surface, curved surface, shiny metal, glass, display screen, and/or mirror that reflects the image, and there may be no restrictions on its material or shape.

For example, when no reflector is detected from the real-world images 510, 530, and 550, the processor 330 may perform a video call using the user's call video generated in response to the user's voice information. A method by which the processor 330 generates an image for a user's call in response to the user's voice information will be described later with reference to the voice-face feature conversion module 451 of FIG. 4 below.

When a reflector is detected from a real-world image, the processor 330 may calculate the position information of the reflector including the distance and direction from the image of the area corresponding to the reflector among the real-world images 510, 530, and 550. For example, the processor 330 may check the direction in which the reflector is located from the reflector detected in the real-world images 510, 530, and 550. Additionally, the processor 330 may check the distance to the reflector based on, for example, an image sensor or a time of flight (ToF) sensor. ToF sensors can emit infrared rays that are reflected back from objects (e.g. reflectors). The processor 330 can measure the distance to an object by the difference between the time when infrared rays are emitted from the ToF sensor and the time when the infrared rays are reflected by the object (reflector) and returned to the sensor.

Hereinafter, 'image of the area corresponding to the reflector' may correspond to an image captured by the camera module 310 of the image reflected on the reflector. For example, the image of the area corresponding to the reflector may include not only the user but also object(s) around the user.

When a reflector is detected from the real-world images 510, 530, and 550, the processor 330 may determine whether the image of the area corresponding to the reflector includes at least a part of the user's body, including the face. For example, the processor 330 may use the face recognition module 420 and/or the body recognition module 430 of the video call application 400, which will be described later with reference to FIG. 4 below, to display the image of the area corresponding to the reflector to the user. It may be determined whether the user's face and/or at least a body part is included.

For example, the processor 330 determines that the real-world images 510, 530, and 550 captured by the camera module 310 by the face recognition module 420 and/or the image of the area corresponding to the reflector include the user's face. You can decide whether to do it or not. If it is determined that the real-world images 510, 530, and 550 do not contain the user's face, the processor 330 processes the user's voice information in response to the user's face until the user's face is detected from the real-world images 510, 530, and 550. You can make a video call using the user's call video you created.

The processor 330 determines that the image of the area corresponding to the reflector includes at least a part of the user's body, including the face, and displays real-world images 510, 530, and 550 acquired using the camera module 310. Alternatively, an image for a user's call can be created using the image of the area corresponding to the user's reflector.

Upon determining that the image of the area corresponding to the reflector does not include at least a body part of the user, the processor 330 detects the reflector based on the user's position in the three-dimensional space, the viewing angle of the camera module 310, and the position of the reflector. The position information of the reflector can be calculated from the image of the area corresponding to . The viewing angle of the camera module 310 may include, for example, 6 degrees of freedom (DoF). For example, when the relative position of a point light source changes in the image of the area corresponding to the reflector, the processor 330 estimates 3D information using the change in relative position to provide position information of the reflector including distance and direction. It can be calculated. At this time, initial values such as calibration information and focal length of the camera module 310 may be given in advance.

The processor 330 controls an output device (e.g., the sound output module 155 and/or the display module 160 in FIG. 1) to provide guidance for the user to move to the location of the reflector according to the location information, thereby displaying the image. You can create a video for your call to be transmitted to the other party on the call. Here, the 'position of the reflector' may correspond to a position within a range where the user's image can be reflected on the reflector. For example, if a reflector is detected from the real-world images 510, 530, and 550, but the user is not detected in the image of the area corresponding to the reflector where the object reflected in the reflector is captured, the processor 330 may detect the object reflected in the reflector according to the location information. The user can be guided to move to a location where the user's appearance can be reflected in the reflector so that the image of the area corresponding to the reflector includes the user's face or body part. Guidance may be provided in voice form, for example, through an audio module (e.g., the sound output module 155 of FIG. 1), or through a display module of the wearable electronic device 300 (e.g., the display module of FIG. 1). It may be provided in text form on the screen of 160)), but is not necessarily limited to this. Instructions may be, for example, “Please move to the right at 11 o’clock towards the mirror 1 meter away.” Alternatively, you can include phrases that guide the direction and distance of movement to the reflector, such as "Please take a step to the left of the mirror."

The processor 330 may acquire an image of the area corresponding to the reflector where the user who moved to the location of the reflector according to the guidance captured the image reflected on the reflector. The processor 330 may generate an image for a user's call using the image of the area corresponding to the reflector. For example, the processor 330 may use the image of the area corresponding to the reflector and/or the real world image as is for the user's call, depending on the user's settings, or the image of the area corresponding to the reflector and/or the real world may be used as the image for the user's call. The reconstructed image reconstructed using the video can also be used as the video for the user's call.

The processor 330 may, for example, determine the user's selection of how to present the user's video for the call to be transmitted to the other party of the video call, and/or reflect the user's face and at least a portion of the user's body by the camera module 310. Depending on whether the video of the corresponding area is acquired, the video for the user's call to be transmitted to the other party of the video call can be generated differently. The processor 330, for example, receives the user's selection of how to express the video for the call through a user interface (UI) provided by the input module 370 and generates the video for the call. Alternatively, a video for a call can be created according to the initial settings. The user's selection is, for example, a first choice of determining the image of the area corresponding to the reflector as the image for the user's call, and determining the reconstructed image reconstructed using the image of the area corresponding to the reflector as the image for the user's call. It may include at least one of a second selection, and a third selection of determining a graphic object reconstructed using the image of the area corresponding to the reflector as the image for the user's call, but is not necessarily limited thereto. For example, the user may set the real world images 510, 530, and 550 to be used as images for the user's call. Here, a 'graphic object' may correspond to, for example, an object that visualizes and expresses various types of data. Graphic objects may include, for example, avatars resembling the user's face and/or body, animals, dolls, characters such as cartoon characters, and/or dynamic emojis, but are not necessarily limited thereto.

For example, when the first selection is input to the input module 370, the processor 330 may determine the image of the area corresponding to the reflector captured by the camera module 310 as the image for the user's call. .

When the second selection is input to the input module 370, the processor 330 may generate a reconstructed image in which the user's appearance is reconstructed using the image of the area corresponding to the reflector. For example, the processor 330 compares the image of the area corresponding to the reflector with the user's image previously stored in the memory 390 ('pre-stored image'), and determines which of the pre-stored images the image of the area corresponding to the reflector is. You can check whether it corresponds to the part. The processor 330 may update ('direct update') the portion corresponding to the image of the area corresponding to the reflector in the previously stored image by the image of the area corresponding to the reflector. Hereinafter, 'direct update' may refer to an update performed by directly using the image of the area corresponding to the reflector. The processor 330 selects the portion that does not correspond to the image of the area corresponding to the reflector in the pre-stored image by, for example, an image predicted by a neural network or an image warped by the image of the area corresponding to the reflector. A reconstructed image can be created by updating ('indirect update'). Hereinafter, 'indirect update' may refer to an update performed using an image generated based on the image of the area corresponding to the reflector, rather than directly using the image of the area corresponding to the reflector. The processor 330 may determine the reconstructed image as the image for the user's call. At this time, the 'reconstructed image' may correspond to an image reconstructed so that, for example, the entire face of the user, or part of the user's body, such as the entire upper body of the user, is visible without being obscured.

When the third selection is input to the input module 370, the processor 330 may generate a graphic object corresponding to the image or reconstructed image of the area corresponding to the reflector. For example, when the user's face and/or the user's body included in the image of the area corresponding to the reflector are all visible without any obscured parts, the processor 330 generates a graphic object corresponding to the image of the area corresponding to the reflector. can do. On the other hand, when the user's face and/or part of the user's body included in the image of the area corresponding to the reflector is occluded, the processor 330 determines that the user's appearance is occluded by the image of the area corresponding to the reflector. A graphic object corresponding to the reconstructed image can be created so that it appears without it. The processor 330 may determine the graphic object as an image for a user's call. A method by which the processor 330 generates a reconstructed image will be described later with reference to the image stabilization module 440 of FIG. 4 below. Additionally, the method by which the processor 330 creates a graphic object will be described later with reference to the avatar creation module 450 of FIG. 4 below.

The communication module 350 may transmit the user's call video generated by the processor 330 to the terminal of the other party of the call ('receiver'). Additionally, the communication module 350 may receive a video for a call of the recipient transmitted from the terminal of the other party to the call, and/or a video for the call of the sender generated from the terminal of the other party to the call.

The input module 370 may receive the user's selection of a video for a call. For example, when a video call application, which will be described later with reference to FIG. 4 , is installed on a wearable electronic device, the input module 370 may receive the user's selection of which image to use as the call video as an initial setting. Alternatively, the input module 370 may receive the user's selection of which video to use as the video call when the video call application is running.

The memory 390 may include the user's pre-stored images. The memory 390 may store at least one program including a voice call application 400, which will be described later. Additionally, the memory 390 can store various information generated during processing by the processor 330. In addition, the memory 390 can store various data and programs. The memory 390 may include volatile memory (e.g., volatile memory 132 of FIG. 1) or non-volatile memory (e.g., non-volatile memory 134 of FIG. 1). The memory 390 may be equipped with a high-capacity storage medium such as a hard disk to store various data.

Additionally, the processor 330 may perform at least one method or a technique corresponding to at least one method related to the wearable electronic device 300 described with reference to FIGS. 1 to 18 . The processor 330 may be an electronic device implemented as hardware having a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program. For example, the wearable electronic device 300 implemented as hardware includes a microprocessor, a central processing unit (CPU), a graphic processing unit (GPU), a processor core, It may include a multi-core processor, multiprocessor, application-specific integrated circuit (ASIC), field programmable gate array (FPGA), and/or neural processing unit (NPU).

FIG. 4 is a diagram illustrating the configuration of a video call application running on a wearable electronic device according to an embodiment. Referring to FIG. 4, a video call application (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) installed and/or driven according to an embodiment. 400 may include modules such as a reflector detection module 410, a face recognition module 420, a body recognition module 430, an image stabilization module 440, and an avatar creation module 450. The operations of the modules of the video call application 400 may be performed by the processor 330 of the wearable electronic device 300.

The reflector detection module 410 detects images acquired through a camera module (e.g., the camera module 180 of FIG. 1 and/or the camera module 310 of FIG. 3) included in the wearable electronic device 300 (e.g., the user The reflector area including the reflector can be obtained from the surrounding real-world image, and the reflector can be detected in the reflector area. The reflector detection module 410 may detect reflectors using publicly available techniques for detecting reflectors (e.g., “Where Is My Mirror?”, Yang et al., 2019), but is not necessarily limited thereto. No. An image of an area corresponding to the reflector detected by the reflector detection module 410 may be captured by the camera module 310. The method by which the reflector detection module 410 detects a reflector will be described with reference to FIG. 5 below.

The face recognition module 420 may recognize the user's face reflected in the reflector area detected by the reflector detection module 410. The face recognition module 420 detects a face included in an image captured by the camera module 310 (e.g., an image of an area corresponding to a reflector) and detects features such as eyes, eyebrows, nose, and/or mouth in the face. Landmarks can be detected. The face recognition module 420 may recognize whether the face detected in the reflector area is the user's face by extracting feature points from landmarks and comparing whether the extracted feature points match pre-stored user facial feature points.

The body recognition module 430 may recognize at least part of the body excluding the user's face reflected in the reflector area detected by the reflector detection module 410. The body recognition module 430 may recognize and detect at least a body part included in an image captured by the camera module 310 (eg, an image of an area corresponding to a reflector). The body recognition module 430 may detect at least a part of the user's body included in the captured image in the same manner as the face recognition module 420.

The detection results of each of the face recognition module 420 and the body recognition module 430 may be coordinate values of a rectangle (eg, bounding box) including the face and at least a part of the body. Various publicly available face and/or body recognition technologies may be used to recognize the target object (e.g., face, body) of each of the face recognition module 420 and the body recognition module 430.

According to one embodiment, the face recognition module 420 and/or the body recognition module 430 recognizes the face and/or the face at the occluded location based on the already acquired face or body part even if the face or body part is obscured for a certain period of time. Alternatively, an image of the body can be estimated.

The image stabilization module 440 may generate a stabilized image to be transmitted to the other party of the video call based on the image of the area corresponding to the reflector obtained from the reflector area. Here, 'stabilized image' can be understood as, for example, an image reconstructed by resolving not only the tremor or blur of the image, but also the occlusion of part of the image through various image processing techniques.

For example, the image stabilization module 440 provides video for a user's call in response to shaking of the image or occlusion of the face and/or body by a reflector based on the recognition results of the image, face, and/or body. The user's appearance to be displayed can be reliably reconstructed. For example, when the user's face and/or part of the body included in the image of the area corresponding to the reflector is obscured, the image stabilization module 440 uses the previously stored image and the captured image together to show the user's face and/or body. Alternatively, the image can be reconstructed so that all hidden parts of the body are visible. A more detailed operation of the image stabilization module 440 will be described with reference to FIG. 6 below.

The avatar creation module 450 receives the user's voice signal and/or the image acquired as a result of the image stabilization module 440 (e.g., face image, body image, and/or reconstructed image) and creates various avatars, such as an avatar. It can be converted into a graphic object in the form of a graphic object. Additionally, the avatar creation module 450 may manipulate pre-stored photos, videos, two-dimensional avatar objects, and/or three-dimensional avatar objects based on various input signals, for example. At this time, the photo, video, 2D avatar object, and/or 3D avatar object may be stored, for example, in the memory 390 of the wearable electronic device 300, a PC connected to the wearable electronic device 300, Or it may be stored on a cloud server.

The avatar creation module 450 may include several sub-modules. Sub-modules include, for example, a voice-face feature conversion module (A) 451, a facial feature point extraction module (B) 453, a body feature point extraction module (C) 455, and a face reproduction module (D) 457. ), and/or a feature point-based avatar manipulation module (E) 459.

The voice-face feature point conversion module (A) 451 can predict the facial feature points that make up the face from the voice signal. For example, the voice-face feature point conversion module (A) 451 records a voice signal, simultaneously captures a face image corresponding to the voice signal, and extracts and generates feature points of the face image corresponding to the voice signal. You can use a deep learning model learned by training data. The voice-face feature point conversion module (A) 451 can acquire facial feature points corresponding to the voice signal using a learned deep learning model. For example, when a voice signal (“let's go”) is input to the learned deep learning model, the deep learning model can extract and output feature points of the face image that utters “let’s go.”

The facial feature point extraction module (B) 453 can extract facial feature points from a face image. The facial feature extraction module (B) 453 is, for example, active appearance models (AAM) (Cootes et al., 1998), 3D morphable models (3DMM) (Blanz and Vetter, 2003), Kazemi (Kazemi and Sullivan) , facial feature points can be extracted from face images by using well-known feature extraction algorithms such as deep learning-based Subpixel Heatmap Regression, AnchorFace, and PFLD (Practical Facial Landmark Detector), or by deep learning-based feature extraction algorithms. It is not necessarily limited to this.

The body feature point extraction module (C) 455 can extract body feature points from the body image. For example, the body feature point extraction module (C) 455 may extract feature points from a body image according to structural proportional information of the body identified using a human body database, but is not necessarily limited to this.

The face reproduction module (D) 457 can convert a user's face image into another person's face through a deep learning network. The method by which the face reproduction module (D) 457 converts a user's face image into another person's face will be described with reference to FIG. 7 below.

The feature point-based avatar manipulation module (E) 459 can manipulate a two-dimensional avatar object or a three-dimensional avatar object based on facial feature points and/or body feature points. The feature point-based avatar manipulation module (E) 459 can change the shape of the avatar by using facial feature points and/or body feature points as input. The feature point-based avatar manipulation module (E) 459 may use facial feature points obtained based on voice signals to change the shape of the avatar.

The processor 330 according to one embodiment may selectively use only some of the sub-modules of the avatar creation module 450. The processor 330 may dynamically determine which sub-module(s) to use for avatar creation, for example, based on the results of the image stabilization module 440 and the user's presets, respectively, or the image stabilization module 440 Based on both the results of and the user's presets, the submodule(s) to be used for avatar creation can be dynamically determined.

For example, as shown in FIG. 8 below, when the image of the user (e.g., the sender) is not reflected in the mirror and only voice information such as a voice signal is transmitted, the processor 330 uses the voice-facial feature point conversion module. The sender's avatar can be implemented using the (A) 451 and the feature point-based avatar manipulation module (E) 459. On the other hand, when the user (e.g., sender) is reflected in the mirror as shown in FIG. 9 below, the processor 330 uses the facial feature point extraction module (B) 453 and the body feature point extraction module (C) ( 455), and the feature point-based avatar manipulation module (E) 459 can be used to implement the sender's avatar.

For example, during a video call, if the sender sets to express himself with a photo of a celebrity he wants instead of his own video, the processor 330 uses the face reproduction module (D) 457 alone The sender's avatar can be implemented.

A user of the video call application 400 according to one embodiment can select a sender display method during a video call. The sender display method may correspond to a user's selectable display method, such as how to show one's appearance to the other party during a video call. For example, the method of displaying the sender may vary depending on the user's data input to the wearable electronic device 300. Each module of the video call application 400 involved depending on the sender display method may also vary. Each module of the video call application 400 used for each sender display method can be organized as shown in [Table 1] below.

The wearable electronic device 300 according to one embodiment may, for example, use an image stabilization module according to whether the user's face and/or the user's body are acquired by the camera module 310, and/or the sender display method selected by the user. It is possible to determine whether to use each of the detailed modules of 440 and the avatar creation module 450. In [Table 1], modules A to E are each a voice-face feature point conversion module 451, a facial feature point extraction module 453, a body feature point extraction module 455, a face reproduction module 457, and a feature point-based avatar manipulation. It may correspond to module 459. In the data section of [Table 1], '○' is displayed for data that is used for each sender display method, 'X' is displayed for data that is not used, and '-' may be displayed if the presence or absence of data is irrelevant. .

For example, if the user wants to use the image captured by the camera module 310 during a video call, the user can select or set 'unprocessed video' as the sender display method. When 'unprocessed video' is selected or set, the wearable electronic device 300 can transmit the video output from the provided camera module 310 to the other party as a video for a call. In this case, as in category a, the wearable electronic device 300 may not use all modules of the video call application 400, including the image stabilization module 440 and the avatar creation module 450.

For example, if the user wants to correct or reconstruct the image captured by the camera module 310 during a video call (e.g., the image of the area corresponding to the reflector reflected in the mirror), the user may use the sender display method. You can select or set ‘stabilized image’. When 'stabilized image' is selected or set, the wearable electronic device 300 corrects the captured image by the image stabilization module 440 and/or creates a reconstructed image ('stabilized image') that reconstructs the user's complete appearance reflected in the mirror. 'Video') can be transmitted to the other party as a video for a call.

At this time, as in category b, when the user's face is recognized in the image of the area corresponding to the reflector, the wearable electronic device 300 sends a stabilized image including the entire appearance of the user's face (or upper body) for a call. You can send it to the other person as a video. Additionally, as in category c, when the user's body is recognized in the image in the area corresponding to the reflector, the wearable electronic device 300 can transmit a stabilized image including the user's entire body to the other party as a call video.

Additionally, if the user wishes to use a graphic object (e.g., an avatar) instead of the image captured by the camera module during a video call, the user must use one of the feature point-based avatar generation method and the reproduction-based avatar generation method as a sender display method. You can select or set.

For example, when the 'feature point-based avatar' creation method is selected or set, the wearable electronic device 300 creates a graphic object (Example 2 A 3D avatar object or a 3D avatar object) can be transmitted to the other party as a video for a call.

For example, as in category d, the 'feature point-based avatar' creation method was selected as the sender display method, but there is no captured image of the user's face and body, and only the user's voice can be input. When only the user's voice is input, the facial feature point extraction module (B) 453 and the body feature point extraction module (C) 455 of the video call application 400 do not operate, so the wearable electronic device 300 -The feature point-based avatar manipulation module (E) 459 can be driven based on the facial feature points acquired using the facial feature point conversion module (A) 451.

In one embodiment, when the 'feature point-based avatar' creation method is selected as the sender display method, the avatar conversion module 450 generates the feature points required for the feature point-based avatar manipulation module (E) 459 according to the type of input data. It can be provided from other modules.

For example, as in segment e or segment f, the user's face or an image (or video) including the user's face and body may be captured. Facial feature points and/or body feature points may be extracted from the facial feature point extraction module (B) 453 and/or body feature point extraction module (C) 455. The feature point-based avatar manipulation module (E) 459 may calculate the movement of the avatar based on the extracted facial feature points and/or body feature points. At this time, classification e or f may be dynamically determined depending on whether facial feature points are extracted from the captured image or whether facial feature points and body feature points are extracted together.

Or, for example, as in segment g, an image containing the user's body excluding the face may be captured, and the user's voice may be input. In this case, there are body feature points extracted from the body feature point extraction module (C) 455, but the user's face is not recognized. The wearable electronic device 300 generates facial feature points from the voice signal by the voice-face feature point conversion module (A) 451, and generates facial feature points based on the facial feature points and body feature points extracted from the body feature point extraction module (C) 455. Thus, the feature point-based avatar manipulation module (E) 459 can be driven.

In addition, as described above, if the user wishes to use a graphic object (e.g., an avatar) instead of the image captured by the camera module 310 during a video call, the user may use a feature point-based avatar creation method as a sender display method and a feature point-based avatar creation method. You can select or set one of the representation-based avatar creation methods.

When the representation-based avatar creation method is selected or set as the sender display method during a video call, the wearable electronic device can express the user's appearance using the avatar face generated by the face reproduction module (D) 457. At this time, if the reproduction-based avatar creation method is selected, the wearable electronic device may generate an avatar based on the face reproduced using different face reproduction methods in the face reproduction module 457, depending on the type of input data.

The modules used together with the face reproduction module (D) 457 by type of input data when creating an avatar in the avatar creation module 450 can be organized as shown in [Table 2] below.

For example, in categories h and k, when face images are not captured and only voice information is input, the face reproduction module (D) 457 can reproduce the user's face by voice-face reproduction method. . The avatar creation module 450 may generate an avatar using the face reproduced by the voice-face reproduction method in the face reproduction module (D) 457.

For example, when a face image is captured, such as in categories i and j, the avatar creation module 450 can recognize the position of the face by extracting facial feature points by the facial feature point extraction module (B) 453. . Additionally, when a face image is captured, as in categories i and j, the face reproduction module (D) 457 may reproduce the user's face based on the image-face reproduction method. The face reproduction module (D) 457 is based on deep learning technology and can apply the user's facial expression to the avatar by using the user's face and a photo or video to be used as an avatar without the process of extracting separate feature points from the face. You can. The avatar creation module 450 may create an avatar using a face reproduced by the face reproduction module (D) 457 using an image-face reproduction method. In addition, when the user's body image is captured, as in categories j and k, the avatar creation module 450 matches the user's facial texture reproduced by the face reproduction module (D) 457 to the head of the 3D avatar model. , and a 3D avatar model can be created and/or manipulated based on the body feature points extracted from the body image by the body feature point extraction module (C) 455. The processor 330 may allow the avatar creation module 450 to manipulate the face and body of the avatar model in different ways.

In addition to the cases described above, the processor 330 may use only some modules of the avatar creation module 450 to suit various situations. For example, if the user sets to transmit only his or her face excluding body parts, the processor 330 may implement the user's avatar using the remaining sub-modules except the body feature point extraction module (C) 455.

A method for the processor 330 to dynamically select sub-modules of the avatar creation module 450 for various situations to generate an image for a user's call (e.g., avatar creation) will be described later with reference to FIGS. 8 to 10 below. do.

FIG. 5 is a diagram illustrating a process for detecting a reflector from a real-world image according to an embodiment. Referring to FIG. 5, a reflector detection module (e.g., the reflector detection module 410 of FIG. 4) according to an embodiment may include a camera module (e.g., the camera module 180 of FIG. 1 and/or the camera module of FIG. 3 (e.g., From the real-world images 510, 530, and 550 obtained by 310)), some areas in the image where reflectors are determined to be present can be detected as square-shaped bounding boxes 515, 535, and 555. Alternatively, the reflector detection module 410 may detect reflector areas 520, 540, and 560 corresponding to the reflector from the real-world images 510, 530, and 550. The reflector detection module 410 according to one embodiment may not need to accurately detect only the reflector areas 520, 540, and 560 from the real-world images 510, 530, and 550.

A wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) according to an embodiment calculates the position of a reflector within the real-world images 510, 530, and 550. When this happens, the location of the reflector is notified to the user and the user is guided to approach the reflector, thereby allowing the user's image reflected in the reflector to be captured by the camera module 310.

Figure 6 is a diagram for explaining the operation of a video stabilization module of a video call application according to an embodiment. Referring to FIG. 6, in the reflector according to one embodiment, an image 610 of an area corresponding to the reflector in which the entire user's appearance is reflected, an image for a call 620, and an image of an area corresponding to the reflector in which a part of the user's appearance is reflected. At 630, a reconstructed image 640 updating the entire appearance of the current sender, a previously stored image 650 of the user, and a reconstructed image 660 including a previously acquired appearance of the user are shown.

For example, the movement or shaking of a user (e.g., a sender) wearing a wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) causes a reflection corresponding to the reflector. As in the area image 630, a situation may occur in which part of the user's appearance is not reflected in a reflector or the user's appearance is obscured by another object. Alternatively, depending on the user's movement, the user's appearance may be obscured by the frame of the reflector, causing part of the user's appearance to not be reflected in the reflector, or a situation may occur in which the user's appearance reflected in the reflector changes in size by enlarging or shrinking. there is.

The wearable electronic device 300 uses an image stabilization module (e.g., the image stabilization module 440 in FIG. 4) to allow the receiver, the other party of the video call, to recognize the sender's status regardless of the status of the user (sender) in the various situations described above. We can provide a reconstructed image that allows you to see the image intact. For example, the image stabilization module 440 may be configured to define an area corresponding to a reflector that includes a previously acquired image of the user (e.g., a pre-stored user image 650) and a portion of the user's image acquired from the current reflector. Based on the image 630, a reconstructed image 640 that updates the entire appearance of the current sender is generated, and/or a previously acquired image of the user (e.g., a previously stored image of the user) is generated until the user's occlusion is resolved. By repeatedly showing the reconstructed image 660 including (650)), the receiver can fully see the sender.

In addition, the image stabilization module 440 corrects shaking and/or blur of the face and/or body parts recognized by, for example, the face recognition module 420 and/or the body recognition module 430. Stabilized images can be created. For example, the image stabilization module 440 compares the shaky image with the feature extraction results through the CNN network structure and the LSTM network structure and stabilizes the image with the opposite of the movement size and direction of the feature point through the difference in feature point location between the previous frame and the current frame. By converting, a stabilized image can be created. The image stabilization module 440 may also generate stabilized images and/or stabilized reconstructed images 640 and 660 using various known image stabilization techniques.

According to one embodiment, the wearable electronic device 300 does not use an avatar during a video call, but displays the user's image reflected in the mirror, such as the image 610 in the area corresponding to the reflector, according to the user's selection or settings. It can also be used as a video 620 for video calls. At this time, the wearable electronic device 300 may provide an image in which shaking and/or blur of the image 610 in the area corresponding to the reflector has been corrected by the image stabilization module 440 as an image for a call to the other party.

For example, the image stabilization module 440 may directly update a part where an image can be directly acquired, such as the area 670, with an image of the area corresponding to the reflector corresponding to the area 670. Unlike the area 670, the image stabilization module 440 may indirectly update the area 680, for which an image cannot be obtained directly, with an image predicted by, for example, deep learning.

Figure 7 is a diagram for explaining the operation of a face reproduction module of a video call application according to an embodiment. Referring to FIG. 7, the face reproduction module (D) 457 of a video call application (e.g., the video call application 400 of FIG. 4) according to an embodiment is used by the user instead of the user's face image 710. A diagram 700 showing a face image 730 reproduced by a selected face image 720 is shown.

The user can set the video call application 400 to use a graphic object such as a photo of a real person or an emoji that replaces the user's own face as an avatar. At this time, either a feature point-based method or a reproduction-based method can be used as a method for generating an avatar's face that reflects the user's own expression.

Reproduction-based methods allow for more elaborate facial expressions than feature-based methods, but the results can be output in the form of a two-dimensional video. For example, when the recipient is in an augmented reality (AR) or virtual reality (VR) environment as shown in drawing 950 of FIG. 9 below, a user interface (UI) that outputs a video rather than an actual avatar as a video for a call The results can be expressed in a similar form.

The face reproduction module (D) 457 may reproduce the user's face based on, for example, an image-face reproduction method. The face reproduction module (D) 457 is based on deep learning technology and can apply the user's facial expression to the avatar by using the user's face and a photo or video to be used as the avatar without the process of extracting separate feature points from the face. You can. The avatar creation module 450 may generate an avatar using a face reproduced by the face reproduction module (D) 457 using an image-face reproduction method.

The avatar creation module 450 may project the user's facial texture reproduced by the face reproduction module (D) 457 onto the face of the 3D avatar model.

Alternatively, when the user's body image is captured, the avatar creation module 450 projects the user's facial texture reproduced by the face reproduction module (D) 457 onto the head of the 3D avatar model, and the body feature point extraction module (C) 455 can create and/or manipulate a 3D avatar model based on body feature points extracted from the body image.

FIG. 8 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment. Referring to FIG. 8, a user wearing a wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) according to an embodiment uses a video call application (e.g., FIG. A diagram 800 is shown to explain the operation when the video call application 400 of 4 is running, but only audio is transmitted.

For example, as shown in Figure 810, when a user executes the video call application 400 while wearing the wearable electronic device 300, the wearable electronic device 300 uses a camera module (e.g., the camera of FIG. 1). Detection may be attempted by recognizing a reflector, a face presumed to be a user, and/or a body in an image captured by the module 180 and/or the camera module 310 of FIG. 3. For example, the wearable electronic device 300 detects a reflector, the user's face, and and/or the user's body may be recognized.

As shown in the drawing 820, only reflectors (e.g., mirrors) that are far away from the user are recognized in the image of the area corresponding to the reflector captured as a result of the recognition of the reflector detection module 410, and the user reflected in the reflector may not be recognized. there is. In this case, the wearable electronic device 300 may display the location of the reflector to the user.

For example, if the user's image is not recognized, but only the user's voice signal is input, the wearable electronic device 300 may detect the user's call video from the user's voice signal 831 through the process shown in figure 830. (837) can be generated.

The wearable electronic device 300 may transmit the voice signal 831 generated by the user's speech to the voice-face feature point conversion module 451. The voice-facial feature point conversion module 451 can convert the voice signal 831 into human facial feature points 833. At this time, if the user sets the video call application 400 to use the avatar's face (or human face) 835 instead of the user's face, the wearable electronic device 300 uses the human facial feature points 833 as the avatar. It can be reflected on the face 835. For example, the avatar's face 835 may be the face of an avatar designated by the user, or may be the face of an avatar set by default in the video call application 400, but is not necessarily limited thereto. For example, the avatar's face 835 may be a virtual animation character, a cartoon character, and/or a real person's face, but is not necessarily limited thereto.

The wearable electronic device 300 reflects the facial expression associated with the voice signal 831, that is, the human facial feature points 833, on the avatar's face 835 through the feature point-based avatar manipulation module 459 to control the user's facial expression. A video 837 for a call can be created. The wearable electronic device 300 can perform a video call by transmitting the user's call video 837 to the other party ('receiver') of the video call.

Since the other party can see the face of the avatar displayed on the screen through the user's call video 837 and hear the actual user's voice signal, they can feel as if they are making a video call with the avatar. For example, a virtual animation character and/or a real person's face may be used as an avatar.

FIG. 9 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment. Referring to FIG. 9, a user wearing a wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) according to an embodiment is shown in the drawing 910. Likewise, a diagram 900 is shown to explain a method of performing a video call when reflected by a reflector.

For example, as shown in the drawing 910, a user wearing the wearable electronic device 300 may move toward a reflector, and the user's image may be displayed on the reflector.

In this situation, when the user executes a video call application (e.g., the video call application 400 of FIG. 4), the wearable electronic device 300 uses a camera module (e.g., the camera module 180 of FIG. 1 and/or Detection may be attempted by recognizing a reflector, a face presumed to be a user, and/or a body in the image captured by the camera module 310 of 3). For example, the wearable electronic device 300 detects a reflector, the user's face, and and/or the user's body may be recognized.

When the user's appearance reflected in the mirror is confirmed in the image of the area corresponding to the reflector through the face recognition module 420, the wearable electronic device 300 determines that the face image recognized through the video call application 400 is the user's own face. You can check whether this is correct.

For example, the wearable electronic device 300 may capture an image of the user reflected in a mirror ('image of the area corresponding to the reflector') as shown in the drawing 920, along with a message such as "Is this your face?" By receiving feedback from the user's answer to the question, it is possible to check whether the recognized face image is the user's own face. Alternatively, the wearable electronic device 300 compares the captured image of the user's appearance reflected in the mirror ('image of the area corresponding to the reflector') with the user's face stored in advance through the video call application 400 to recognize the user's face. You can check whether the face image is the user's own face. In this case, the confirmation operation for the user may be omitted.

The face of the user whose identity has been confirmed in the process described above through drawing 920 may be stored and reused later. For example, during a video call, the user's face may be hidden for a moment and then shown again. At this time, if the re-seen face captured through the camera module 310 matches the stored face, the wearable electronic device 300 may not attempt user confirmation again.

When facial recognition for the captured image is completed, the wearable electronic device 300 may generate an image 937 for the user's call through the processes shown in the drawing 930. The wearable electronic device 300 may transmit the face image 931 recognized through the face recognition module 420 to the facial feature point extraction module 451. The facial feature point extraction module 451 can extract facial feature points 933 from the recognized face image 931.

For example, if the user sets the video call application 400 to use the avatar's face (or human face) 935 instead of the user's face, the wearable electronic device 300 uses the person's facial feature points 933. can be reflected on the avatar's face 935. For example, the avatar's face 935 may be the face of an avatar designated by the user, or may be the face of an avatar set by default in the video call application 400, but is not necessarily limited thereto. The wearable electronic device 300 can generate an image 937 for a user's call by reflecting the human facial feature points 933 on the avatar's face 935 through the feature point-based avatar manipulation module 459.

Additionally, according to one embodiment, the user's image is reflected in the full-length mirror, and an image of an area corresponding to the reflector including the user's entire body reflected in the full-length mirror may be captured by the camera module. In this case, the wearable electronic device 300 may detect the user's full body image by recognizing the user's entire body from the image of the area corresponding to the reflector through the body recognition module 430. The wearable electronic device 300 extracts body feature points from a full-body image using the body feature point extraction module 455, and transmits the extracted body feature points to the feature point-based avatar manipulation module 459 to use them to manipulate the 3D avatar. The above-described operations may be performed separately from the facial feature point extraction module 453 and the feature point-based avatar manipulation module 459.

As shown in the drawing 950, the wearable electronic device 300 according to an embodiment generates an image 957 for a call for the real environment when the recipient is in a real environment, and the recipient uses augmented reality (augmented reality). When in an augmented reality (AR) or virtual reality (VR) environment, an image 959 for a call can be created for the augmented reality (AR) or virtual reality (VR) environment.

The wearable electronic device 300 may transmit the face image 951 recognized through the face recognition module 420 to the facial feature point extraction module 451. The facial feature point extraction module 451 can extract facial feature points 953 from the recognized face image 951. At this time, if the user sets the video call application 400 to use the avatar 955 designated by the user instead of the user, the wearable electronic device 300 may reflect the facial feature points 953 on the face of the avatar 955. You can. At this time, if there is a full-body image of the user detected from the image of the area corresponding to the reflector through the body recognition module 430, the wearable electronic device 300 uses the body feature points extracted from the full-body image by the body feature point extraction module 455. It can be used to manipulate the avatar 955 by transmitting it to the feature point-based avatar manipulation module 459. The wearable electronic device 300 may generate call images 957 and 959 by reflecting facial feature points 953 and body feature points on the avatar 955.

For example, if the user is in a real environment, the wearable electronic device 300 may generate a call image 957 in which facial feature points 953 and body feature points are reflected in the avatar 955.

In contrast, if the user is in an augmented reality (AR) or virtual reality (VR) environment, the wearable electronic device 300 creates and displays a virtual mobile phone in the augmented reality (AR) or virtual reality (VR) environment, and It is possible to create a call video 959 in which an avatar representing a full-body image including a face is expressed on the mobile phone screen. At this time, the avatar displayed on the virtual mobile phone screen may be expressed in a form similar to a user interface (UI) that outputs a video rather than an actual avatar.

In one embodiment, the avatar 955 designated by the user has predefined control points of the 3D model corresponding to the facial feature points, so the wearable electronic device 300 extracts the facial feature points extracted by the facial feature point extraction module 451. According to (953), the facial expression of the avatar (955) can be freely changed. Additionally, the wearable electronic device 300 creates a three-dimensional model of the avatar 955 to suit the recipient's video call environment, such as, for example, a general display environment and/or an augmented reality (AR) or virtual reality (VR) environment. It can be displayed.

FIG. 10 is a diagram illustrating a method of making a video call by a user wearing a wearable electronic device according to an embodiment. Referring to FIG. 10, a diagram 1000 is shown to explain a method of performing a video call by updating the user's avatar when a change occurs in the user's appearance reflected by a reflector according to an embodiment.

For example, as shown in Figure 1010, when the user's appearance reflected in the mirror is limited to the face, a wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) ) can extract facial feature points by the facial feature point extraction module 453, but the body feature point extraction module 455 cannot extract body feature points. In this case, the feature point-based avatar manipulation module 459 can manipulate the avatar based on facial feature points.

On the other hand, as shown in the drawing 1020, when the user's appearance reflected in the mirror changes from the face to the whole body, the wearable electronic device 300 can generate an image 1036 for a call through the same process as shown in the drawing 1030. there is.

For example, when the image of the area corresponding to the reflector includes the user's full body, the feature point extraction module 453 extracts facial feature points 1032 corresponding to the face 1031 from the full body image and body feature points. The extraction module 455 may extract body feature points 1034 corresponding to the whole body 1033.

The facial feature points 1032 and body feature points 1034 can be used by the feature point-based avatar manipulation module 459 to manipulate the avatar.

For example, if there is an avatar 1035 designated by the user, the feature point-based avatar manipulation module 459 can manipulate the movement of the avatar 1035 based on the facial feature points 1032 and body feature points 1034. . At this time, as various feature points transmitted to the feature point-based avatar manipulation module 459 change, the update area of the avatar displayed in the call video 1036 may change. In other words, when the user's appearance reflected in the mirror changes from drawing 1010 to drawing 1020, the update area of the avatar in the call video 1036 changes from the first update area 1037 to the second update area ( 1038) and can be dynamically changed. For example, the wearable electronic device 300 may perform a direct update on the first update area 1037 and an indirect update on the second update area 1038.

For example, when the user is far away from the mirror and the facial feature points cannot be extracted by the facial feature point extraction module 453, the wearable electronic device 300 extracts the body feature points 1034 extracted by the body feature point extraction module 455. ) and the second update area 1038 corresponding to the entire body of the avatar can be indirectly updated based on the facial feature points predicted through the voice signal. Alternatively, for example, the body feature point extraction module 455 may not properly extract facial feature points and body feature points, such as when the user moves too far away from the mirror or disappears from the mirror to the extent that the body feature point extraction module 455 cannot extract the body feature point. If there is none, the wearable electronic device 300 may directly update the first update area 1037 corresponding to the avatar's face using facial feature points extracted based on the voice signal by the voice-face feature point conversion module 451. .

Figure 11 is a flowchart showing a method of operating a wearable electronic device according to an embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

In operation 1110, the wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) responds to the execution of an application for a video call on the wearable electronic device 300. , real-world images (e.g., real-world images 510, 530, and 550 of FIG. 3) surrounding the user can be obtained.

In operation 1120, the wearable electronic device 300 may detect a reflector from the real-world images 510, 530, and 550 acquired in operation 1110.

In operation 1130, the wearable electronic device 300 may calculate the position information of the reflector including the distance and direction from the image of the area corresponding to the reflector detected in operation 1120 among the real-world images 510, 530, and 550. The method by which the wearable electronic device 300 calculates the position information of the reflector will be described with reference to FIG. 12 below.

In operation 1140, the wearable electronic device 300 uses an output device (e.g., the sound output module 155 of FIG. 1 and/or By controlling the display module 160, the user's call video (e.g., call video 837 in FIG. 8, call video 937, 957, 959 in FIG. 9) to be transmitted to the other party of the video call, and/ Alternatively, the call video (1035, 1036) of FIG. 10) can be generated. For example, the wearable electronic device 300 may use a camera module (e.g., the camera module of FIG. 1) provided in the wearable electronic device 300 according to a determination that the image of the area corresponding to the reflector includes at least a part of the user's body. Generate images (837, 937, 957, 959, 1035, 1036) for the user's call using the image of the area corresponding to the reflector obtained using (180) and/or the camera module 310 of FIG. 3). can do.

The wearable electronic device 300 acquires an image of the area corresponding to the reflector reflected by the user who moves to the location of the reflector according to the guidance, and uses the image of the area corresponding to the reflector to capture the user's call video (837, 937, 957, 959, 1035, 1036) can be generated. The method by which the wearable electronic device 300 generates the user's call images 837, 937, 957, 959, 1035, and 1036 will be described with reference to FIG. 13 below.

Figure 12 is a flowchart showing a method of calculating location information from an image of an area corresponding to a reflector reflected on a reflector according to an embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

In operation 1210, when a reflector is detected from a real-world image (e.g., the real-world images 510, 530, and 550 of FIG. 5), the wearable electronic device 300 determines that the image of the area corresponding to the reflector includes a face of the user. It can be determined whether or not it includes at least part of the body.

In operation 1220, upon determining that the image of the area corresponding to the reflector does not include at least a body part of the user, the wearable electronic device 300 determines the user's location in three-dimensional space and a camera module (e.g., the camera module of FIG. 1 ). Position information of the reflector may be calculated based on the viewing angle of (180) and/or the camera module 310 of FIG. 3) and the location of the reflector.

Figure 13 is a flowchart showing a method of generating a video for a user's call sent to the other party of a video call, according to an embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

In operation 1310, the wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) converts the image of the area corresponding to the reflector into the user's call image (e.g., FIG. 8). A first selection determined by the call video 837, the call video 937, 957, 959 of FIG. 9, and/or the call video 1035, 1036) of FIG. 10, an area corresponding to the reflector A second choice of determining the reconstructed image reconstructed using the image (e.g., the reconstructed image 640, 660 of FIG. 6) as the user's call video (837, 937, 957, 959, 1035, 1036), and The user's choice of any one of the third choices that determines the graphic object reconstructed using the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036) can be input. there is.

In operation 1320, the wearable electronic device 300 captures the user's face and at least a body part of the user by a camera module (e.g., the camera module 180 in FIG. 1 and/or the camera module 310 in FIG. 3). Depending on at least one of whether an image of the area corresponding to the reflector is acquired and the user's selection, images 837, 937, 957, 959, 1035, and 1036 for the user's call may be generated. For example, when the first selection is input, the wearable electronic device 300 may determine the image of the area corresponding to the reflector as the user's call image 837, 937, 957, 959, 1035, and 1036. For example, when the second selection is input, the wearable electronic device 300 may generate reconstructed images 640 and 660 that reconstruct the user's appearance using the image of the area corresponding to the reflector. The wearable electronic device 300 may compare the image of the area corresponding to the reflector with the user's pre-stored image to determine which part of the pre-stored image the image of the area corresponding to the reflector corresponds to. The wearable electronic device 300 updates the part corresponding to the image of the area corresponding to the reflector in the pre-stored image by the image of the area corresponding to the reflector, and updates the part corresponding to the image of the area corresponding to the reflector in the pre-stored image. The video for the user's call can be reconstructed by updating the part with the video predicted by the neural network. The wearable electronic device 300 may determine the reconstructed images 640 and 660 as the user's call images 837, 937, 957, 959, 1035, and 1036. Alternatively, according to the third selection, the wearable electronic device 300 may generate a graphic object that reconstructs the user's call image corresponding to the image of the area corresponding to the reflector.

FIG. 14 is a flowchart showing a method of operating the wearable electronic device 300 according to an embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

In operation 1410, the wearable electronic device (e.g., the electronic device 101 of FIG. 1 and/or the wearable electronic device 300 of FIG. 3) allows the user to wear the wearable electronic device 300 and use a video call application (e.g., When the video call application 400 of FIG. 4 is executed, the surrounding real-world image (FIG. 5) is captured through a provided camera module (e.g., the camera module 180 of FIG. 1 and/or the camera module 310 of FIG. 3). Real-world images (510, 530, 550) can be obtained.

In operation 1420, the wearable electronic device 300 may determine whether a reflector is detected in the real-world images 510, 530, and 550 obtained in operation 1420. The wearable electronic device 300 may detect reflectors in the real-world images 510, 530, and 550 using the reflector detection module 410. If it is determined that no reflector is detected in operation 1420, the wearable electronic device 300 may update the avatar according to the user's voice information (e.g., voice signal 831 in FIG. 8) in operation 1460. After performing operation 1460, the wearable electronic device 300 may end the operation as the user ends the video call, as shown in FIG. 14, or may detect a reflector in operation 1420 and determine whether there is a person reflected in the reflector. You can also check it repeatedly.

Alternatively, if it is determined that a reflector is detected in operation 1420, the wearable electronic device 300 may determine whether there is a person reflected in the reflector in operation 1430. The wearable electronic device 300 may detect a person from the real-world images 510, 530, and 550 by, for example, the face recognition module 420 and/or the body recognition module 430. If it is determined in operation 1430 that there is no person reflected in the reflector, in operation 1440, the wearable electronic device 300 determines whether the user is A user interface (UI) guide can be provided so that you can check the image reflected on the reflector. The wearable electronic device 300 may guide the user to approach the reflector through a user interface guide.

If it is determined in operation 1430 that there is a person reflected in the reflector, in operation 1450 the wearable electronic device 300 may determine whether the person reflected in the reflector is the user. When a face is recognized in the image of the area corresponding to the reflector captured from the reflector by the face recognition module 420 of the video call application 400, the wearable electronic device 300 can confirm whether the face is the user's face. there is.

If it is determined in operation 1450 that the person reflected in the reflector is not the user, the wearable electronic device 300 may update the avatar according to the user's voice information through operation 1460. The wearable electronic device 300, for example, uses the voice-face feature point conversion module 451 of the avatar creation module 450 until a face identified as the user is detected in the image of the area corresponding to the reflector obtained from the reflector. ) and the feature point-based avatar manipulation module 459 can update the avatar according to the user's voice information.

If it is determined in operation 1450 that the person reflected in the reflector is the user, in operation 1470 the wearable electronic device 300 detects the user reflected in the reflector and creates an avatar according to the detected image information of the user, that is, the image of the area corresponding to the reflector. can be updated. In operation 1450, if it is determined that the person reflected in the reflector, that is, the person included in the image in the area corresponding to the reflector, is the user, the wearable electronic device 300 reports the detection result in the image in the area corresponding to the reflector to the image stabilization module 440. ) can be passed as input. The wearable electronic device 300 may update the user's avatar based on image information according to the processing results of the image stabilization module 440 and the avatar creation module 450.

In operation 1480, the wearable electronic device 300 may end its operation as the user ends the video call.

The above-described operations 1420, 1430, 1440, and 1450 may be performed repeatedly until the user's appearance is confirmed in the image of the area corresponding to the reflector captured from the reflector.

FIG. 15 is a diagram illustrating a method of performing a video call between a sender terminal and a recipient terminal using a video call application according to an embodiment. Referring to FIG. 15, according to one embodiment, the sender terminal 1510 of the sender terminal 1510 and the receiver terminal 1550 on which the same video call application (e.g., the video call application 400 of FIG. 4) is installed is installed. A diagram 1500 is shown to explain a method of performing a video call when it is difficult to transmit large amounts of data such as video due to poor network conditions. Hereinafter, in the video call application 400 installed in each terminal, a module indicated by a solid line may indicate an activated module, and a module indicated by a dotted line may indicate a deactivated module.

As described above, when it is difficult to transmit large data such as video due to poor network conditions on the sender terminal 1510, the video call application 400 may, for example, connect the sender terminal 1510 and the receiver terminal 1550. ) A call video (e.g., call video 837 in FIG. 8, call video 837 in FIG. 8, call video 837 in FIG. Some of the operations for generating the video for use (937, 957, 959) and/or the video for call (1035, 1036) of FIG. 10 may be delegated to the recipient terminal 1550. Each module (1520, 1525, 1530, 1535, 1540 and/or 1560, 1565, 1570, 1575) of the video call application shown in FIG. 15 is the module of the video call application 400 shown in FIG. 4. (410, 420, 430, 440, 450) can correspond to each.

In this case, the sender terminal 1510 can detect a reflector from the image by the reflector detection module 1520 of the video call application 400. The face recognition module 420 and the body recognition module 430 may recognize and detect a face and/or body from an image of an area corresponding to a reflector captured from a reflector.

The facial feature point extraction module 1542 may extract facial feature points in the form of two-dimensional or three-dimensional coordinates from a face recognized in the image of the area corresponding to the reflector. Additionally, the body feature point extraction module 1543 may extract body feature points in the form of two-dimensional or three-dimensional coordinates from the body recognized in the image of the area corresponding to the reflector. The video application may transmit the extracted facial feature points and/or body feature points to the recipient terminal 1550.

The recipient terminal 1550 uses the feature point-based avatar manipulation module 1585 based on facial and/or body feature points in the form of two-dimensional or three-dimensional coordinates transmitted by the sender terminal 1510 to create the call video (837, 937) of the sender. , 957, 959, 1035, 1036), an avatar to be used can be created and/or manipulated. The receiver terminal 1550 can show the sender's call video (837, 937, 957, 959, 1035, 1036) generated by the feature point-based avatar manipulation module 1585 to the user (sender) of the sender terminal 1510. there is. In this case, it may take time for the sender terminal 1510 to receive the sender's call video (837, 937, 957, 959, 1035, 1036) generated by the receiver terminal 1550.

FIG. 16 is a diagram illustrating a method of performing a video call between a sender terminal and a recipient terminal using a video call application according to an embodiment. Referring to FIG. 16, according to one embodiment, the sender terminal 1510 of the sender terminal 1510 and the receiver terminal 1550 on which the same video call application (e.g., the video call application 400 of FIG. 4) is installed is installed. Performing a video call in a situation where the operation of the video stabilization module 1535 and the avatar creation module 1540 is difficult due to the high processor (e.g., processor 120 in FIG. 1 and/or processor 330 in FIG. 3) occupancy rate. A diagram 1600 is shown to illustrate the method.

For example, in a situation where it is difficult to operate the video stabilization module 1535 and the avatar creation module 1540 due to the high processor 330 occupancy rate of the sender terminal 1510 that wants to perform a video call, the recipient terminal, which is the other party of the video call, There may be room in the processor 330 share on the (1550) side.

In this case, the sender terminal 1510 receives the image of the area corresponding to the reflector captured by the reflector detection module 1520, the face recognition result recognized by the face recognition module 1525 and/or the body recognition module 1530, and /Or the body recognition result may be transmitted to the recipient terminal 1550. The sender terminal 1510 may have the receiver terminal 1550 perform processing (e.g., image stabilization and/or avatar creation) on the image, face recognition result, and/or body recognition result of the area corresponding to the transmitted reflector. there is.

The recipient terminal 1550, which has received the image, face recognition result, and/or body recognition result of the area corresponding to the reflector, transfers the image stabilized by the image stabilization module 1575 to the avatar creation module 1580 to make the call of the sender. Create an avatar to be used as a video (e.g., call video 837 in FIG. 8, call video 937, 957, 959 in FIG. 9, and/or call video 1035, 1036 in FIG. 10). and/or can be manipulated. The recipient terminal 1550 may display the sender's call video (837, 937, 957, 959, 1035, 1036) generated by the avatar creation module 1580 to the user (sender) of the sender terminal 1510.

In this case, it may take time for the sender terminal 1510 to receive the sender's call video (837, 937, 957, 959, 1035, 1036) generated by the receiver terminal 1550.

FIG. 17 is a diagram illustrating a method of performing a video call between a sender terminal on which a video call application is installed and a recipient terminal on which another video call application is installed, according to an embodiment. Referring to FIG. 17, a video call is made between a sender terminal 1510 on which a video call application (e.g., the video call application 400 of FIG. 4) according to an embodiment is installed and a recipient terminal 1710 on which a third-party video call application is installed. A diagram 1700 is shown to illustrate how to perform.

According to one embodiment, the recipient terminal 1710, which is the other party of the video call, does not need to use the same video call application 400 as the video call application 400 according to one embodiment. Even if the recipient terminal 1710 uses a different video call application, the sender terminal 1710 uses the reflector detection module 1520, face recognition module 1525, body recognition module 1530, video stabilization module 1535, and avatar creation. A call video that performs all processing on the real-world video using the module 1540 (e.g., the call video 837 in FIG. 8, the call video 937, 957, and 959 in FIG. 9, and/or A video call can be performed by generating 10 call videos (1035, 1036)) and then transmitting them to the recipient terminal 1710.

FIG. 18 is a diagram illustrating a method of performing a video call between a recipient terminal on which a video call application is installed and a sender terminal on which a voice call application is installed, according to an embodiment. Referring to FIG. 18, a diagram 1800 is shown to explain a method of performing a video call when the sender terminal 1810 transmits only voice information to the receiver terminal 1550 according to an embodiment.

For example, when only voice information is transmitted from the sender terminal 1810 to the recipient terminal 1550 using a voice call application (e.g., the voice call application 400 in FIG. 4), the recipient terminal 1550 transmits the video. The avatar basically stored in the avatar creation module 1580 of the call application 400, and the voice-facial feature point conversion module 1581 convert the sender's voice information (e.g., the voice signal 831 in FIG. 8) into facial feature points. Based on the result of conversion, the avatar whose face shape is manipulated according to the voice information of the sender (831) is converted into the sender's call video (837, 937, 957, 959, 1035, 1036).

Even if only voice information 831 is transmitted from the sender terminal 1810, the receiver terminal 1550 uses the sender's call video (837, 937, 957, 959, 1035, 1036) generated by the receiver terminal 1550 for a video call. By displaying the screen on the sender's side, you can make the recipient feel as if they are making a video call while seeing the sender's actual face.

According to one embodiment, the wearable electronic devices 101 and 300 display real-world images 510, 530 and 550 around the user in response to the execution of the application 400 for video calls in the wearable electronic devices 101 and 300. A camera module (180, 310) that acquires, detects a reflector from the real-world images (510, 530, 550), and determines the distance and direction from the image of the area corresponding to the reflector among the real-world images (510, 530, 550) By calculating the location information of the reflector including and controlling the output devices (155. 160) to provide guidance for the user to move to the location of the reflector according to the location information, the video call can be transmitted to the other party. A processor (120, 330) that generates the user's call video (837, 937, 957, 959, 1035, 1036), and the user's call video (837, 937, 957, 959, 1035, 1036) It may include communication modules 190 and 350 that transmit.

According to one embodiment, when the processor (120, 330) detects the reflector from the real-world images (510, 530, 550), the image of the area corresponding to the reflector is at least the body of the user including the face. determining whether the image of the area corresponding to the reflector does not include at least a body part of the user, the location of the user in three-dimensional space, the camera modules 180, 310; The location information can be calculated based on the viewing angle and the location of the reflector.

According to one embodiment, when the reflector is not detected from the real-world images 510, 530, and 550, the processors 120 and 330 generate the user's call in response to the user's voice information 831. The video call can be performed using the video calls 837, 937, 957, 959, 1035, and 1036.

According to one embodiment, the processor 120, 330 determines that the image of the area corresponding to the reflector includes at least a body part of the user, and the processor 120, 330 obtains the image using the camera module 180, 310. The user's call images (837, 937, 957, 959, 1035, 1036) can be generated using the image of the area corresponding to the reflector.

According to one embodiment, the processors 120 and 330 acquire an image of the area corresponding to the reflector reflected on the reflector by the user who moves to the location of the reflector according to the guidance, and capture the image of the area corresponding to the reflector. Using video, the user's call video (837, 937, 957, 959, 1035, 1036) can be created.

The wearable electronic device (101, 300) according to one embodiment includes an input module (150, 370) that receives the user's selection of the user's call video (837, 937, 957, 959, 1035, 1036). Further comprising, the processors 120, 330 determine whether an image of an area corresponding to the reflector including the user's face and at least a body part of the user is acquired by the camera modules 180, 310, and Depending on at least one of the user's selections, an image (837, 937, 957, 959, 1035, 1036) for the user's call is generated, and the user's selection generates an image of the area corresponding to the reflector for the user's call. The first choice is to determine the user's call video (837, 937, 957, 959, 1035, 1036), and the reconstructed images (640, 660) reconstructed using the image of the area corresponding to the reflector ( 837, 937, 957, 959, 1035, 1036), and a graphic object reconstructed using the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959). , 1035, 1036).

According to one embodiment, when the first selection is input, the processors 120 and 330 convert the image of the area corresponding to the reflector into the user's call image 837, 937, 957, 959, 1035, 1036. ) can be determined.

According to one embodiment, when the second selection is input, the processors 120 and 330 generate reconstructed images 640 and 660 that reconstruct the user's appearance using the image of the area corresponding to the reflector, The reconstructed images 640 and 660 may be determined as the user's call images 837, 937, 957, 959, 1035, and 1036.

According to one embodiment, the wearable electronic devices (101, 300) further include memories (130, 390) including a pre-stored image (650) of the user, and the processor (120, 330) Compare the image of the corresponding area with the user's pre-stored image 650 to determine which part of the pre-stored image 650 the image of the area corresponding to the reflector corresponds to, and compare the pre-stored image 650 of the user with the image of the corresponding area. ), the part corresponding to the image of the area corresponding to the reflector is updated by the image of the area corresponding to the reflector, and the part of the pre-stored image 650 that does not correspond to the image of the area corresponding to the reflector is updated by the image of the area corresponding to the reflector. The reconstructed images 640 and 660 can be generated by updating the images predicted by the neural network.

According to one embodiment, when the third selection is input, the processors 120 and 330 generate an image of an area corresponding to the reflector or a graphic object corresponding to the reconstructed images 640 and 660, and The graphic object may be determined to be the user's call video (837, 937, 957, 959, 1035, 1036).

According to one embodiment, the processors 120 and 330 select the user's face from the real world images 510, 530 and 550 when the real world images 510, 530 and 550 do not include the user's face. Until this is detected, the video call can be performed using the user's call video (837, 937, 957, 959, 1035, 1036) generated in response to the user's voice information (831).

According to one embodiment, the method of operating the wearable electronic device (101, 300) is to display real-world images (510, 530) around the user in response to the execution of the application (400) for video calling in the wearable electronic device (101, 300). , 550), operation 1110, detecting a reflector from the real-world images 510, 530, 550, operation 1120, distance and direction from the image of the area corresponding to the reflector among the real-world images 510, 530, 550 Operation 1130 of calculating the location information of the reflector including, and controlling the output devices 155 and 160 to provide guidance for the user to move to the location of the reflector according to the location information, thereby making the video call possible. Operation 1140 may include generating the user's call video (837, 937, 957, 959, 1035, 1036) to be transmitted to the other party.

According to one embodiment, the operation of calculating the location information is performed when a reflector is detected from the real-world images 510, 530, and 550, and the image of the area corresponding to the reflector is at least a body part of the user including a face. Operation 1210 of determining whether the image of the area corresponding to the reflector does not include at least a body part of the user, the location of the user in three-dimensional space, the camera modules 180, 310 ) may include an operation 1220 of calculating the location information based on the viewing angle of ) and the location of the reflector.

According to one embodiment, a method of operating the wearable electronic device (101, 300) includes, according to a determination that the image of the area corresponding to the reflector includes at least a body part of the user, the wearable electronic device (101, 300) An operation of generating the user's call video (837, 937, 957, 959, 1035, 1036) using the image of the area corresponding to the reflector obtained using the provided camera module (180, 310) is further performed. It can be included.

According to one embodiment, the operation of generating the user's call images (837, 937, 957, 959, 1035, 1036) involves the user moving to the location of the reflector according to the guidance and moving to the reflector reflected on the reflector. It may further include an operation of acquiring an image of the corresponding area, and an operation of generating an image (837, 937, 957, 959, 1035, 1036) for the user's call using the image of the area corresponding to the reflector. there is.

According to one embodiment, the operation of generating the user's call video (837, 937, 957, 959, 1035, 1036) is performed on the user's call video (837, 937, 957, 959, 1035, 1036). An operation of receiving the user's selection, and corresponding to the reflector including the user's face and at least a body part of the user by the camera module (180, 310) provided in the wearable electronic device (101, 300) An operation of generating an image (837, 937, 957, 959, 1035, 1036) for a call of the user according to at least one of whether an image of the area is acquired and the user's selection, and the user's selection. The first option is to determine the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036), and reconstruction is performed using the image of the area corresponding to the reflector. A second choice of determining the images (640, 660) as the user's call images (837, 937, 957, 959, 1035, 1036), and a graphic object reconstructed using the image of the area corresponding to the reflector It may include at least one of the third selections determined by the user's call video (837, 937, 957, 959, 1035, 1036).

According to one embodiment, the operation of generating the user's call images (837, 937, 957, 959, 1035, 1036) is to generate the image of the area corresponding to the reflector to the user when the first selection is input. It may include an operation of determining the call video (837, 937, 957, 959, 1035, 1036).

According to one embodiment, the operation of generating the user's call images (837, 937, 957, 959, 1035, 1036) is performed by the image of the area corresponding to the reflector when the second selection is input. An operation of generating reconstructed images (640, 660) that reconstruct the appearance of the user, and determining the reconstructed images (640, 660) as the user's call images (837, 937, 957, 959, 1035, 1036) Can include actions.

According to one embodiment, the operation of generating reconstructed images 640 and 660 that reconstruct the user's call images 837, 937, 957, 959, 1035, and 1036 includes the image of the area corresponding to the reflector and the Comparing the user's pre-stored image 650 to determine which part of the pre-stored image 650 the image of the area corresponding to the reflector corresponds to, and comparing the pre-stored image 650 to the reflector The part corresponding to the image of the corresponding area is updated by the image of the area corresponding to the reflector, and the part of the pre-stored image 650 that does not correspond to the image of the area corresponding to the reflector is predicted by a neural network. It may include an operation of reconstructing the user's call video (837, 937, 957, 959, 1035, 1036) by updating by video.

According to one embodiment, the operation of generating the user's call images (837, 937, 957, 959, 1035, 1036) is performed in response to the image of the area corresponding to the reflector when the third selection is input. It may include creating a graphic object that reconstructs the user's call video (837, 937, 957, 959, 1035, 1036).

Claims (20)

A camera module (180, 310) that acquires real-world images (510, 530, 550) around the user in response to the execution of the application (400) for video calling in the wearable electronic device (101, 300);
Detecting a reflector from the real-world images 510, 530, and 550, and calculating location information of the reflector including distance and direction from an image of an area corresponding to the reflector among the real-world images 510, 530, and 550. , By controlling the output devices (155, 160) to provide guidance for the user to move to the location of the reflector according to the location information, the user's call video (837, 937, Processors 120, 330 generating 957, 959, 1035, 1036); and
Communication modules (190, 350) that transmit the user's call video (837, 937, 957, 959, 1035, 1036)
Wearable electronic devices (101, 300), including. According to paragraph 1,
The processors 120 and 330 are
When the reflector is detected from the real-world images 510, 530, and 550, determine whether the image of the area corresponding to the reflector includes at least a body part of the user including the face,
Based on the user's location in three-dimensional space, the viewing angle of the camera modules 180, 310, and the location of the reflector, upon determining that the image of the area corresponding to the reflector does not include at least a body part of the user A wearable electronic device (101, 300) that calculates the location information. According to any one of claims 1 and 2,
The processors 120 and 330 are
If the reflector is not detected from the real-world images (510, 530, 550), the user's call video (837, 937, 957, 959, 1035, 1036) generated in response to the user's voice information (831) ), a wearable electronic device (101, 300) that performs the video call. According to any one of claims 1 to 3,
The processors 120 and 330 are
According to the determination that the image of the area corresponding to the reflector includes at least a body part of the user, the user's call is made using the image of the area corresponding to the reflector obtained using the camera modules 180 and 310. A wearable electronic device (101, 300) that generates images (837, 937, 957, 959, 1035, 1036). According to any one of claims 1 to 4,
The processors 120 and 330 are
The user, who moves to the location of the reflector according to the guidance, acquires an image of the area corresponding to the reflector reflected on the reflector,
A wearable electronic device (101, 300) that generates an image (837, 937, 957, 959, 1035, 1036) for the user's call using the image of the area corresponding to the reflector. According to any one of claims 1 to 5,
Input modules (150, 370) that receive the user's selection of the user's call video (837, 937, 957, 959, 1035, 1036)
It further includes,
The processors 120 and 330 are
Depending on at least one of whether an image of the area corresponding to the reflector including the user's face and at least a body part of the user is acquired by the camera module 180, 310 and the user's selection, the user's Create video for calls (837, 937, 957, 959, 1035, 1036),
The user's choice is
A first selection to determine the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036),
A second choice of determining the reconstructed images (640, 660) reconstructed using the image of the area corresponding to the reflector as the user's call images (837, 937, 957, 959, 1035, 1036), and
A third option to determine the graphic object reconstructed using the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036)
A wearable electronic device (101, 300) including at least one of the following. According to any one of claims 1 to 6,
The processors 120 and 330 are
When the first selection is input, the wearable electronic device (101, 300) determines the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036). According to any one of claims 1 to 7,
The processors 120 and 330 are
When the second selection is input, reconstructed images 640, 660 are generated by reconstructing the user's appearance using the image of the area corresponding to the reflector, and the reconstructed images 640, 660 are used for the user's call. A wearable electronic device (101, 300) that determines using images (837, 937, 957, 959, 1035, 1036). According to any one of claims 1 to 8,
Memory (130, 390) containing the user's pre-stored image (650)
It further includes,
The processors 120 and 330 are
Compare the image of the area corresponding to the reflector with the user's pre-stored image 650 to determine which part of the pre-stored image 650 the image of the area corresponding to the reflector corresponds to,
The part corresponding to the image of the area corresponding to the reflector in the pre-stored image 650 is updated by the image of the area corresponding to the reflector, and the image of the area corresponding to the reflector in the pre-stored image 650 is updated. A wearable electronic device (101, 300) that generates the reconstructed images (640, 660) by updating parts that do not correspond to the image predicted by a neural network. According to any one of claims 1 to 9,
The processors 120 and 330 are
When the third selection is input, a graphic object corresponding to the image of the area corresponding to the reflector or the reconstructed image 640, 660 is created, and the graphic object is converted into the user's call video 837, 937, 957, 959, 1035, 1036), wearable electronic device (101, 300). According to any one of claims 1 to 10,
The processors 120 and 330 are
If the real-world images 510, 530, and 550 do not include the user's face, the user's voice information 831 is added until the user's face is detected from the real-world images 510, 530, and 550. A wearable electronic device (101, 300) that performs the video call using the correspondingly generated call video (837, 937, 957, 959, 1035, 1036) of the user. An operation of acquiring real-world images (510, 530, 550) around the user in response to execution of the application (400) for video calling in the wearable electronic device (101, 300);
An operation of detecting a reflector from the real world images (510, 530, 550);
An operation of calculating position information of the reflector including distance and direction from an image of an area corresponding to the reflector among the real-world images 510, 530, and 550; and
By controlling the output devices (155, 160) to provide guidance for the user to move to the location of the reflector according to the location information, the user's call video (837, 937, 957) to be transmitted to the other party of the video call , 959, 1035, 1036).
A method of operating a wearable electronic device (101, 300) including. According to clause 12,
The operation of calculating the location information is
When a reflector is detected from the real-world images 510, 530, and 550, determining whether an image of an area corresponding to the reflector includes at least a body part of the user including a face; and
Based on the user's location in three-dimensional space, the viewing angle of the camera modules 180, 310, and the location of the reflector, upon determining that the image of the area corresponding to the reflector does not include at least a body part of the user The operation of calculating the location information
A method of operating a wearable electronic device (101, 300) including. According to any one of claims 12 to 13,
According to the determination that the image of the area corresponding to the reflector includes at least a body part of the user, the image corresponding to the reflector acquired using the camera module 180, 310 provided in the wearable electronic device 101, 300 An operation of generating a video (837, 937, 957, 959, 1035, 1036) for the user's call using the video of the area
A method of operating a wearable electronic device (101, 300), further comprising: According to any one of claims 12 to 14,
The operation of generating the video (837, 937, 957, 959, 1035, 1036) for the user's call is
An operation of the user moving to the location of the reflector according to the guidance to acquire an image of an area corresponding to the reflector reflected on the reflector; and
An operation of generating an image (837, 937, 957, 959, 1035, 1036) for the user's call using the image of the area corresponding to the reflector.
A method of operating a wearable electronic device (101, 300), further comprising: According to any one of claims 12 to 15,
The operation of generating the video (837, 937, 957, 959, 1035, 1036) for the user's call is
An operation of receiving the user's selection of the user's call video (837, 937, 957, 959, 1035, 1036); and
Whether an image of the area corresponding to the reflector including the user's face and at least a body part of the user is acquired by the camera module 180, 310 provided in the wearable electronic device 101, 300, and the user An operation of generating a video (837, 937, 957, 959, 1035, 1036) for the user's call according to at least one of the selection of
Including,
The user's choice is
A first selection to determine the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036),
A second choice of determining the reconstructed images (640, 660) reconstructed using the image of the area corresponding to the reflector as the user's call images (837, 937, 957, 959, 1035, 1036), and
A third option of determining the graphic object reconstructed using the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036)
A method of operating a wearable electronic device (101, 300), including at least one of: According to any one of claims 12 to 16,
The operation of generating the video (837, 937, 957, 959, 1035, 1036) for the user's call is
When the first selection is input, an operation of determining the image of the area corresponding to the reflector as the user's call image (837, 937, 957, 959, 1035, 1036)
A method of operating a wearable electronic device (101, 300) including. According to any one of claims 12 to 17,
The operation of generating the video (837, 937, 957, 959, 1035, 1036) for the user's call is
When the second selection is input, generating reconstructed images (640, 660) in which the user's appearance is reconstructed using the image of the area corresponding to the reflector; and
An operation of determining the reconstructed images (640, 660) as the user's call images (837, 937, 957, 959, 1035, 1036)
A method of operating a wearable electronic device (101, 300) including. According to any one of claims 12 to 18,
The operation of generating reconstructed images (640, 660) that reconstruct the user's call images (837, 937, 957, 959, 1035, 1036) is
Comparing the image of the area corresponding to the reflector with the user's pre-stored image 650 to determine which part of the pre-stored image 650 the image of the area corresponding to the reflector corresponds to;
The part corresponding to the image of the area corresponding to the reflector in the pre-stored image 650 is updated by the image of the area corresponding to the reflector, and the image of the area corresponding to the reflector in the pre-stored image 650 is updated. An operation to reconstruct the user's call video (837, 937, 957, 959, 1035, 1036) by updating the portion that does not correspond to the video predicted by a neural network.
A method of operating a wearable electronic device (101, 300) including. According to any one of claims 12 to 19,
The operation of generating the video (837, 937, 957, 959, 1035, 1036) for the user's call is
When the third selection is input, an operation of creating a graphic object that reconstructs the user's call video (837, 937, 957, 959, 1035, 1036) in response to the image of the area corresponding to the reflector.
A method of operating a wearable electronic device (101, 300) including.

Images