KR20240020259A - Wearable device, method and computer readable medium for identifying gesture based on at least one of image, or electromyogram - Google Patents

Abstract

According to one embodiment, the processor of the wearable device may be configured to obtain information representing the electromyogram of the hand from an external electronic device through a communication circuit of the wearable device. The processor may be configured to acquire an image including the hand using the camera based on identifying the electromyogram having an amplitude difference that exceeds a specified threshold based on the information. The processor may be configured to display, in a first state in which the palm of the hand is identified in the image, a first visual object including a color in association with the palm shown through the display. The processor is configured to display, in a second state different from the first state, a second visual object having the shape of the trace on the image, based on the trace of at least one finger included in the hand. It can be. For example, a wearable device can selectively execute different functions for editing an image based on the orientation of the hand as defined by the palm or the back of the hand.

Description

Wearable device, method, and computer-readable storage medium for identifying a gesture based on at least one of electromyography, or image

The descriptions below relate to a wearable device, method, and computer-readable storage medium for identifying a gesture based on at least one of an electromyogram or an image.

To provide an enhanced user experience, we provide augmented reality (AR) services that display computer-generated information in conjunction with external objects in the real-world. Electronic devices are being developed. The electronic device may be a wearable device that can be worn by a user. For example, the electronic device may be AR glasses and/or a head-mounted device (HMD).

According to an embodiment, a wearable device may include a display, a camera, a communication circuit, and a processor. The processor may be configured to obtain information representing the electromyogram of the hand from an external electronic device through the communication circuit. The processor may be configured to acquire an image including the hand using the camera based on identifying the electromyogram having an amplitude difference that exceeds a specified threshold based on the information. The processor may be configured to display, in a first state in which the palm of the hand is identified in the image, a first visual object including a color in association with the palm shown through the display. The processor is configured to display, in a second state different from the first state, a second visual object having the shape of the trace on the image, based on the trace of at least one finger included in the hand. It can be.

According to one embodiment, a method of using a wearable device may include acquiring information representing the electromyogram of the hand from an external electronic device through a communication circuit of the wearable device. The method may include acquiring an image including the hand using a camera of the wearable device based on identifying the electromyogram having an amplitude difference exceeding a specified threshold based on the information. . The method includes displaying a first visual object including a color in association with the palm shown through the display of the wearable device, within a first state in which the palm of the hand is identified in the image. can do. The method includes, in a second state different from the first state, based on the trace of at least one finger included in the hand, displaying a second visual object having the shape of the trace on the image. It can be included.

According to one embodiment, in a non-transitory computer-readable storage medium storing one or more programs, the one or more programs, when executed by a processor of a wearable device, are transmitted from an external electronic device through a communication circuit of the wearable device. and instructions that cause the wearable device to acquire information representing electromyography of the hand. The one or more programs, when executed by the processor, are configured to obtain an image including the hand using a camera of the wearable device based on identifying the electromyogram that exceeds a specified threshold based on the information, It may include instructions that trigger the wearable device. The one or more programs, when executed by the processor, in association with the palm shown through the display of the wearable device, within a first state that identifies the palm of the hand in the image, include a color. 1 may include instructions that cause the wearable device to display a visual object. The one or more programs are configured to display, in a second state different from the first state, a second visual object having the shape of the trace on the image, based on the trace of at least one finger included in the hand. , may include instructions that cause the wearable device.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
FIG. 2 illustrates an example of a user interface (UI) provided by a wearable device, according to an embodiment.
Figure 3A shows an example of a perspective view of a wearable device, according to one embodiment.
FIG. 3B shows an example of one or more hardware deployed within a wearable device, according to one embodiment.
4A to 4B show an example of the appearance of a wearable device, according to an embodiment.
Figure 5 is a block diagram of a wearable device and an external electronic device connected to the wearable device, according to an embodiment.
Figure 6 shows an example of an operation related to electromyography of a wearable device, according to an embodiment.
FIG. 7 illustrates an example of a UI displayed by a wearable device to view one or more images, according to an embodiment.
FIGS. 8A to 8D illustrate an example of an operation performed by a wearable device based on a hand included in an image, according to an embodiment.
9A to 9C illustrate an example of an operation in which a wearable device receives input indicating that it adjusts color based on a hand included in an image, according to an embodiment.
Figure 10 shows an example of a UI displayed by a wearable device to adjust color, according to an embodiment.
FIG. 11 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment.
FIG. 12 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment.
FIG. 13 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment.

Hereinafter, various embodiments of this document are described with reference to the attached drawings.

The various embodiments of this document and the terms used herein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various changes, equivalents, and/or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar components. Singular expressions may include plural expressions, unless the context clearly dictates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as "first", "second", "first" or "second" can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another. The components are not limited. When a component (e.g., a first) component is said to be "connected (functionally or communicatively)" or "connected" to another (e.g., a second) component, it means that the component is connected to the other component. It may be connected directly to the component or may be connected through another component (e.g., a third component).

The term “module” used in this document includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted, or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or operations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a auxiliary processor 123, the auxiliary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, connected to the plurality of antennas by the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band), And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play Store??) or on two user devices. It can be distributed (e.g. downloaded or uploaded) directly between smartphones (e.g. smartphones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

FIG. 2 illustrates an example of a user interface (UI) provided by the wearable device 101, according to an embodiment. According to one embodiment, the wearable device 101 may have the form of glasses that can be worn on a user's body part (eg, head). The wearable device 101 may include a head-mounted display (HMD). For example, the housing of the wearable device 101 may be made of a flexible material such as rubber and/or silicone that is shaped to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). may include. For example, the housing of the wearable device 101 may include one or more straps capable of being twined around the user's head, and/or one or more temples attachable to the ears of the head. ) may include. According to one embodiment, the form of the wearable device 101 may be described with reference to FIGS. 3A to 3B and/or 4A to 4B.

According to one embodiment, the wearable device 101 may include one or more sensors for interacting with a user wearing the wearable device 101. For example, the wearable device 101 may include a touch sensor and/or one or more buttons to receive input from the user. The touch sensor may be disposed within the wearable device 101 to identify a part of the user's body (e.g., a fingertip) that is in contact with at least a portion of the surface of the housing of the wearable device 101. For example, the wearable device 101 uses at least one of a camera, a depth sensor, and/or a time-of-flight (ToF) sensor to detect an external object (e.g., the user's hand) adjacent to the wearable device 101. The external object 220) corresponding to can be recognized. In order to more accurately recognize a user's gesture based on an external object adjacent to the wearable device 101, the wearable device 101 may communicate with an external electronic device 210 that is attachable to the external object. To support communication between the wearable device 101 and the external electronic device 210, one or more pieces of hardware included in the wearable device 101 and the external electronic device 210 are described with reference to FIG. 5 .

Referring to FIG. 2 , an example of a wearable device 101 and an external electronic device 210 connected to the wearable device 101, according to an embodiment, is shown. The external electronic device 210 may include an electronic device, such as a smart watch, that is wearable on a user's designated body part 220 (e.g., a body part adjacent to the hand, such as the wrist). The embodiment is not limited to this, and the external electronic device 210 may include various types of electronic devices and/or smart accessories for measuring motion of body parts of the user. For example, external electronic device 210 may include a necklace, wrist strap, and/or smart clothing. According to one embodiment, the wearable device 101 is worn based on one or more sensors included in the wearable device 101 and/or a signal received from the external electronic device 210. A user's intent can be estimated. The wearable device 101 may execute at least one function supported by the wearable device 101 based on the result of estimating the user's intention. The at least one function provides a user wearing the wearable device 101 with a user experience based on augmented reality (AR), virtual reality (VR), and/or mixed reality. It can be executed based on

Referring to FIG. 2, according to one embodiment, the wearable device 101 is visualized as overlapping within the user's field of view (e.g., FoV 230) while the user is wearing the wearable device 101. A (visualized) UI (user interface) can be displayed. For example, wearable device 101 may include a display for displaying text, images, and/or video within the user's FoV 230. In one embodiment, the display may be configured to transmit light directed to the first side to a second side opposite to the first side. The second surface may be disposed toward the direction of both eyes of the user (eg, the direction of gaze) when the user is wearing the wearable device 101.

According to one embodiment, the wearable device 101 may identify an external object 220 included in the FoV 230 or adjacent to the wearable device 101. The external object 220 may be referred to as a subject and/or a tangible object. According to one embodiment, the wearable device 101 may include at least one camera disposed within the wearable device 101 to capture an external space that at least partially overlaps the FoV 230. Based on the image acquired from the at least one camera, the wearable device 101 may identify the external object 220. According to one embodiment, the wearable device 101 may conditionally activate the at least one camera based on information related to the external object 220 identified by the external electronic device 210. For example, the wearable device 101 initiates acquiring the image using at least one camera based on whether the motion of the external object 220 is identified based on the external electronic device 210. )can do. Because the wearable device 101 conditionally activates the at least one camera, power consumed by the at least one camera may be reduced. Because the power consumed by the at least one camera is reduced, the wearable device 101 operates for a longer period of time without requiring charging of a battery (not shown) included within the wearable device 101. can do. Because power consumed by the at least one camera is reduced, heat generated by driving the at least one camera can be reduced. Because the heat is reduced, heat transferred from the wearable device 101 including the at least one camera to the user wearing the wearable device 101 may be reduced. An exemplary operation in which the wearable device 101 receives and processes information related to the external object 220 from the external electronic device 210 is described with reference to FIG. 6 .

According to one embodiment, the wearable device 101 may display a screen in the user's FoV 230 that includes augmented or annotated information based on the environment shown to the user. . Referring to FIG. 2 , an example state in which the wearable device 101 displays visual objects 240 and 250 overlapping within the user's FoV 230 is shown, according to one embodiment. Visual objects 240 and 250 may be referred to as virtual (or imaginary) objects. At least one of the visual objects 240, 250 is an application executed by a processor of the wearable device 101, and/or based on input received from a user wearing the wearable device 101, the wearable device ( 101) may be displayed in the FoV 230. At least one of the visual objects 240 and 250 may be displayed by the wearable device 101 to support interaction between the wearable device 101 and the user wearing the wearable device 101.

According to one embodiment, the wearable device 101 may execute functions for browsing and/or editing at least one image stored in the wearable device 101. Referring to FIG. 2, the visual object 240 displayed by the wearable device 101 in the FoV 230 may indicate an image stored in the wearable device 101. An example of a UI that the wearable device 101 displays to a user to view a plurality of images stored in the wearable device 101 is described with reference to FIG. 7 . The wearable device 101 may use the visual object 240 displayed within the FoV 230 to provide a user experience as if the image were placed in an external space adjacent to the user wearing the wearable device 101. . According to one embodiment, the wearable device 101 may execute a function for editing the image based on the visual object 240 displayed in the FoV 230. The functionality may include distorting and/or modifying the image or adding one or more shapes to the image. The one or more shapes may include straight lines, curves, and/or polygons such as triangles or squares, and/or circles.

According to one embodiment, the wearable device 101 receives an input indicating adding one or more shapes to the visual object 240, which is an image stored in the wearable device 101, based on the motion of the external object 220. can be identified. An operation of the wearable device 101 adding one or more shapes to the visual object 240 based on the input, according to one embodiment, is described with reference to FIGS. 8A to 8D. The wearable device 101 may use the visual object 250 displayed in connection with the external object 220 to identify an input indicating selection of the color of the one or more shapes. Referring to FIG. 2 , the wearable device 101 may display a visual object 250 by overlapping it on at least a portion (eg, the palm) of the external object 220 shown through the FoV 230. The color of the visual object 250 may represent the color of one or more shapes to be displayed on the visual object 240, which is an image, based on the motion of the external object 220 measured by the wearable device 101. . An operation performed by the wearable device 101 based on an input indicating selection of the color of one or more shapes, according to an embodiment, is described with reference to FIGS. 9A to 9C and/or FIG. 10.

As described above, the wearable device 101 according to one embodiment may execute a function for editing an image based on the motion of the external object 220. The wearable device 101 may obtain one or more shapes to be added to the image based on the motion. The wearable device 101 may display a UI for adjusting the color of the one or more shapes using the visual object 250 displayed overlapping on the external object 220 within the FoV 230. The color displayed based on the visual object 250 may be determined by the motion of the body part (e.g., the user's hand, such as the external object 220) associated with the visual object 250 (e.g., the direction of the palm, and/ or the angle of joints in at least one finger). According to one embodiment, the wearable device 101 may acquire an electromyogram of the body part by communicating with an external electronic device 210 attached to the body part. The wearable device 101 can use the electromyogram to more accurately identify the point in time to identify the motion of the body part. For example, the wearable device 101 can more accurately identify the time point based on the amplitude difference of the electromyogram. The wearable device 101 may at least temporarily deactivate hardware (eg, a camera) for identifying motion of a body part within a viewpoint different from the identified viewpoint, thereby reducing power consumption by the hardware.

Below, with reference to FIGS. 3A to 3B and/or 4A to 4B, the structure of an embodiment of a wearable device 101 that can be worn on a user's head is exemplarily described.

FIG. 3A shows an example of a perspective view of a wearable device 101, according to one embodiment. FIG. 3B shows an example of one or more hardware disposed within the wearable device 101, according to one embodiment. The wearable device 101 of FIGS. 3A and 3B may be an example of the wearable device 101 of FIGS. 1 and 2 . Referring to FIG. 3A, the wearable device 101 according to one embodiment may include at least one display 350 and a frame 300 supporting the at least one display 350.

According to one embodiment, the wearable device 101 may be worn on a part of the user's body. The wearable device 101 may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device 101. For example, the wearable device 101 may, in response to a user's specified gesture obtained through the motion recognition camera 360-2 of FIG. 3B, display a gesture provided by at least one optical device 382, 384 of FIG. 3B. A virtual reality image may be displayed on at least one display 350.

According to one embodiment, at least one display 350 may provide visual information to the user. For example, at least one display 350 may include a transparent or translucent lens. At least one display 350 may include a first display 350-1 and/or a second display 350-2 spaced apart from the first display 350-1. For example, the first display 350-1 and the second display 350-2 may be placed at positions corresponding to the user's left eye and right eye, respectively.

Referring to FIG. 3B, at least one display 350 provides visual information transmitted from external light to the user through a lens included in the at least one display 350, and other visual information distinct from the visual information. can do. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display 350 may include a first surface 331 and a second surface 332 opposite to the first surface 331 . A display area may be formed on the second surface 332 of at least one display 350. When a user wears the wearable device 101, external light may be incident on the first surface 331 and transmitted through the second surface 332, thereby being transmitted to the user. For another example, at least one display 350 displays an augmented reality image in which a virtual reality image provided by at least one optical device 382 or 384 is combined with a real screen transmitted through external light, and a second display. It can be displayed in the display area formed on the surface 332.

In one embodiment, the at least one display 350 includes at least one waveguide 333, 334 that diffracts light emitted from the at least one optical device 382, 384 and transmits it to the user. may include. At least one waveguide 333 or 334 may be formed based on at least one of glass, plastic, or polymer. A nanopattern may be formed on the outside or at least a portion of the inside of at least one waveguide (333, 334). The nanopattern may be formed based on a polygonal and/or curved grating structure. Light incident on one end of the at least one waveguide (333, 334) may propagate to the other end of the at least one waveguide (333, 334) by the nanopattern. At least one waveguide 333 or 334 may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)), or a reflective element (e.g., a reflective mirror). . For example, at least one waveguide 333 or 334 may be disposed within the wearable device 101 to guide the screen displayed by at least one display 350 to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) generated within at least one waveguide (333, 334).

The wearable device 101 analyzes objects included in real-world images collected through a shooting camera (not shown), and selects a virtual object among the analyzed objects corresponding to the object that is the target of providing augmented reality. ) can be combined and displayed on at least one display 350. The virtual object may include at least one of text and images for various information related to the object included in the real image. The wearable device 101 can analyze objects based on multi-cameras, such as stereo cameras. For the object analysis, the wearable device 101 may perform time-of-flight (ToF), and/or simultaneous localization and mapping (SLAM), supported by multi-cameras. A user wearing the wearable device 101 can watch images displayed on at least one display 350.

According to one embodiment, the frame 300 may be made of a physical structure that allows the wearable device 101 to be worn on the user's body. According to one embodiment, the frame 300 is configured so that when the user wears the wearable device 101, the first display 350-1 and the second display 350-2 correspond to the user's left eye and right eye. It can be configured so that it can be located. The frame 300 may support at least one display 350. For example, the frame 300 may support the first display 350-1 and the second display 350-2 to be positioned at positions corresponding to the user's left and right eyes.

Referring to FIG. 3A , when the user wears the wearable device 101, the frame 300 may include an area 320 at least partially in contact with a portion of the user's body. For example, the area 320 of the frame 300 that is in contact with a part of the user's body is a part of the user's nose, a part of the user's ear, and a side part of the user's face that the wearable device 101 touches. Can include areas. According to one embodiment, the frame 300 may include a nose pad 310 that contacts a part of the user's body. When the wearable device 101 is worn by a user, the nose pad 310 may be in contact with a portion of the user's nose. The frame 300 may include a first temple 304 and a second temple 305 that are in contact with another part of the user's body that is distinct from the part of the user's body.

For example, the frame 300 includes a first rim 301 surrounding at least a portion of the first display 350-1 and a second rim 301 surrounding at least a portion of the second display 350-2. 302), a bridge 303 disposed between the first rim 301 and the second rim 302, a first bridge disposed along a portion of the edge of the first rim 301 from one end of the bridge 303 Pad 311, a second pad 312 disposed along a portion of the edge of the second rim 302 from the other end of the bridge 303, and a first pad that extends from the first rim 301 and is fixed to a portion of the wearer's ear. It may include a temple 304 and a second temple 305 that extends from the second rim 302 and is fixed to a portion of the ear opposite the ear. The first pad 311 and the second pad 312 may be in contact with a portion of the user's nose, and the first temple 304 and the second temple 305 may be in contact with a portion of the user's face and a portion of the ear. may come into contact with. The temples 304 and 305 may be rotatably connected to the rim through the hinge units 306 and 307 of FIG. 3B. The first temple 304 may be rotatably connected to the first rim 301 through a first hinge unit 306 disposed between the first rim 301 and the first temple 304. . The second temple 305 may be rotatably connected to the second rim 302 through a second hinge unit 307 disposed between the second rim 302 and the second temple 305. According to one embodiment, the wearable device 101 detects an external object that touches the frame 300 using a touch sensor, a grip sensor, and/or a proximity sensor formed on at least a portion of the surface of the frame 300. Gestures performed by, for example, a user's fingertip, and/or the external object may be identified.

According to one embodiment, the wearable device 101 may include hardware that performs various functions (eg, hardware described above based on the block diagram of FIG. 5). For example, the hardware includes a battery module 370, an antenna module 375, at least one optical device 382, 384, speakers (e.g., speakers 355-1, 355-2), and a microphone. It may include (e.g., microphones 365-1, 365-2, and 365-3), a light emitting module (not shown), and/or a printed circuit board 390. Various hardware may be placed within frame 300.

According to one embodiment, a microphone (e.g., microphones 365-1, 365-2, and 365-3) of the wearable device 101 is disposed in at least a portion of the frame 300 to acquire a sound signal. You can. A first microphone 365-1 disposed on the nose pad 310, a second microphone 365-2 disposed on the second rim 302, and a third microphone disposed on the first rim 301. Although microphones 365-3 are shown in FIG. 3B, the number and placement of microphones 365 are not limited to the embodiment of FIG. 3B. When the number of microphones 365 included in the wearable device 101 is two or more, the wearable device 101 uses a plurality of microphones disposed on different parts of the frame 300 to determine the direction of the sound signal. can be identified.

According to one embodiment, at least one optical device 382 or 384 may project a virtual object on at least one display 350 in order to provide various image information to the user. For example, at least one optical device 382, 384 may be a projector. The at least one optical device 382 or 384 may be disposed adjacent to the at least one display 350 or may be included within the at least one display 350 as part of the at least one display 350 . According to one embodiment, the wearable device 101 includes a first optical device 382 corresponding to the first display 350-1 and a second optical device (382) corresponding to the second display 350-2. 384) may be included. For example, the at least one optical device 382, 384 may include a first optical device 382 disposed at an edge of the first display 350-1 and an edge of the second display 350-2. It may include a second optical device 384. The first optical device 382 may transmit light to the first waveguide 333 disposed on the first display 350-1, and the second optical device 384 may transmit light to the second display 350-1. -2) Light can be transmitted to the second waveguide 334 disposed on the top.

In one embodiment, the camera 360 may include an imaging camera, an eye tracking camera (ET CAM) 360-1, and/or a motion recognition camera 360-2. The shooting camera, eye tracking camera 360-1, and motion recognition camera 360-2 may be placed in different positions on the frame 300 and perform different functions. The gaze tracking camera 360-1 may output data representing the gaze of the user wearing the wearable device 101. For example, the wearable device 101 may detect the gaze from an image including the user's pupils obtained through the gaze tracking camera 360-1. An example in which the gaze tracking camera 360-1 is positioned toward the user's right eye is shown in FIG. 3B, but the embodiment is not limited thereto, and the gaze tracking camera 360-1 is positioned solely toward the user's left eye. It may be placed towards, or towards both eyes.

In one embodiment, the capture camera may capture a real image or background to be matched with a virtual image to implement augmented reality or mixed reality content. The capturing camera may capture an image of a specific object that exists at a position where the user is looking (eg, FoV 230 in FIG. 2) and provide the image to at least one display 350. At least one display 350 displays information about a real image or background including an image of the specific object acquired using a shooting camera, and a virtual image provided through at least one optical device 382 or 384. One overlapping image can be displayed. In one embodiment, the imaging camera may be placed on the bridge 303 disposed between the first rim 301 and the second rim 302.

The gaze tracking camera 360-1 tracks the gaze of the user wearing the wearable device 101, thereby matching the user's gaze with the visual information provided on at least one display 350 to achieve more realistic augmentation. Reality can be realized. For example, when the user looks forward, the wearable device 101 may naturally display environmental information related to the user's front view on at least one display 350 at the location where the user is located. The eye tracking camera 360-1 may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera 360-1 may receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera 360-1 may be placed at positions corresponding to the user's left and right eyes. For example, the eye tracking camera 360-1 may be placed within the first rim 301 and/or the second rim 302 to face the direction in which the user wearing the wearable device 101 is located. You can.

The motion recognition camera 360-2 can provide a specific event on the screen provided on at least one display 350 by recognizing the movement of the entire or part of the user's body, such as the user's torso, hands, or face. . The gesture recognition camera 360-2 may recognize a user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display 350. The processor may identify a signal corresponding to the operation and perform a designated function based on the identification. In one embodiment, the motion recognition camera 360-2 may be disposed on the first rim 301 and/or the second rim 302.

The camera 360 included in the wearable device 101 is not limited to the eye tracking camera 360-1 and the motion recognition camera 360-2 described above. For example, the wearable device 101 uses the camera 360 positioned toward the user's FoV (e.g., FoV 230 in FIG. 2) to detect an external object included within the FoV (e.g., an external object in FIG. 2). Object 220) can be identified. Identification of an external object by the wearable device 101 is performed based on a sensor for identifying the distance between the wearable device 101 and the external object, such as a depth sensor and/or a time of flight (ToF) sensor. It can be. The camera 360 disposed toward the FoV may support an autofocus function and/or an optical image stabilization (OIS) function. For example, the wearable device 101 includes a camera 360 (e.g., a face tracking (FT) camera) disposed toward the face of a user wearing the wearable device 101 to obtain an image including the face. ) may include.

Although not shown, according to one embodiment, the wearable device 101 radiates light toward a subject (e.g., the user's eyes, face, and/or an external object within the FoV) captured using the camera 360. It may further include a light source (eg, LED). The light source may include an LED with an infrared wavelength. The light source may be disposed in at least one of the frame 300 and the hinge units 306 and 307.

According to one embodiment, the battery module 370 may supply power to electronic components of the wearable device 101. In one embodiment, the battery module 370 may be disposed within the first temple 304 and/or the second temple 305. For example, the battery module 370 may be a plurality of battery modules 370 . A plurality of battery modules 370 may be disposed on the first temple 304 and the second temple 305, respectively. In one embodiment, the battery module 370 may be disposed at an end of the first temple 304 and/or the second temple 305.

The antenna module 375 may transmit a signal or power to the outside of the wearable device 101, or may receive a signal or power from the outside. In one embodiment, the antenna module 375 may be disposed within the first temple 304 and/or the second temple 305. For example, the antenna module 375 may be placed close to one surface of the first temple 304 and/or the second temple 305.

The speaker 355 may output sound signals to the outside of the wearable device 101. The sound output module may be referred to as a speaker. In one embodiment, the speaker 355 may be placed within the first temple 304 and/or the second temple 305 to be placed adjacent to the ear of the user wearing the wearable device 101. For example, the speaker 355 is disposed within the first temple 304 to be adjacent to the user's right ear, and the second speaker 355-2 is disposed within the second temple 305 to be adjacent to the user's left ear. It may include a first speaker 355-1, which is disposed adjacent to the ear.

A light emitting module (not shown) may include at least one light emitting device. In order to visually provide information about the specific state of the wearable device 101 to the user, the light emitting module may emit light in a color corresponding to a specific state or emit light in an operation corresponding to the specific state. For example, when the wearable device 101 requires charging, it may emit red light at regular intervals. In one embodiment, the light emitting module may be disposed on the first rim 301 and/or the second rim 302.

Referring to FIG. 3B, according to one embodiment, the wearable device 101 may include a printed circuit board (PCB) 390. The PCB 390 may be included in at least one of the first temple 304 or the second temple 305. The PCB 390 may include an interposer disposed between at least two sub-PCBs. On the PCB 390, one or more hardware included in the wearable device 101 (eg, hardware shown by different blocks in FIG. 5) may be disposed. The wearable device 101 may include a flexible PCB (FPCB) for interconnecting the hardware.

According to one embodiment, the wearable device 101 includes a gyro sensor for detecting the posture of the wearable device 101 and/or the posture of a body part (e.g., head) of a user wearing the wearable device 101; It may include at least one of a gravity sensor and/or an acceleration sensor. The gravity sensor and acceleration sensor may each measure gravitational acceleration and/or acceleration based on designated three-dimensional axes (eg, x-axis, y-axis, and z-axis) that are perpendicular to each other. A gyro sensor can measure the angular velocity of each of designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device 101 may identify a user's motion and/or gesture performed to execute or stop a specific function of the wearable device 101 based on the IMU.

4A to 4B show an example of the appearance of the wearable device 101, according to an embodiment. The wearable device 101 of FIGS. 4A and 4B may be an example of the wearable device 101 of FIG. 2 . An example of the appearance of the first side 410 of the housing of the wearable device 101, according to one embodiment, is shown in FIG. 4A, and the second side 420 is opposite to the first side 410. ) can be shown in Figure 4b.

Referring to FIG. 4A, according to one embodiment, the first surface 410 of the wearable device 101 may have a form attachable to a user's body part (e.g., the user's face). . Although not shown, wearable device 101 may include a strap for securing on a part of the user's body, and/or one or more temples (e.g., first temple 304 in FIGS. 3A-3B, and/or It may further include 2 temples (305). A first display 350-1 for outputting an image to the left eye of the user's eyes, and a second display 350-2 for outputting an image to the right eye of the user's eyes, have a first surface 410. It can be placed on top. The wearable device 101 is formed on the first surface 410 and emits light (e.g., external light (e.g., external light) different from the light emitted from the first display 350-1 and the second display 350-2. Rubber or silicone packing may be further included to prevent interference due to ambient light.

According to one embodiment, the wearable device 101 includes a camera for photographing and/or tracking both eyes of a user adjacent to each of the first display 350-1 and the second display 350-2. It may include (360-3, 360-4). The cameras 360-3 and 360-4 may be referred to as ET cameras. According to one embodiment, the wearable device 101 may include cameras 360-5 and 360-6 for photographing and/or recognizing the user's face. The cameras 360-5 and 360-6 may be referred to as FT cameras.

Referring to FIG. 4B, on the second side 420, which is opposite to the first side 410 of FIG. 4A, a camera (e.g., cameras 360) for acquiring information related to the external environment of the wearable device 101 -7, 360-8, 360-9, 360-10, 360-11, 360-12)), and/or sensors (e.g., depth sensor 430) may be disposed. For example, the cameras 360-7, 360-8, 360-9, and 360-10 are used to recognize an external object (e.g., the external object 220 in FIG. 2) that is different from the wearable device 101. , may be placed on the second side 420. For example, using the cameras 360-11 and 360-12, the wearable device 101 may obtain an image and/or video to be transmitted to each of the user's eyes. The camera 360-11 will be disposed on the second side 420 of the wearable device 101 to acquire an image to be displayed through the second display 350-2 corresponding to the right eye among the two eyes. You can. The camera 360-12 may be disposed on the second side 420 of the wearable device 101 to acquire an image to be displayed through the first display 350-1 corresponding to the left eye of the two eyes. there is.

According to one embodiment, the wearable device 101 may include a depth sensor 430 disposed on the second surface 420 to identify the distance between the wearable device 101 and an external object. Using the depth sensor 430, the wearable device 101 stores spatial information (spatial information) about at least a portion of the FoV (e.g., FoV 230 in FIG. 2) of the user wearing the wearable device 101. Yes, you can obtain a depth map.

Although not shown, a microphone for acquiring sound output from an external object may be disposed on the second side 420 of the wearable device 101. The number of microphones may be one or more depending on the embodiment.

As described above, according to one embodiment, the wearable device 101 includes hardware (e.g., cameras 240-11, 240-12, and/or depth sensor) for identifying body parts including the user's hands. (430)). The wearable device 101 can identify gestures that appear due to motion of body parts. The wearable device 101 may provide a UI based on the identified gesture to a user wearing the wearable device 101. The UI may support functions for editing images and/or videos stored in the wearable device 101. The wearable device 101 may communicate with an external electronic device different from the wearable device 101 (eg, the external electronic device 210 of FIG. 2) in order to more accurately identify the gesture.

Below, with reference to FIG. 5 , an example of an operation in which the wearable device 101 communicates with the external electronic device according to an embodiment will be described.

FIG. 5 is a block diagram of a wearable device 101 and an external electronic device 210 connected to the wearable device 101, according to an embodiment. The wearable device 101 of FIG. 5 may be an example of the electronic device 101 of FIG. 1 and/or the wearable device 101 of FIG. 2 . For example, the wearable device 101 and the external electronic device 210 of FIG. 2 may include the wearable device 101 and the external electronic device 210 of FIG. 5 .

Figure 5 shows an example of a block diagram of a wearable device 101, according to one embodiment. The wearable device 101 of FIG. 5 may be connected to the external electronic device 210 wired or wirelessly. Referring to FIG. 5, according to one embodiment, the wearable device 101 includes at least one of a processor 120, a memory 130, a communication circuit 510, a camera 520, and a display 530. can do. The processor 120, memory 130, communication circuit 510, camera 520, and display 530 are electronic components such as a communication bus 505 within the wearable device 101. ) may be electrically and/or operably coupled to each other. Hardwares are operatively coupled so that a particular piece of hardware is controlled by another piece of hardware, and/or that another piece of hardware is controlled by the piece of hardware, such that a direct or indirect connection between the pieces of hardware is wired. It may mean established wirelessly or wirelessly. The type and/or number of hardware included in the wearable device 101 is not limited to the example of FIG. 5 .

According to one embodiment, the processor 120 of the wearable device 101 may include a hardware component for processing data based on one or more instructions. Hardware components for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), and/or a central processing unit (CPU). . The number of processors 120 may be one or more. For example, the processor 120 may have the structure of a multi-core processor, such as a dual core, quad core, or hexa core. The processor 120 of FIG. 5 may include the processor 120 of FIG. 1 .

According to one embodiment, the memory 130 of the wearable device 101 includes hardware components for storing data and/or instructions input to and/or output from the processor 120. can do. Memory 130 may include, for example, volatile memory such as random-access memory (RAM), and/or non-volatile memory such as read-only memory (ROM). You can. The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory includes, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi media card (eMMC). can do. The memory 130 of FIG. 5 may include the memory 130 of FIG. 1 .

Within memory 130, one or more instructions indicating operations and/or operations to be performed by processor 120 on data may be stored. A set of one or more instructions may be referred to as firmware, operating system, process, routine, sub-routine and/or application. For example, the wearable device 101 and/or the processor 120 executes a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, and/or application. , at least one of the operations of FIGS. 11 to 13 may be performed. Hereinafter, the fact that an application is installed on the wearable device 101 means that one or more instructions provided in the form of an application are stored in the memory 130 of the wearable device 101, and the one or more applications are installed in the wearable device 101. This may mean stored in an executable format (eg, a file with an extension specified by the operating system of the wearable device 101) by the processor 120.

According to one embodiment, the communication circuit 510 of the wearable device 101 includes hardware to support transmission and/or reception of electrical signals between the wearable device 101 and an external electronic device 210. can do. The communication circuit 510 may include, for example, at least one of a modem (MODEM), an antenna, and an optical/electronic (O/E) converter. The communication circuit 510 includes Ethernet, local area network (LAN), wide area network (WAN), wireless fidelity (WiFi), Bluetooth, bluetooth low energy (BLE), ZigBee, long term evolution (LTE), Based on various types of protocols, such as 5G NR (new radio), it can support transmission and/or reception of electrical signals. The communication circuit 510 of FIG. 5 may include the communication module 190 and/or the antenna module 197 of FIG. 1 .

According to one embodiment, the camera 520 of the wearable device 101 is an optical sensor (e.g., a charged coupled device (CCD) sensor, a complementary metal sensor (CMOS) that generates an electrical signal representing the color and/or brightness of light. It may include one or more oxide semiconductor sensors). A plurality of optical sensors included in the camera 520 may be arranged in the form of a 2-dimensional array. The camera 520 acquires electrical signals from each of the plurality of optical sensors substantially simultaneously, corresponds to the light reaching the optical sensors of the two-dimensional grid, and generates an image including a plurality of pixels arranged in two dimensions. can do. For example, photo data captured using the camera 520 may mean one image acquired from the camera 520. For example, video data captured using the camera 520 may mean a sequence of a plurality of images acquired from the camera 520 according to a designated frame rate. The wearable device 101 according to one embodiment may be arranged to face a direction in which the camera 520 receives light, and may further include a flash light for outputting light in this direction. The number of cameras 520 included in the wearable device 101 may be one or more, as described above with reference to FIGS. 3A to 3B and/or 4A to 4B.

According to one embodiment, the camera 520 of the wearable device 101 may include a depth camera. The depth camera may include a flash light and/or an infrared diode that emits light to the outside. The depth camera may include one or more infrared light sensors that detect the intensity of infrared light. The depth camera may measure the degree to which infrared light emitted from the infrared diode is reflected using the one or more infrared light sensors. In one embodiment, the degree to which the infrared light is reflected may be measured substantially simultaneously by a plurality of infrared light sensors in the depth camera. The depth camera may generate frame data including a depth value based on the degree to which infrared light measured by a plurality of infrared light sensors is reflected. The depth value may be related to the distance between the camera 520 and a subject captured by the camera 520.

In one embodiment, the FoV of the camera 520 is an area formed based on the view angle at which the lens of the camera 520 can receive light, and is an area corresponding to the image generated by the camera 520. We can respond. Hereinafter, a subject and/or an external object refers to an object that is included within the FOV of the camera 520 and is distinct from the wearable device 101. In one embodiment, the FoV of camera 520 may at least partially match the environment shown to the user through display 530, such as FoV 230 of FIG. 2 .

According to one embodiment, the display 530 of the wearable device 101 displays information visualized to the user (e.g., the screens of FIGS. 7, 8A to 8D, 9A to 9B, and/or 10). at least one of) can be output. The number of displays 530 included in the wearable device 101 may be one or more. For example, the display 530 may be controlled by the processor 120 and/or a graphic processing unit (GPU) to output visualized information to the user. The display 530 may include a flat panel display (FPD) and/or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and/or micro LED. The LED may include an organic LED (OLED). The display 530 of FIG. 5 may include the display module 160 of FIG. 1 .

In one embodiment, transmission of light may occur in at least a portion of the display 530. The wearable device 101 can provide a user experience related to augmented reality by providing a combination of light output through the display 530 and light passing through the display 530. As described above with reference to FIGS. 3A to 3B and/or 4A to 4B, according to one embodiment, the display 530 of the wearable device 101 is worn on a part of the user's body, such as the head. Within the state, it may have a structure to cover the entire user's field-of-view (FoV), or to radiate light toward the FoV.

Although not shown, the wearable device 101 may include other output means for outputting information in forms other than visual and auditory forms. For example, the wearable device 101 may include at least one speaker for outputting an audio signal, and/or a motor for providing haptic feedback based on vibration. Meanwhile, although shown based on different blocks, the embodiment is not limited thereto, and some of the hardware components shown in FIG. 5 (e.g., the processor 120, the memory 130, and at least a portion of the communication circuit 510) ) may be included in a single integrated circuit, such as a system on a chip (SoC).

Referring to FIG. 5 , an example block diagram of an external electronic device 210 connectable to the wearable device 101 is shown, according to an embodiment. The external electronic device 210 may include at least one of a processor 120, a memory 130, a communication circuit 510, or an electromyogram (EMG) sensor 540. Processor 120, memory 130, communication circuit 510, and EMG sensor 540 may be electrically and/or operatively connected to each other by a communication bus 505 within external electronic device 210. . The processor 120, memory 130, and communication circuit 510 in the external electronic device 210 are substantially similar to the processor 120, memory 130, and communication circuit 510 in the wearable device 101. It can perform a similar function. In the following description of the processor 120, memory 130, and communication circuit 510 in the external electronic device 210, the processor 120, memory 130, and communication circuit 510 of the wearable device 101 will be described. ), overlapping descriptions may be omitted to reduce repetition.

According to one embodiment, the wearable device 101, based on a communication link established between the communication circuit 510 of the wearable device 101 and the communication circuit 510 of the external electronic device 210, Information can be obtained from the device 210. The communication link may be established based on a short-range wireless communication protocol such as Bluetooth, Bluetooth low-energy (BLE), Wi-Fi Direct, and/or near field communication (NFC). For example, the communication link may be established based on pairing of a Bluetooth-based wearable device 101 and an external electronic device 210. The communication link is not limited to the short-range wireless communication protocol described above, but is a wired communication protocol such as universal serial bus (USB), and/or dedicated for communication of the wearable device 101 and external electronic device 210. (dedicated) Can be established by protocol.

According to one embodiment, the processor 120 of the external electronic device 210 may acquire an electromyogram of a body part using the EMG sensor 540. For example, while the external electronic device 210 is in contact with a user's body part (e.g., wrist), the external electronic device 210 uses the EMG sensor 540 to connect the nerves contained in the wrist to the muscles. It is possible to detect electrical signals applied to (e.g., muscles included in the hand). In one embodiment where the external electronic device 210 has a form factor that can be attached to a wrist, such as a smartwatch, the EMG sensor 540 is configured to: The external electronic device 210 adjacent to the wrist may be contacted with the skin of the wrist through one surface of the housing. The electromyogram acquired by the external electronic device 210 using the EMG sensor 540 may be a set of one or more numerical values representing the electrical signal. The processor 120 of the external electronic device 120 may acquire the electromyogram using the EMG sensor 540 repeatedly based on a designated cycle.

In one embodiment, the processor 120 of the external electronic device 210 may transmit a signal including information related to the electromyogram to the wearable device 101 using the communication circuit 510. The information is obtained from the EMG sensor 540 and may include sensor data indicating the amplitude, magnitude, and/or frequency of the electromyogram. The information is used by the wearable device 101 to identify the start and/or end of motion (e.g., motion of muscles included in the hand) occurring in the body part to which the external electronic device 210 is attached. can be processed.

According to one embodiment, the processor 120 of the wearable device 101 receives a signal from a designated body part (e.g., the outside of FIG. 2) from the external electronic device 210 through the communication circuit 510 of the wearable device 101. Sensor data including information related to the electromyogram of the object 220 may be obtained. Based on the information and identifying an amplitude difference that exceeds a specified threshold, the processor 120 may acquire at least one image using the camera 520. The processor 120 may identify the designated body part (eg, hand) based on the at least one image. Based on identifying the beginning of motion of the body part, for example, from the electromyogram obtained from an external electronic device 210 and having an amplitude difference exceeding a specified threshold, the processor 120 of the wearable device 101 ) can acquire an image including the specified body part using the camera 520.

According to one embodiment, the processor 120 of the wearable device 101, from an image obtained using the camera 520 and including a body part to which the external electronic device 210 is attached, determines the body part of the body part. Motion can be identified. In one embodiment where the external electronic device 210 is attached to the user's wrist, the wearable device 101 can identify the motion of the hand based on the angles of the joints included in the hand extending from the wrist from the image. You can. For example, the processor 120 may identify the posture of the hand based on joints included in the hand from the image. The hand posture may indicate the user's intention related to executing or stopping a specific function supported by the wearable device 101. According to one embodiment, the wearable device 101 may execute or stop the specific function based on the hand posture.

Referring to FIG. 5, according to one embodiment, an application executed by the wearable device 101 to recognize an external object such as a hand using the external electronic device 210 and/or the camera 520, and/ Or processes are shown. For example, hand database 550, image recognizer 560, gesture detector 570, and/or drawing application 580, stored within memory 130 of wearable device 101, may be stored in memory 130 of wearable device 101. As executed by the processor 120, one or more operations for recognizing the external object may be performed. For example, the processor 120 of the wearable device 101 may detect a sensor, transmitted from the external electronic device 210 through the communication circuit 510 of the wearable device 101, based on the execution of the gesture detector 570. Electromyography can be identified from the data.

In one embodiment, based on identifying the electromyogram with an amplitude difference increased above a specified threshold, the processor 120 of the wearable device 101 acquires an image containing an external object based on the camera 520. can do. The processor 120 of the wearable device 101 may identify a captured portion of a designated body part, such as a hand, from the acquired image based on the execution of the image recognizer 560. The processor 120 of the wearable device 101 may perform three-dimensional modeling of the hand captured within the identified portion of the image based on execution of the image recognizer 560. Based on the performance of the 3D modeling, the processor 120 of the wearable device 101 may obtain data indicating the position and/or direction of the joints and/or skeleton included in the hand. In the hand database 550, data representing the layout of the hand corresponding to a specific gesture may be stored based on the angle and/or direction of joints included in the hand. According to one embodiment, the processor 120 of the wearable device 101 may compare the acquired data with data stored in the hand database 550 to identify the gesture displayed by the hand captured within the portion. .

According to one embodiment, the processor 120 of the wearable device 101 detects the wearable device 101 when the amplitude difference of the electromyogram identified from the external electronic device 210 decreases below a specified threshold based on the execution of the gesture detector 570. The processor 120 of 101 may determine that the performance of the gesture by the user has been stopped. For example, when the processor 120 of the wearable device 101 identifies the gesture using the image recognizer 560, the time interval in which the electromyogram with an amplitude difference greater than the threshold specified by the gesture detector 570 is identified is identified. can be performed within The time section may be a time section in which a single gesture is performed.

According to one embodiment, the processor 120 of the wearable device 101 includes a drawing application 580 based on the gesture identified by execution of the gesture detector 570 and/or the image recognizer 560. You can select one or more features. The one or more functions may include adding one or more shapes to an image displayed through the display 530 of the wearable device 101 (e.g., the image represented by the visual object 240 in FIG. 2). You can. The one or more functions may include a function for setting the color of the one or more shapes.

As described above, according to one embodiment, the wearable device 101 can identify a gesture of a designated body part, such as a hand, using the external electronic device 210 and the camera 520. In one embodiment, the processor 120 uses at least one camera 520 based on the sensor data acquired from the external electronic device 210 and/or the electromyogram indicated by the sensor data. Because image acquisition is conditional (eg, whether the EMG has an amplitude difference exceeding a specified threshold), the time at which the camera 520 is activated may be reduced. Because the time at which the camera 520 is activated is reduced, power consumption and/or heat of the camera 520 and the wearable device 101 may be reduced.

Below, with reference to FIG. 6 , an example of an operation in which the wearable device 101 activates the camera 520 based on the electromyogram detected by the external electronic device 210 according to an embodiment will be described.

Figure 6 shows an example of an operation related to electromyography of a wearable device, according to an embodiment. The wearable device in FIG. 6 may be an example of the wearable device 101 in FIGS. 2 to 5 . According to one embodiment, a wearable device receives information related to the electromyogram from an external electronic device (e.g., the external electronic device 210 of FIG. 2) attached to a body part (e.g., wrist) of a user wearing the wearable device. can receive.

Referring to FIG. 6 , an example graph 610 of an electromyogram displayed by the information received by the wearable device from an external electronic device is shown, according to one embodiment. The x-axis of the graph 610 corresponds to the time axis, and the y-axis corresponds to the size and/or amplitude of the electromyogram. For example, the waveform shown in the graph 610 may represent the amplitude and/or intensity of the electromyogram measured by an external electronic device. An external electronic device may transmit a signal representing the amplitude and/or strength of the EMG detected from an EMG sensor (eg, EMG sensor 540 in FIG. 5) to the wearable device.

Referring to the graph 610 of FIG. 6, the electromyogram measured by an external electronic device may be changed based on the motion of a body part adjacent to the external electronic device. For example, each time a user moves their hand, the amplitude of the electromyogram, as measured by an external object attached to the hand, may increase. For example, when the user does not move his hand, the amplitude of the electromyogram, as measured by an external object attached to the hand, may decrease to zero. For example, based on the amplitude of the electromyogram that increases each time the hand moves, the wearable device that receives the electromyogram from an external electronic device receives the electromyogram at a first time when the hand motion starts, or at a second time when the motion stops. You can obtain at least one of the following. For example, by comparing a specified threshold based on the error range of the EMG sensor of the external electronic device (e.g., an error range of +5% to -5%) with the amplitude difference of the electromyogram received from the external electronic device, the wearable device may identify at least one of the first viewpoint or the second viewpoint. The wearable device may determine the time interval between the first viewpoint and the second viewpoint as the interval in which the user performed the gesture using the hand.

Referring to FIG. 6 , time sections 621, 622, 623, 624, 625, and 626 identified by the wearable device using the electromyogram represented by the graph 610 are shown. The time sections 621, 622, 623, 624, 625, and 626 may be time sections in which a gesture of the body part for which the EMG was detected was performed. The time sections 621, 622, 623, 624, 625, and 626 may be time sections in which the EMG detected by the wearable device has an amplitude difference that exceeds a specified threshold. The wearable device may acquire images related to the body part using a camera (eg, camera 520 in FIG. 5) in each of the time sections 621, 622, 623, 624, 625, and 626. The image may include a depth map obtained based on a depth camera. Referring to FIG. 6, images 631, 632, 633, 634, 635, and 636 acquired by the wearable device using a camera in each of the time sections 621, 622, 623, 624, 625, and 626 are shown as examples. It is shown.

According to one embodiment, the wearable device captures the body from the images 631, 632, 632, 633, 634, 635, and 636 acquired in each of the time sections 621, 622, 623, 624, 625, and 626. Information indicating the posture, shape, and/or direction of the area may be obtained. The information may include 3D modeling information of body parts including the hand. In order to obtain the above information, the wearable device may use a pre-stored database such as the hand database 550 of FIG. 5. The wearable device generates a gesture indicated by the motion of the body part in each of the time intervals 621, 622, 623, 624, 625, and 626, based on information indicating the posture, shape, and/or direction of the body part. The type can be identified.

For example, based on information identified from the images 631 and 633, the wearable device may identify that all joints included in the hand are included within a specified angle range including a straight angle. Based on the images 631 and 633, the wearable device may determine that the hand is in a fully unfolded posture in each of the time intervals 621 and 623. Based on the area and/or shape of the palm included in the images 631 and 633, the wearable device may determine that the direction of the hand is toward the camera in each of the time intervals 621 and 623. For example, based on information identified from the image 632, the wearable device may identify that, among joints included in the hand, joints within the index finger have an angle that is different from the specified angle range. . Based on the image 632, the wearable device may determine that the hand posture is such that the thumb is in a bent posture within the time interval 622. Similarly, based on the image 634, the wearable device may determine that the hand posture is such that the thumb is bent toward external space within the time interval 624. Based on the image 635, the wearable device may determine that the hand posture is in a posture in which the index finger is spread out and the other fingers except the index finger are fully bent within the time interval 625. Based on the image 636, the wearable device may determine that the hand posture is fully bent within the time interval 626. Based on the image 636, the wearable device may determine that the direction of the palm included in the hand has a difference of less than or equal to the direction of the camera and the specified angle range.

According to one embodiment, the wearable device determines the posture and/or direction of the hand in each of the time intervals 621, 622, 623, 624, 625, and 626, based on obtaining information including the posture and/or direction of the hand. /Or a gesture corresponding to the direction may be identified. The wearable device may execute a function corresponding to the identified gesture. The wearable device may identify the gesture, and/or the function from the posture and/or the direction based on information stored in the hand database 550 of FIG. 5 . For example, the wearable device may match the posture and/or orientation stored in the hand database 550 of FIG. 5 to images acquired from a camera (e.g., images 631, 632, 632, 633, 634, 635, 636). ) can be compared with the posture, and/or direction identified by. The wearable device may obtain similarities by comparing the image acquired from the camera and different information stored in the hand database 550. Among the similarities, the wearable device may determine the gesture corresponding to the information with the highest similarity as a gesture performed by a body part (eg, hand) included in the image.

As described above, according to one embodiment, the wearable device captures at least one device within a time section (e.g., time sections 621, 622, 623, 624, 625, and 626) divided by the amplitude difference of the electromyogram. Based on the image (e.g., images 631, 632, 633, 634, 635, 636), a body part included in the at least one image and a gesture performed by the body part can be identified. . When a wearable device acquires the at least one image that includes at least a portion of the user's FoV (e.g., FoV 230 in FIG. 2), the body part included in the at least one image appears on the display of the wearable device. It can be displayed to the user through According to one embodiment, a wearable device may display a visual object in association with a body part shown through the display. The visual object may be associated with a gesture identified by the wearable device. The visual object may be displayed for editing of the image displayed within the FoV by a wearable device.

Below, with reference to FIG. 7 , an example of an operation in which a wearable device receives an input indicating that an image to be edited is selected based on the visual object is described, according to an embodiment.

FIG. 7 illustrates an example of a UI displayed by the wearable device 101 to view one or more images, according to an embodiment. The wearable device 101 of FIG. 7 may be an example of the wearable device 101 of FIGS. 2 to 5 and the wearable device of FIG. 6 . For example, the wearable device 101 of FIG. 2 may include the wearable device 101 of FIG. 7 . The external electronic device 210 of FIG. 2 may include the external electronic device 210 of FIG. 7 .

Referring to FIG. 7 , an example state 710 is shown in which the wearable device 101 displays a UI within the FoV 230 using a display (e.g., display 530 in FIG. 5), according to one embodiment. do. The UI may be displayed based on the execution of a designated application installed in the wearable device 101, such as a gallery application, for viewing one or more images and/or videos stored in the wearable device 101. there is. The wearable device 101 may display overlapping visual objects 712, 714, and 240 on the FoV 230 based on execution of the specified application.

According to one embodiment, in a state 710 where the wearable device 101 displays a UI for viewing images and/or videos stored within the wearable device 101, the visual displayed within the FoV 710 Objects 712, 714, and 240 may represent either the images and/or the videos. For example, each of visual objects 712, 714, and 240 may include thumbnails of different images stored within wearable device 101.

According to one embodiment, the wearable device 101 may receive an input indicating selection of one of the visual objects 712, 714, and 240 within the state 710. The input may be identified based on a gesture performed by a user wearing the wearable device 101. The gesture is related to a body part 220 acquired through a camera of the wearable device 101 (e.g., camera 520 in FIG. 5) and/or an external electronic device 210 connected to the wearable device 101. Based on the information, it can be identified. For example, the wearable device 101 may obtain information representing the electromyogram of the body part 220 from the external electronic device 210 connected to the wearable device 101. Based on the electromyogram, the wearable device 101 can identify the start or end of the gesture based on the body part 220. After the gesture starts, the wearable device 101 may acquire an image including the body part 220 using a camera. Based on identifying the posture and/or orientation of the body part 220 from the image, the wearable device 101 may identify the gesture performed by the body part 220 .

Referring to FIG. 7 , within state 710, according to one embodiment, wearable device 101 may identify the posture and orientation of body part 220 from an image including an external object 220. there is. For example, wearable device 101 may determine that a hand included in body part 220 includes an extended index finger, and that the index finger and other fingers have a bent posture, within state 710. there is. The wearable device 101 may determine that the gesture performed by the body part 220 is a gesture pointing to a point 730 within the FoV 710, based on the direction the index finger is facing. Based on identifying the gesture, wearable device 101 may identify an input indicating selection of a visual object 240 that overlaps point 730 within FoV 230. The input may include an input for specifying one of the images stored in the wearable device 101. Although an embodiment in which the wearable device 101 receives input for viewing and selecting images has been described, the embodiment is not limited thereto. The wearable device 101 may execute a function to transmit an image selected by the user (eg, an image corresponding to the visual object 240) based on the gesture of the body part 220.

According to one embodiment, the wearable device 101 may display an image corresponding to the visual object 240 in the FoV 230 in response to an input indicating selection of the visual object 240. The wearable device 101 may at least temporarily stop displaying the visual objects 712 and 714 based on the input. While the wearable device 101 displays an image corresponding to a visual object 240 in the FoV 230, the size of the image displayed in the FoV 230 is adjusted based on the gesture of the body part 220. Can be adjusted (e.g. enlarged or reduced).

Within state 710 of FIG. 7, in response to an input indicating selection of an image corresponding to visual object 240, wearable device 101 may display a UI for viewing and/or editing the image. You can. According to one embodiment, the wearable device 101 provides functions for editing the image (e.g., mixing colors, and/or adding a shape with the color) based on the UI and one hand of the user. function) can be executed. Hereinafter, with reference to FIGS. 8A to 8D, 9A to 9C, and/or 10, the wearable device 101 performs based on the UI displayed for viewing and/or editing the image. An example of operation is described.

FIGS. 8A to 8D illustrate an example of an operation performed by the wearable device 101 based on a hand included in an image, according to an embodiment. The wearable device 101 of FIGS. 8A to 8D may be an example of the wearable device 101 of FIGS. 2 to 7 . For example, states 810, 820, 830, and 840 of FIGS. 8A to 8D include an input indicating selection of an image corresponding to visual object 240, based on state 710 of FIG. 7. It may include the status of the wearable device 101 after reception. For example, the states 810 , 820 , and 830 may include a state of the wearable device 101 for executing a function related to editing an image corresponding to the visual object 240 .

The handles 812-1, 812-2, 812-3, 812-4, 812-5, and 812-6 displayed by the wearable device 101 on the boundary line of the visual object 240 are the visual object 240. It may be mapped to different functions associated with object 240. For example, the handle 812-1 may correspond to a function for storing an image based on the visual object 240 shown through the FoV 230. For example, the handle 812-2 may correspond to a function for resetting changes applied to the image of the visual object 240 viewed through the FoV 230. For example, the handle 812-3 may correspond to a function for rotating the visual object 240 displayed through the FoV 230 along a specified axis (eg, y-axis). For example, the handle 812-4 may correspond to a function for removing an image corresponding to the visual object 240. For example, handle 812-5 rotates visual object 240 viewed through FoV 230 along another axis (e.g., x-axis) that is different from the specified axis corresponding to handle 812-3. It can respond to functions for: For example, the handle 812-6 may correspond to a function for rotating the visual object 240 viewed through the FoV 230 along another axis (eg, z-axis) that is different from the above axes. According to one embodiment, the wearable device 101 provides an input indicating selection of one of the handles 812-1, 812-2, 812-3, 812-4, 812-5, and 812-6. Based on this, the function corresponding to the handle selected by the input can be executed.

Referring to FIG. 8A , an example of a state 810 in which the wearable device 101 receives an input indicating drawing a curve 815 on an image corresponding to a visual object 240 is shown, according to an embodiment. . The wearable device 101 may identify the input based on the body part 220 based on information representing the motion of the body part 220 including the hand. The information is obtained, for example, by electromyography of the body part 220 measured by the external electronic device 210, and/or by the camera of the wearable device 101 (e.g., camera 520 in FIG. 5). It may be related to the posture and/or orientation of the body part 220 identified within the image. The wearable device 101 may obtain the information based on the operation described above with reference to FIG. 6 .

Referring to FIG. 8A , based on identifying a body part 220 whose posture points to a point within the visual object 240 within the FoV 230, the wearable device 101 displays a curve on the visual object 240. You can identify gestures for drawing. The posture may be identified based on the shape of the back of the hand and/or the index finger captured in the image containing the body part 220. In response to identifying the gesture, wearable device 101 may display visual object 815 representing a curve, superimposed on visual object 240 displayed within FoV 230. The curve represented by the visual object 815 is the trajectory of a portion of the body part 220 (e.g., the tip of the index finger) during the time interval in which the wearable device 101 identified the gesture of the body part 220. can respond.

According to one embodiment, the wearable device 101 identifies a gesture for drawing a curve, such as a visual object 815, based on an image including the back of the hand and at least one finger extending from the back of the hand. You can. The shape of the visual object 815 that the wearable device 101 displays in response to the gesture as superimposed on the visual object 240 within the FoV 230 includes the number of fingers obtained from the image, and/ Or it may be related to type. For example, in the state 810 of identifying the index finger extending from the back of the hand among a plurality of fingers from the image, the wearable device 101 displays a visual object including a solid line based on the trajectory of the index finger ( 815) can be displayed. For example, when identifying a little finger extending from the back of the hand among a plurality of fingers from an image, the wearable device 101 may display a dashed line based on the trajectory of the little finger. For example, when identifying a thumb and an index finger extending from the back of the hand among a plurality of fingers from an image, the wearable device 101 may display an arrow based on the trajectory of the index finger.

Referring to FIG. 8A, based on a gesture for drawing a curve on the visual object 240, within a state in which a visual object 815 representing the curve is displayed, the wearable device 101 displays the visual object 815. A line (e.g., border line) surrounding can be displayed. On this line, the wearable device 101 includes handles 815-1, 815-2, 815-3, 815-4, 815-5, and 815-6 for executing functions related to the visual object 815. can be displayed. The functions mapped to the handles 815-1, 815-2, 815-3, 815-4, 815-5, and 815-6 are the handles 812-1, 812-2, 812-3, 812- It can correspond to each of the functions mapped to 4, 812-5, and 812-6). For example, the handle 815-1 may correspond to a function for storing a visual object 815 shown through the FoV 230 and an image changed based on a combination of the visual object 240. For example, handle 815-2 may correspond to a function for resetting changes associated with visual object 815 viewed through FoV 230. For example, the handle 815-3 may correspond to a function for rotating the visual object 815 displayed through the FoV 230 based on the y axis. For example, handle 815-4 may correspond to a function for removing visual object 815. For example, the handle 815-5 may correspond to a function for rotating the visual object 815 based on the x-axis. For example, the handle 815-6 may correspond to a function for rotating the visual object 815 based on the z-axis.

According to one embodiment, the wearable device 101 identifies the back of the hand from an image acquired from a camera and displays it through the FoV 230 based on the number and/or type of fingers extending from the back of the hand. At least one of the visual objects 240 and 815 may be changed. For example, when two fingers (e.g., index finger and middle finger) extending from the back of the hand toward the visual object 815 along parallel directions are identified from the image, the wearable device 101 identifies the visual object 815 ) can be moved within the FoV (230).

Referring to FIG. 8B , after displaying visual object 815 based on state 810, wearable device 101 receives input indicating to remove at least a portion of visual object 815 displayed within FoV 230. An example of one state 820 is shown. The wearable device 101 may identify the time period in which the input was performed based on the electromyogram of the body part 220 identified based on the external electronic device 210. Within the time interval, the wearable device 101 may acquire at least one image including the body part 220. The wearable device 101 may identify a body part 220 including a hand whose palm is completely obscured by the back of the hand from the at least one image in state 820 . In the state 820, the wearable device 101 is configured such that the direction of the palm included in the body part 220 corresponds to the direction of the user's gaze, and the fingers included in the body part 220 are perpendicular to the direction of the palm. It can be identified that it is facing a different direction. For example, the wearable device 101 may identify, within the at least one image, a body part 220 having a shape and/or posture indicating B in the American manual alphabet. . The posture and/or orientation of the body part 220 illustrated in FIG. 8B may represent a gesture for removing at least a portion of the visual object 815 displayed within the FoV 230.

Referring to FIG. 8B, according to one embodiment, the wearable device 101 may remove at least a portion of the visual object 815 displayed within the FoV 230 based on identifying the gesture of the body part 220. You can. For example, the wearable device 101 may create a region formed based on a part of the body part 220 (e.g., hand, etc.) moved along the direction 822 during the time interval in which the gesture of the body part 220 is identified. (824) can be identified. The wearable device 101 may remove a portion of the visual object 815 that overlaps the identified area 824 . For example, a gesture received by the wearable device 101 in state 820 of FIG. 8B may include a function for removing a visual object 815 and/or a shape added by the user (e.g., an eraser). ) can correspond to functions with designated names such as ).

Referring to FIG. 8C, after state 820, an example of a state 830 in which the wearable device 101 receives input indicating that it stores the edited image based on the visual object 240 displayed within the FoV 230 is It is shown. The wearable device 101 may determine whether to combine one or more shapes on the image corresponding to the visual object 240 based on the images 832 and 836 including the body part 220. As shown in images 832 and 836, when a body part 220 including a clenched fist is identified, the wearable device 101 determines the orientation of a portion of the body part 220 (e.g., the back of the hand). , it is possible to determine whether to combine one or more shapes on the image corresponding to the visual object 240.

Within the state of acquiring image 832 of FIG. 8C, wearable device 101 can identify from image 832 a body part 220 including the back of the hand that has a direction toward the camera of wearable device 101. there is. The posture and/or shape of the body part 220 included in the image 832 may indicate that the result of editing the image corresponding to the visual object 240 is not stored in the wearable device 101. For example, within the state of acquiring the image 832, the wearable device 101 may not combine one or more shapes received from the user, such as the visual object 815, to the image corresponding to the visual object 240. You can. For example, the visual object 815 received through the state 830 may not be stored in the wearable device 101.

In the state in which the image 836 of FIG. 8C is acquired, the wearable device 101 displays a body part 220, including the back of the hand, facing in a direction perpendicular to the direction facing the camera of the wearable device 101 from the image 836. can be identified. The posture and/or shape of the body part 220 included in the image 836 may indicate that the result of editing the image corresponding to the visual object 240 is stored in the wearable device 101. For example, within the state of acquiring the image 836, the wearable device 101 may combine one or more shapes received from the user, such as the visual object 815, to the image corresponding to the visual object 240. . For example, a visual object 815 displayed within FoV 230 via state 830 may be stored within an image corresponding to visual object 240 . The embodiment is not limited to the above example, and for example, the wearable device 101 may include information for reproducing the visual object 815 and metadata (e.g., xml (extended marked-up) corresponding to the image. It can be saved in a file based on language. The information and/or the stored image may be transmitted to another electronic device that is different from the wearable device 101. For example, the wearable device 101 may receive the information and/or the image from another electronic device connected through a network service such as SNS (social network service).

The body part 220 included in at least one of the images 832 and 836 of FIG. 8C is used to view one or more images and/or one or more videos stored in the wearable device 101 from state 830. It may represent a gesture for switching to a state (e.g., state 710 in FIG. 7). For example, from the body part 220 included in the image 832, the wearable device 101 may have a function to discard the result of editing the image corresponding to the visual object 240, and the state of FIG. A gesture for executing the switching function can be identified (710). For example, from the body part 220 included in the image 836, the wearable device 101 has the function of storing the result of editing the image corresponding to the visual object 240, and the state 710 of FIG. You can identify the gesture to execute the function to switch to. For example, when at least one of the images 8732 and 836 is acquired from a camera, the wearable device 101 stores an image stored in the wearable device 101 within the FoV 230, such as state 710 of FIG. One or more thumbnails may be displayed corresponding to one or more images and/or one or more videos.

As described above with reference to FIGS. 8A to 8C, according to one embodiment, the wearable device 101 converts a shape based on the trajectory of a body part 220, such as a hand, into an image corresponding to the visual object 240. can be combined with The examples are not limited thereto. For example, the wearable device 101 may, based on a gesture performed by the body part 220, add to the image another pre-stored image (e.g., at least one of designated images referred to as icons and/or stickers). ) can be combined. Referring to FIG. 8D, within state 820 of FIG. 8B, a state 840 is shown in which the wearable device 101 has received an input indicating display of a list of pre-stored images.

Referring to FIG. 8D, according to one embodiment, the wearable device 101 is identified from the external electronic device 210 and, based on identifying the electromyogram, having an increased amplitude difference exceeding a specified threshold, the wearable device 101 An image including a body part 220, such as a hand, can be acquired using the camera 101. Referring to FIG. 8D, based on identifying the back of the hand in the image and the two fingers extending from the back of the hand and perpendicular to each other (e.g., the index finger and the thumb), the wearable device 101 determines the FoV ( 230), a visual object 842 containing a list of pre-stored images may be displayed within the portion where the fingers are identified. The list may include pre-stored images arranged based on a two-dimensional grid. Although an example of identifying a gesture based on the thumb and index finger being perpendicular to each other is shown, the embodiment is not limited thereto.

According to one embodiment, the wearable device 101 may receive an input indicating adding a predetermined image to the visual object 240 based on the visual object 842 in the state 840 . For example, an input indicating that the wearable device 101 selects an image from a list of pre-stored images included in the visual object 842 based on the posture and/or orientation of the hand obtained from the image. can receive. Based on the input, the wearable device 101 may display the pre-stored image selected by the input in an overlapping manner on the visual object 240.

As described above, according to one embodiment, the wearable device 101 may perform an operation to edit an image corresponding to the visual object 240 displayed within the FoV 230. The wearable device 101 may perform the above operation based on the gesture of the body part 220. The wearable device 101 may receive a gesture for adjusting the color of one or more shapes combined with the image based on the gesture. The wearable device 101 may display a UI for adjusting the color of the one or more shapes in connection with the body part 220 shown through the FoV 230. For example, the UI for adjusting the color may be displayed in an overlapping manner on a designated part (eg, palm) of the body part 220. The color may be adjusted based on the motion of other parts (e.g., fingers) of body part 220 that are different from the designated part.

Below, with reference to FIGS. 9A to 9C and/or FIG. 10 , an example of a UI displayed by the wearable device 101 to adjust color according to an embodiment will be described.

FIGS. 9A to 9C illustrate an example of an operation in which the wearable device 101 receives an input indicating that the wearable device 101 adjusts a color based on a hand included in an image, according to an embodiment. The wearable device 101 of FIGS. 9A to 9C may be an example of the wearable device 101 of FIGS. 2 to 7 and/or 8A to 8D. For example, the wearable device 101 of FIG. 2 may include the wearable device 101 of FIGS. 9A to 9C. For example, the states of FIGS. 9A and 9B represent the state of the wearable device 101 after receiving an input indicating selection of an image corresponding to the visual object 240, based on the state 710 of FIG. 7. Can include status. For example, the states of FIGS. 9A and 9B may include a state of the wearable device 101 for executing a function related to editing an image corresponding to the visual object 240.

Referring to FIG. 9A , an example of a state 910 in which the wearable device 101 has identified a body part 220 having a specified posture and a specified direction is shown, according to an embodiment. In state 910, the wearable device 101 may acquire at least one image including the body part 220 using a camera (eg, camera 520 in FIG. 5). Based on the at least one image, the wearable device 101 may identify a body part 220 having a designated posture with all fingers spread. Within state 910, wearable device 101 may identify a body part 220 that includes the palm having a direction toward the camera. The wearable device 101 may identify the palm and/or the direction of the palm based on whether one or more palm marks included in the palm are identified from the image. The specified direction may be set so that the direction of the palm substantially coincides with the direction toward the camera. For example, within state 910, wearable device 101 may identify a body part 220 that includes a hand fully extended toward the user. The body part 220, which has a posture including a hand fully spread out toward the user, executes a function to adjust the color of the figure to be drawn based on a different posture (e.g., the posture in state 810 of FIG. 8A). You can respond to gestures to do this. The function may be expressed with a designated name, such as a palette.

Referring to FIG. 9A, based on acquiring at least one image including a body part 220 having a specified posture and a specified direction indicating the posture and/or direction of the hand stretched out toward the camera, the wearable device ( 101) may display a visual object 901 to indicate a color in connection with the body part 220 shown through the display. Referring to example state 910 of FIG. 9A , visual object 901 may be displayed overlaid on a portion of body part 220 visible through FoV 230 . For example, the wearable device 101 may display a visual object 901 among the body parts 220 shown through the FoV 230 by superimposing it on the palm facing the camera. For example, based on identifying, from at least one image, a body part 220 that includes a palm spread out toward the camera, wearable device 101 may display a visual object 901 to represent a color. there is.

According to one embodiment, the wearable device 101 is associated with a body part 220 and includes one or more visual objects 902, 903, 904, 905, and 906 for adjusting the color displayed by the visual object 901. can be displayed. Referring to the example state 910 of FIG. 9A , with visual objects 901 overlapping each of the fingers extending from the overlapping palm, the wearable device 101 displays visual objects 902, 903, 904, 905, 906) can be displayed. For example, wearable device 101 displays, on each of the fingers extending from the palm, different visual objects 902, 903, 904, 905, 906 representing different parameters related to the color of visual object 901. can do.

In one embodiment, the parameters represented by the visual objects 902, 903, 904, 905, and 906 may be factors for forming the color of the visual object 901. For example, the parameters may include the brightness of each of the primary colors included in the color, such as red, green, and blue. The parameters may include transparency of the color. The transparency can be expressed by a numeric value, such as an alpha value. The parameters may include the thickness of the line with the color of the visual object 901. The parameters are not limited to the examples above. For example, the parameters may include the luminance of primary colors in a color space based on subtractive mixing, such as CMYK (cyan magenta yellow key plate). For example, the parameters may include values for representing the color of the visual object 901 based on a color space based on luminance and chrominance components, such as YCbCr.

Within the states 910 and 920 of FIGS. 9A and 9B , visual objects 902, 903, 904, 905, and 906 that the wearable device 101 displays overlapping within the FoV 230, according to one embodiment. ) may be mapped to different parameters related to the color of the visual object 901. For example, the wearable device 101 may display a visual object 903 indicating the brightness of red included in the color overlaid on the index finger shown through the FoV 230. For example, the wearable device 101 may display a visual object 904 indicating the brightness of green included in the color, overlaid on the middle finger shown through the FoV 230. For example, the wearable device 101 may display a visual object 905 representing the brightness of blue included in the color overlaid on the ring finger shown through the FoV 230. For example, the wearable device 101 may display a visual object 906 indicating the transparency (or alpha value) of the color overlaid on the little finger shown through the FoV 230. For example, the wearable device 101 is a visual object that represents the thickness of the curve added by the gesture of the body part 220 overlapping with the visual object 901, overlapping with the thumb shown through the FoV 230. (902) can be displayed. The parameters corresponding to the visual objects 902, 903, 904, 905, 906 are not limited to the above example and may be personalized by the user wearing the wearable device 101. For example, the wearable device 101 may obtain parameters corresponding to each of the visual objects 902, 903, 904, 905, and 906 based on information adjustable by the user.

According to one embodiment, the wearable device 101 displays parameters related to the color of the visual objects 901 based on the shape and/or size of each of the visual objects 902, 903, 904, 905, and 906. You can visualize it. For example, the length d1 of the visual object 903 may be related to a parameter corresponding to the visual object 903 (eg, brightness of red). The wearable device 101 may change the length d1 of the visual object 903 based on the ratio between the length of the index finger shown through the FoV 230 and the brightness of red. For example, the ratio between the length of the index finger shown through FoV 230 and the length d1 of the visual object 903 is the maximum value that the brightness of red can have (e.g., 255), and the visual It may match the ratio between the brightness of red within the color of the object 901. Similarly, the wearable device 101 determines the length of the finger shown through the FoV 230 and the parameter represented by the visual object superimposed on the finger among the visual objects 902, 903, 904, 905, and 906. Based on this, the size of the visual object can be adjusted. For example, the length of the visual object 906 may indicate transparency adjusted between 0% and 100%. For example, the length of the visual object 902 may represent a thickness that is adjusted between 1 point and a specified upper limit (eg, 100 points).

Similar to what was described above with reference to the visual objects 902, 903, 904, 905, and 906, the wearable device 101 displays the color of the visual object 901 superimposed on the palm as seen through the FoV 230, The size and/or transparency may be adjusted based on parameters corresponding to each of the visual objects 902, 903, 904, 905, and 906. For example, the wearable device 101 may adjust the size of the visual object 901 based on the thickness corresponding to the visual object 902. For example, the wearable device 101 may adjust the transparency of the visual object 901 based on the transparency expressed through the length of the visual object 906. The color of the visual object 901 may be a combination of red, green, and blue with luminance corresponding to the lengths of the visual objects 903, 904, and 905.

Within the state 910 in which the visual object 901 is displayed, according to one embodiment, the wearable device 101 identifies a change in the posture of the body part 220 based on the image including the body part 220. can do. The wearable device 101 may change the color of the visual object 901 based on identifying the change in posture. The change in posture identified by the wearable device 101 may be caused by a motion of a joint included in the body part 220 (eg, joints included in a finger).

Within state 910, according to one embodiment, wearable device 101 may monitor the angles of fingers included in body part 220. Referring to FIG. 9B , a state 920 is shown after a state 910 in which a visual object 901 for color adjustment is overlapped on a body part 220. In state 920, it is assumed that the wearable device 101 has identified a gesture of bending the index finger among the fingers included in the body part 220. For example, wearable device 101 may identify the gesture from an image containing body part 220 within state 920 . From the image, the wearable device 101 can identify the angle of at least one of the fingers included in the body part 220.

According to one embodiment, the wearable device 101 displays visual objects 902, 903, 904, 905, and 906 overlaid on the fingers based on the angle of at least one of the fingers included in the body part 220. ), at least one parameter corresponding to at least one of the following can be changed. The wearable device 101 may change the display of at least one of the visual objects 902, 903, 904, 905, and 906 based on the angle of at least one of the fingers. The wearable device 101 may change the color and/or transparency of the visual object 901 based on the at least one parameter adjusted based on the angle of at least one of the fingers. As in state 920 of FIG. 9B, based on identifying a gesture of bending the index finger among the fingers included in the body part 220, the wearable device 101 determines a parameter (e.g., The brightness of red can be adjusted.

Referring to Figure 9B, in response to identifying a gesture of bending the index finger, wearable device 101 may change the display of a visual object 903 superimposed on the index finger within FoV 230. . For example, the wearable device 101 may reduce the brightness of red corresponding to the visual object 903 based on the index finger being bent toward the camera. Referring to state 910 of FIG. 9A and state 920 of FIG. 9B, since the brightness of red decreases, the length of the visual object 903 representing the brightness of red also decreases from the length d1. It can be reduced to a length (d2) less than (d1). The length d2 of the visual object 903 in state 920 may indicate reduced brightness of red based on a bent index finger. As described above with reference to FIG. 9B, the wearable device 101 may reduce parameters corresponding to the finger based on identifying the finger that is bent toward the camera. The degree to which the parameter is reduced may be related to at least one of the angle of the finger or the time the finger is bent. The wearable device 101 may increase the parameter corresponding to the finger based on identifying the finger bent in a direction different from the direction toward the camera. Similarly, the degree to which the parameter is increased may be related to at least one of the angle of the finger or the time the finger is bent.

While identifying the palm from an image containing the body part 220, the function that the wearable device 101 executes based on the angle of the fingers extending from the palm is not limited to the one embodiment described above. For example, based on the palm included in the image, while identifying a hand (e.g., body part 220) extended toward wearable device 101, wearable device 101 may identify a designated finger extending from the palm. Within a bent state (e.g., thumb), the type of visual object to be added by the hand can be adjusted based on the angle of the designated finger and other fingers. The types may include types that resemble tangible lines, such as pens, pencils, and/or brushes. The types may include types based on partial distortion of the image corresponding to the visual object 240, such as mosaic and/or blur. For example, fingers different from the designated finger may correspond to different types of visual objects to be added by the hand. For example, in response to identifying that the index finger is bent within a thumb bent state, wearable device 101 may receive input indicating adding a line similar to a line drawn by a pen. Within the state illustrated above, in response to identifying the middle finger being bent, wearable device 101 may receive input indicating adding a line similar to a line drawn by a pencil. Within the state illustrated above, in response to identifying the fourth finger being bent, wearable device 101 may receive input indicating adding a line similar to the line drawn by the paintbrush. Within the state illustrated above, in response to identifying that the little finger is bent, the wearable device 101 may blur the portion of the visual object 240 that overlaps the little finger's trajectory based on a blur effect and/or mosaic. Thus, an input indicating distortion can be received.

Within state 920 of FIG. 9B , wearable device 101 may enter another state that is different from state 920 based on the posture and/or orientation of body part 220 . For example, a body part 220 having a posture pointing to a visual object 240 displayed as an overlap on the FoV 230 (e.g., a body part 220 having the same posture as the image 635 of FIG. 6) Based on the identification, wearable device 101 may transition to state 810 in FIG. 8A. After entering state 810 of FIG. 8A, the wearable device 101 determines the size of the body part 220 based on the color, transparency, and/or thickness of the visual object 901 in state 920 of FIG. 9B. A curve based on a trajectory (e.g., the curve represented through visual object 815 in FIG. 8A) can be added. For example, the curve may have the color of the visual object 901. For example, the curve may have a thickness related to a parameter corresponding to the visual object 902. For example, the curve may have a transparency associated with a parameter corresponding to the visual object 906.

As described above with reference to FIGS. 9A and 9B, while editing an image corresponding to a visual object 240, the wearable device 101 according to one embodiment may edit the color of at least one shape added to the image. Visual objects (e.g., visual objects 901, 902, 903, 904, 905, 906) for controlling can be displayed. The wearable device 101 may display the visual object in connection with the body part 220 shown through the FoV 230. Within the state of adjusting the color based on the visual object (e.g., states 910 and 920 of FIGS. 9A and 9B), the wearable device 101 controls the body part ( 220) different gestures can be received. Based on sequentially receiving the gestures, the wearable device 101 can sequentially change the color. According to one embodiment, the wearable device 101 includes a function to sequentially cancel the sequential changes of the color (e.g., an undo function), and/or a function to cancel all of the sequential changes to the color. (e.g. reset function) can be executed. FIG. 9C is an exemplary diagram for explaining a gesture indicating at least partially canceling the sequential changes in color.

According to one embodiment, the wearable device 101 displays visual objects (e.g., visual objects) for adjusting color by overlapping on the body part 220, such as the states 910 and 920 of FIGS. 9A and 9B. In the state where (901, 902, 903, 904, 905, 906)) is displayed, an input indicating cancellation of the color change caused by gestures of the body part 220 may be received. The posture 932 of FIG. 9C , like the state 920 of FIG. 9B , is based on the bending of the index finger to determine the visual object 903 and/or parameters corresponding to the visual object 903 (e.g., brightness of red). ) can indicate the posture of the body part 220 after adjusting. For example, posture 932 may represent a posture of the body part 220 fully unfolded. The wearable device 101 may identify that the posture of the body part 220 changes from the posture 932 of FIG. 9C to the posture 934 in which all fingers included in the body part 220 are bent. For example, posture 934 may include a clenched fist posture while the palm within body part 220 is facing the camera of wearable device 101 .

Based on identifying the change to posture 934 , the wearable device 101 may identify a period of time during which the posture of the body part 220 is maintained at posture 934 . The wearable device 101 may identify the period based on a change in the posture of the body part 220 identified through the image. The wearable device 101 may identify the period based on the amplitude difference of the electromyogram measured from the external electronic device 210 attached to the body part 220. For example, while the posture of the body part 220 is maintained at the posture 934, the electromyogram measured from the external electronic device 210 may be maintained at a constant size and/or amplitude. The period during which the posture of the body part 220 is maintained at the posture 934 may include a period during which the amplitude difference of the electromyogram received from the external electronic device 210 is maintained constant.

According to one embodiment, the wearable device 101 may restore parameters related to the color corresponding to the visual object 901 based on the period during which the posture of the body part 220 was maintained in the posture 934. . Referring to FIG. 9C, when the posture of the body part 220 maintains the posture 934 for a period of time less than a specified threshold, the wearable device 101 receives a visual object ( 901), the received input can be canceled to adjust the color-related parameters. For example, when the posture 932 of FIG. 9C represents the posture of the body part 220 after adjusting the brightness of red based on the bending of the index finger, as in the state 920 of FIG. 9B, the body part 220 Based on identifying that the posture of the region 220 maintains the posture 934 for a period of time less than a specified threshold, the wearable device 101 may cancel the adjustment of the brightness of red. Referring to FIG. 9C , based on identifying the posture 936 of the body part 220 after the posture 934, the wearable device 101 adjusts the brightness of the red color to be superimposed on the body part 220. Based on the cancellation of , the visual object 903 having the length d1 of the state 910 of FIG. 9A before the state 920 of FIG. 9B may be displayed.

In one embodiment, when the period during which the posture of the body part 220 is maintained in the posture 934 exceeds the specified threshold, the wearable device 101 determines all parameters related to the color of the visual object 901. It can be initialized. Initialization may mean changing the parameters based on initial values predetermined by the wearable device 101 and/or the user of the wearable device 101.

Based on the color of the visual object 901, adjusted based on the operations described above with reference to FIGS. 9A-9C, the wearable device 101 receives other inputs indicating editing the visual object 240, which is an image. can do. For example, the posture of the body part 220 may change from a first specified posture, including an unfolded posture and/or orientation toward the camera and/or the user, to a second specified posture that is different from the first specified posture. You can. The second designated posture may include a posture of the body part 220 pointing toward the visual object 240 . Based on identifying the posture of the body part 220 that has changed to the second designated posture, the wearable device 101 superimposes, based on the trajectory of the body part 220, onto the FoV 230 using the display. , a visual object representing the trajectory (eg, the visual object 815 in FIGS. 8A to 8D) can be displayed. The visual object may have the color of the visual object 901 of FIGS. 9A to 9C.

As described above, according to one embodiment, the wearable device 101 may select or adjust a color based on a gesture of the user's one hand. The wearable device 101 determines the hand electromyography measured from the external electronic device 210, and/or the hand posture identified from the image acquired using the camera of the wearable device 101, and/or Based on the direction, parameters related to the color can be changed. The wearable device 101 may visualize the parameters and/or the color using visual objects 901, 902, 903, 904, 905, and 906 displayed in conjunction with the hand.

Below, with reference to FIG. 10 , an example of an operation in which the wearable device 101 displays a UI visualizing the color and/or the parameter is described, according to an embodiment.

FIG. 10 illustrates an example of a UI displayed by the wearable device 101 to adjust color, according to an embodiment. The wearable device 101 of FIG. 10 may be an example of the wearable device 101 of FIGS. 2 to 7 and 8A to 8D. For example, the wearable device 101 of FIG. 2 may include the wearable device 101 of FIG. 10 . For example, state 1010 of FIG. 10 is a state of the wearable device 101 after receiving an input indicating selection of an image corresponding to a visual object 240 based on state 710 of FIG. 7. Can include status. For example, state 1010 of FIG. 10 is a wearable displaying a visual object 901 for adjusting color-related parameters based on identifying a body part 220 that includes a hand extended toward the user. It may include the status of the device 101.

Referring to FIG. 10 , within state 1010, according to one embodiment, the wearable device 101 is configured to replace a color within a visual object 901 within the FoV 230 with another color that is different from the color. The above visual objects can be displayed. The one or more visual objects may represent a history of one or more colors that have been selected by the user. For example, wearable device 101 may display one or more colors that were previously displayed via visual object 901 in state 1010, overlaid on portion 1020 of FoV 230 adjacent to visual object 240. Visual objects (1021, 1022, 1023, 1024, 1025) corresponding to the objects may be displayed. The wearable device 101 may display visual objects 1021, 1022, 1023, 1024, and 1025 based on specified transparency. For example, because the visual objects 1021, 1022, 1023, 1024, and 1025 are displayed based on specified transparency, a user wearing the wearable device 101 can view the visual objects 1021, 1022, 1023, 1024, 1025) and the overlapping external space can be seen. The visual objects 1021, 1022, 1023, 1024, and 1025 may display colors selected by the user based on the body part 220 in chronological order. For example, the color of the visual object 1021 adjacent to the top of the FoV 230 within the portion 1020 is selected by the user among the colors of each of the visual objects 1021, 1022, 1023, 1024, and 1025. The most recently selected color can be displayed. Although an example state 1010 in which five visual objects 1021, 1022, 1023, 1024, and 1025 are displayed is shown, the embodiment is not limited thereto.

According to one embodiment, the wearable device 101 may display a list of designated colors in an overlapping manner on the portion 1030 of the FoV 230 adjacent to the visual object 240. Referring to FIG. 10 , an example state 1010 is shown in which visual objects 1031, 1032, 1033, 1034, and 1035 representing each of the five colors included in the list are displayed by the wearable device 101. . The embodiment is not limited to this, and the list displayed by the wearable device 101 through the portion 1030 has a number different from the number of visual objects 1031, 1032, 1033, 1034, and 1035 illustrated in the state 1010. Can include colors. Similar to the visual objects 1021, 1022, 1023, 1024, and 1025, the wearable device 101 may display the visual objects 1031, 1032, 1033, 1034, and 1035 based on specified transparency. Designated colors may include black, and/or white, or may include one or more colors predetermined by the user.

In one embodiment, responding to input indicating selection of any one of visual objects 1021, 1022, 1023, 1024, 1025, 1031, 1032, 1033, 1034, 1035 included within portions 1020, 1030. Accordingly, the wearable device 101 may replace the color of the visual object 901 displayed in an overlapping manner on the body part 220 with the color of a visual object selected by the input. For example, in response to input indicating selection of visual object 1031 within part 1030, wearable device 101 may change the color of visual object 901 superimposed on body part 220 to select the visual object 1031. It can be replaced with the color of (1031). Within the above example, wearable device 101 may add, within portion 1020, a visual object that represents the color of visual object 901 before being replaced by the input. Based on the addition of the visual object within portion 1020, the wearable device 101 selects any one (e.g., most) of the visual objects 1021, 1022, 1023, 1024, and 1025 that were being displayed within portion 1020. Display of a visual object 1025 corresponding to a color selected in the past may be stopped.

According to one embodiment, the wearable device 101 displays visual objects 1021, 1022, and 1023 included in the parts 1020 and 1030, based on the motion of a body part 220, such as a hand, identified through a camera. , 1024, 1025, 1031, 1032, 1033, 1034, 1035) can be identified. For example, the wearable device 101 may identify a change in the angle between the palm and the wrist while the palm is identified in the image acquired through the camera. For example, the wearable device 101 may identify the rotation of the hand, including the palm, from the image. Referring to FIG. 10 , the wearable device 101 may identify a palm rotating along one of the directions 1042 and 1044 from the image. The wearable device 101 determines the color of the visual object 901 superimposed on the palm based on identifying the palm rotating along one of the directions 1042 and 1044, and the visual objects 1021, It can be replaced with any one of the colors (1022, 1023, 1024, 1025, 1031, 1032, 1033, 1034, 1035). For example, when identifying a palm rotating along direction 1042, wearable device 101 changes the color of visual object 901 to visual objects 1021, 1022, 1023, 1024, and 1025 in portion 1020. ) can be replaced with the color of the visual object 1021. Within the above example, upon identifying a palm that further rotates along direction 1042, wearable device 101 changes the color of visual object 901 to visual objects 1021, 1022, 1023, It can be replaced with the color of the visual object 1022 among 1024 and 1025. Within the above example, upon identifying a palm that further rotates along direction 1044, wearable device 101 may restore the color of visual object 901.

Although the operation of the wearable device 101 has been described as an example based on the body part 220 including the user's right hand, the embodiment is not limited thereto. According to one embodiment, the wearable device 101 may support a function of independently adjusting color using each of the user's hands. For example, similar to the visual object 901 displayed overlapping on body part 220, which is the right hand, the wearable device 101 may display another visual object for color adjustment on another body part, which is the left hand. You can. The wearable device 101 may adjust the color corresponding to the visual object 901 based on the gesture of the right hand, while adjusting the color corresponding to the other visual object 901 based on the gesture of the left hand. The wearable device 101 adds one or more shapes having a color corresponding to the visual object 901 based on the gesture of the right hand, while adding one or more shapes having a color corresponding to the other visual object 901 based on the gesture of the left hand. You can add them.

Below, with reference to FIGS. 11 to 13 , one or more operations of the wearable device 101 described above with reference to FIGS. 1 to 10 are described.

FIG. 11 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment. The wearable device of FIG. 11 may include the wearable device 101 of FIGS. 1 to 10 . For example, at least one of the operations of FIG. 11 may be performed by the wearable device 101 of FIG. 2 and/or the processor 120 of the wearable device 101.

Referring to FIG. 11, within operation 1110, a wearable device according to one embodiment may obtain information representing the electromyogram of the hand. For example, a wearable device can identify the electromyogram of a designated body part, such as the hand. The wearable device may obtain information related to the electromyogram from an external electronic device (eg, the external electronic device 210 of FIG. 2) connected to the wearable device. The information may include sensor data indicating the size and/or amplitude of the electromyogram. The external electronic device may be attached to a designated body part (eg, body part 220 of FIG. 2) of a user wearing a wearable device in order to identify the electromyogram. The wearable device can identify changes in electromyogram over time, as shown in the graph 610 of FIG. 6 .

Referring to FIG. 11, within operation 1120, the wearable device according to one embodiment may identify whether the amplitude difference of the electromyogram identified by operation 1110 exceeds a specified threshold. The wearable device may compare the amplitude difference between each repeatedly identified EMG based on different time points with a designated threshold. The specified threshold may be predetermined or heuristically adjusted to identify gestures of body parts (e.g., wrist, and/or hand, such as body part 220 of FIG. 2) associated with the EMG. You can. Before the amplitude difference of the EMG exceeds the specified threshold (1120-No), the wearable device may repeatedly identify the EMG based on the movement (1110).

In response to identifying an electromyogram with an amplitude difference exceeding a specified threshold (1120-Yes), within operation 1130, according to one embodiment, the wearable device displays a camera (e.g., camera 520 of FIG. 5). ), you can obtain an image including a hand. For example, a wearable device may acquire at least one image using a camera. In one embodiment where the camera is positioned toward the user's FoV (e.g., FoV 230 in FIG. 2), the at least one image may include at least a portion of the external space visible to the user through the FoV. there is. The wearable device 101 acquiring the at least one image based on the operation 1130 may be performed while the wearable device 101 identifies an electromyogram having an amplitude difference exceeding a specified threshold. Images 631, 632, 633, 634, 635, and 636 of FIG. 6 may be an example of the at least one image acquired by the wearable device 101 based on the operation 1130.

Referring to FIG. 11, within the operation 1140, a wearable device according to one embodiment may identify at least one of the hand posture or direction based on the image of the operation 1130. For example, the wearable device may identify at least one of the posture or orientation of a specified body part, including the hand, from at least one image. For example, the wearable device may identify the position and/or angle of one or more joints included in the specified body part based on identifying the specified body part from at least one image. The posture or direction that the wearable device identifies based on movement 1140 may be indicated by the position and/or angle of the one or more joints.

Referring to FIG. 11, within operation 1150, the wearable device according to one embodiment may execute a function related to image editing based on at least one of the posture or direction of the hand. For example, the wearable device may execute a function related to image editing based on at least one of the posture or orientation of a designated body part, including the hand. The posture or direction of the designated body part identified based on the movement 1140 may represent a gesture for executing a specific function among a plurality of functions related to image editing. According to one embodiment, the wearable device edits the image, as described above with reference to FIGS. 8A to 8D, 9A to 9C, and/or 10, based on at least one of the posture or orientation of the specified body part. You can perform different functions related to .

For example, a wearable device may identify a hand open toward the wearable device based on identifying the palm from an image. In the first state in which the palm is identified from the image, the wearable device may execute a function to adjust the color in conjunction with the palm shown through the display. For example, the first state may include states 910 and 920 of FIGS. 9A to 9C. In the first state, the wearable device may display a first visual object including a color (eg, visual object 901 in FIG. 9A) in association with the palm. In a second state different from the first state, the wearable device displays, based on the trajectory of at least one finger included in the hand, a second visual object (e.g. , the visual object 815 of FIGS. 8A to 8D) can be displayed. The second state may include a state in which the wearable device identifies an image including the back of the hand using a camera.

Below, with reference to FIG. 12 , the operation of the wearable device while performing the operations 1130, 1140, and 1150 of FIG. 11 will be described in more detail.

FIG. 12 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment. The wearable device of FIG. 12 may include the wearable device 101 of FIGS. 1 to 10 . At least one of the operations of FIG. 12 may be performed by the wearable device 101 of FIG. 2 and/or the processor 120 of the wearable device 101. At least one of the operations in FIG. 12 may be related to at least one of the operations in FIG. 11 .

Referring to FIG. 12, within operation 1210, a wearable device according to one embodiment may identify the electromyogram of the hand using an external electronic device. For example, a wearable device can identify an electromyogram with amplitude differences for a specified body part, including the hand, from an external electronic device. The wearable device may perform operation 1210 of FIG. 12 , similar to operation 1110 of FIG. 11 . Referring to FIG. 12, within operation 1220, the wearable device according to one embodiment may determine whether the amplitude difference of the identified EMG exceeds a specified threshold. The wearable device may perform operation 1220 of FIG. 12 , similar to operation 1120 of FIG. 11 . In response to identifying an electromyogram with an amplitude difference exceeding a specified threshold (1220-Yes), within operation 1230, according to one embodiment, the wearable device uses a camera to display at least one device comprising a hand. Images can be obtained. Referring to FIG. 12, within operation 1240, according to one embodiment, the wearable device may identify the posture, or orientation, of the hand based on the hand captured within at least one image of operation 1230. You can.

Referring to FIG. 12, within operation 1250, according to one embodiment, the wearable device may determine whether the amplitude difference of the identified electromyogram is reduced below a specified threshold. If the amplitude difference of the electromyogram exceeds the specified threshold (1250-No), the wearable device may perform operations 1230 and 1240. For example, within a time interval in which the amplitude difference of the electromyogram exceeds a specified threshold, the wearable device may repeatedly perform operations 1230 and 1240. Because the wearable device repeatedly performs operations 1230 and 1240, the wearable device can identify the motion of a specified body part within the time interval. For example, the motion of the designated body part may include a gesture performed by the designated body part.

When the amplitude difference of the electromyogram decreases below the specified threshold (1250-yes), according to one embodiment, the wearable device detects at least one of the hand posture or direction identified within the time interval in which the amplitude difference of the electromyogram exceeds the specified threshold. Based on one, the gesture expressed by the hand can be identified.

Referring to FIG. 12, within operation 1270, the wearable device, according to one embodiment, may execute a function corresponding to the identified gesture. The function that the wearable device executes based on the gesture may include a function for editing an image overlaid on the FoV by the wearable device. A function for editing an image may include a function for adjusting the color of one or more shapes to be added to the image.

FIG. 13 shows an example of a flowchart for explaining the operation of a wearable device, according to an embodiment. The wearable device of FIG. 13 may include the wearable device 101 of FIGS. 1 to 10 . At least one of the operations of FIG. 13 may be performed by the wearable device 101 of FIG. 2 and/or the processor 120 of the wearable device 101. At least one of the operations in FIG. 13 may be related to at least one of the operations in FIGS. 11 and 12 .

Referring to FIG. 13, within operation 1310, a wearable device according to one embodiment may acquire at least one image including a hand using a camera. For example, the wearable device may identify at least one of the posture or direction of a specified body part, including a hand, based on the at least one image. The wearable device may perform operation 1310 of FIG. 13 , similar to operation 1140 of FIG. 11 and/or operations 1250 and 1260 of FIG. 12 .

Referring to FIG. 13, in operation 1320, the wearable device according to one embodiment may identify whether the palm has been identified from at least one image. For example, the wearable device may determine whether a specified body part with a specified posture and a specified direction has been identified from at least one image. In an embodiment in which the designated body part includes the hand of a user wearing a wearable device, the designated posture may include a posture in which the hand is wide open. The designated posture may be indicated by information including the angles of joints included in the hand in a state where the hand is wide open. The designated direction may be related to the direction the palm of the hand faces. For example, the specified direction may include a direction toward the camera of the wearable device.

In response to identifying the palm from the at least one image of operation 1310 (1320-Yes), within operation 1330, according to one embodiment, the wearable device configures, in association with the palm, a color to indicate a color. Visual objects can be displayed. The visual object may include the visual object 901 of FIGS. 9A to 9C. In one embodiment where the designated body part includes the hand of a user wearing a wearable device, the visual object may be displayed superimposed on the palm of the hand by the wearable device.

Referring to FIG. 13, within operation 1340, according to one embodiment, the wearable device may change the color of a visual object based on a change in the posture of the hand including the palm. For example, the wearable device may receive input indicating that it adjusts the color of a visual object and/or parameters related to the color based on changes in the fingers included in the hand. Based on the input, the operation of adjusting the color of the visual object and/or the parameters may be performed similarly to the operation described above with reference to FIGS. 9A to 9C and/or 10.

As described above, according to one embodiment, a wearable device may identify a gesture of a user wearing the wearable device based on information acquired using the wearable device and/or an external electronic device. Based on the gesture, the wearable device may execute one or more functions for editing the image. The one or more functions executed by the wearable device may include a function for adjusting color. The function for adjusting color may be executed based on the user's hand orientation and/or posture identified by the wearable device.

A method of editing images based on the user's gestures using a wearable device may be required.

As described above, according to one embodiment, a wearable device (e.g., wearable device 101 of FIG. 2 and/or FIG. 5) includes a display (e.g., display 530 of FIG. 5), It may include a camera (eg, camera 520 in FIG. 5), a communication circuit (eg, communication circuit 510 in FIG. 5), and a processor (eg, processor 120 in FIG. 5). The processor may be configured to obtain information representing the electromyogram of the hand from an external electronic device (e.g., the external electronic device 210 of FIGS. 2 and/or 5) through the communication circuit. The processor uses the camera to generate an image containing the hand (e.g., images 631 and 632 of FIG. 6 ) based on identifying the electromyogram having an amplitude difference exceeding a specified threshold based on the information. , 633, 634, 635, 636)). The processor, in a first state that identifies the palm of the hand in the image (e.g., states 910 and 920 of FIGS. 9A and 9B), associates the palm shown on the display with the palm, It may be configured to display a first visual object including a color (eg, the visual object 901 in FIGS. 9A and 9B). The processor, in a second state different from the first state (e.g., states 810, 820, 830, 840 of FIGS. 8A-8D), based on the trajectory of at least one finger included in the hand Thus, it may be configured to display a second visual object (eg, visual object 815 in FIGS. 8A to 8D) having the shape of the trajectory on the image. For example, a wearable device can selectively execute different functions for editing an image based on the orientation of the hand as defined by the palm or the back of the hand.

For example, the processor may be configured to display, within the first state, the first visual object overlaid on a portion of the display where the palm is visible.

For example, the processor may display different visual objects representing different parameters associated with the color included in the first visual object (e.g., , may be configured to display visual objects 902, 903, 904, 905, and 906 of FIGS. 9A to 9C.

For example, the parameters may include at least one of the brightness of each of red, green, and blue adjusted to form the color, or the transparency of the color.

For example, the parameters may include the thickness of at least one stroke represented by the second visual object in the second state.

For example, the processor may be configured to identify the angle of a first one of the fingers from the image obtained from the camera. The processor may be configured to change a first parameter corresponding to the first finger among the parameters, based on the angle of the first finger.

For example, the processor determines the size of a third visual object corresponding to the first parameter among the visual objects shown on the portions where each of the fingers is visible, based on the changed first parameter. It can be configured to adjust it.

For example, the processor may identify that an angle of a second finger that is different from the first finger is within an angle range that is different from a straight angle, while the angle of the first finger is within the angle range. It may be configured to adjust the type of the second visual object based on whether or not it is included.

For example, the processor is configured to transition, within the first state, to the second state based on identifying the back of the hand opposite the palm from the image. It can be. The processor may be configured to display the second visual object based on the color of the first visual object in the first state after switching to the second state.

For example, the processor may be configured to refrain from displaying the first visual object in association with the palm based on identifying the back of the hand from the image.

For example, the processor is based on identifying, within the second state, from the image the back of the hand, and the at least one finger extending from the back of the hand and straightened toward the image. Thus, it may be configured to display the second visual object.

For example, the processor may be configured to adjust the shape of the second visual object based on at least one of the number or type of the at least one finger extended toward the image.

As described above, the method of using a wearable device according to an embodiment may include an operation of acquiring information representing the electromyogram of the hand from an external electronic device through a communication circuit of the wearable device. The method may include acquiring an image including the hand using a camera of the wearable device based on identifying the electromyogram that exceeds a specified threshold based on the information. The method includes displaying a first visual object including a color in association with the palm shown through the display of the wearable device, within a first state in which the palm of the hand is identified in the image. can do. The method includes, in a second state different from the first state, based on the trace of at least one finger included in the hand, displaying a second visual object having the shape of the trace on the image. It can be included.

For example, displaying the first visual object may include displaying the first visual object on a portion of the display where the palm is visible.

For example, the act of displaying the first visual object may include displaying different parameters associated with the color included in the first visual object on different portions of the display where each of the fingers extending from the palm is visible. It may include an operation of displaying different visual objects that are represented.

For example, the parameters may include at least one of the brightness of each of red, green, and blue adjusted to form the color, or the transparency of the color.

For example, the parameters may include the thickness of at least one stroke represented by the second visual object in the second state.

For example, the method may include identifying the angle of a first one of the fingers from the image obtained from the camera. The method may include changing a first parameter corresponding to the first finger among the parameters based on the angle of the first finger.

For example, the act of displaying the second visual object may be based on identifying, from the image, the back of the hand, and the at least one finger extending from the back of the hand and extending toward the image. It may include an operation of displaying a second visual object.

As described above, according to one embodiment, in the non-transitory computer-readable storage medium storing one or more programs, the one or more programs, when executed by a processor of a wearable device, are transmitted through a communication circuit of the wearable device. , may include instructions that cause the wearable device to acquire information representing electromyography of the hand from an external electronic device. The one or more programs, when executed by the processor, are configured to obtain an image including the hand using a camera of the wearable device based on identifying the electromyogram that exceeds a specified threshold based on the information, It may include instructions that trigger the wearable device. The one or more programs, when executed by the processor, in association with the palm shown through the display of the wearable device, within a first state that identifies the palm of the hand in the image, include a color. 1 may include instructions that cause the wearable device to display a visual object. The one or more programs are configured to display, in a second state different from the first state, a second visual object having the shape of the trace on the image based on the trace of at least one finger included in the hand. , may include instructions that cause the wearable device.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for the purpose of being interpreted by or providing instructions or data to the processing device. there is. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (20)

In the wearable device 101,
display 530;
camera 520;
communication circuit 510; and
Includes a processor 120,
The processor,
Obtain information representing the electromyogram of the hand from an external electronic device 210 through the communication circuit;
Acquire images 631; 632; 633; 634; 635; 636) including the hand using the camera, based on identifying the electromyogram having an amplitude difference exceeding a threshold specified based on the information; ;
In a first state (910; 920) in which the palm of the hand is identified in the image, a first visual object (901) comprising a color, in association with the palm shown through the display. display; and
In a second state (810; 820; 830; 840) different from the first state, a second visual object having the shape of the trajectory on the image, based on the trajectory of at least one finger included in the hand. configured to display (815),
Wearable devices. The method of claim 1, wherein the processor:
configured to display, within the first state, the first visual object in an overlapping manner on a portion of the display where the palm is visible,
Wearable devices. The method of claim 2, wherein the processor:
On different parts of the display where each of the fingers extending from the palm are visible, different visual objects 902, 903, 904, 905, 906 representing different parameters associated with the color included in the first visual object. ), configured to display,
Wearable devices. The method of claim 3, wherein the parameters are:
At least one of the brightness of each of red, green, and blue adjusted to form the color, or the transparency of the color,
Wearable devices. The method of claim 3, wherein the parameters are:
Containing the thickness of at least one stroke represented by the second visual object in the second state,
Wearable devices. The method of claim 3, wherein the processor:
identify, from the image obtained from the camera, an angle of a first one of the fingers;
configured to change a first parameter corresponding to the first finger among the parameters, based on the angle of the first finger,
Wearable devices. The method of claim 6, wherein the processor:
configured to adjust the size of a third visual object corresponding to the first parameter among the visual objects shown on the portions where each of the fingers is visible based on the changed first parameter,
Wearable devices. The method of claim 6, wherein the processor:
Based on whether the angle of the first finger is within the angle range while identifying that the angle of the second finger, which is different from the first finger, is within the angle range that is different from a straight angle, configured to control the type of the second visual object,
Wearable devices. The method of claim 1, wherein the processor:
within the first state, transition to the second state based on identifying a back of the hand opposite the palm from the image;
After switching to the second state, configured to display the second visual object based on the color of the first visual object in the first state,
Wearable devices. The method of claim 9, wherein the processor:
configured to, based on identifying the back of the hand from the image, refrain from displaying the first visual object in association with the palm,
Wearable devices. The method of claim 1, wherein the processor:
Within the second state, the second visual object is based on identifying, from the image, the back of the hand, and the at least one finger extending from the back of the hand and straightened toward the image. configured to display,
Wearable devices. The method of claim 11, wherein the processor:
configured to adjust the shape of the second visual object based on at least one of the number or type of the at least one finger extended toward the image,
Wearable devices. In the method of the wearable device,
Obtaining information representing electromyogram of the hand from an external electronic device through a communication circuit of the wearable device;
Acquiring an image including the hand using a camera of the wearable device based on identifying the electromyogram having an amplitude difference exceeding a specified threshold based on the information;
In a first state in which the palm of the hand is identified in the image, displaying a first visual object including a color in association with the palm shown through the display of the wearable device; and
In a second state different from the first state, based on the trajectory of at least one finger included in the hand, displaying a second visual object having the shape of the trajectory on the image,
method. The method of claim 13, wherein the operation of displaying the first visual object includes:
Displaying the first visual object on the portion of the display where the palm is visible,
method. The method of claim 14, wherein the operation of displaying the first visual object includes:
displaying, on different portions of the display where each of the fingers extending from the palm are visible, different visual objects representing different parameters associated with the color included in the first visual object.
method. 16. The method of claim 15, wherein the parameters are:
At least one of the brightness of each of red, green, and blue adjusted to form the color, or the transparency of the color,
method. 16. The method of claim 15, wherein the parameters are:
Containing the thickness of at least one stroke represented by the second visual object in the second state,
method. According to clause 15,
identifying an angle of a first finger among the fingers from the image obtained from the camera; and
Based on the angle of the first finger, further comprising changing a first parameter corresponding to the first finger among the parameters,
method. The method of claim 13, wherein the operation of displaying the second visual object includes:
displaying the second visual object based on identifying, from the image, the back of the hand, and the at least one finger extending from the back of the hand and extended toward the image,
method. A non-transitory computer-readable storage medium storing one or more programs, wherein when executed by a processor of a wearable device, the one or more programs:
Obtaining information representing the electromyogram of the hand from an external electronic device through a communication circuit of the wearable device;
acquire an image including the hand using a camera of the wearable device based on identifying the electromyogram having an amplitude difference exceeding a specified threshold based on the information;
In a first state in which the palm of the hand is identified in the image, display a first visual object including a color in association with the palm shown through the display of the wearable device; and
In a second state different from the first state, based on the trajectory of at least one finger included in the hand, cause the wearable device to display a second visual object having the shape of the trajectory on the image. Containing instructions that do,
A non-transitory computer-readable storage medium.

Images