KR20230085794A - Waearable electronic device displaying virtual object and method for controlling the same - Google Patents

Abstract

According to an embodiment, a wearable electronic device includes a first sensor for detecting a location of the wearable electronic device, a second sensor for detecting the surroundings of the wearable electronic device, a display, a memory, and at least one processor, and the at least one processor obtains location information of a real space where the wearable electronic device is located through a first sensor, acquires object information related to a plurality of real objects included in the real space through a second sensor, and stores the location information and If the feature point information of the user's motion related to the object information is stored, when the wearable electronic device is located in a real space, a virtual object notifying that there is motion information of the feature point is displayed on the display, and through the virtual object, the motion information related to the feature point is displayed. When a user input for displaying an image is received, a 3D virtual object image may be obtained based on motion information of feature points, and the 3D virtual object image may be displayed on a display.

Description

Wearable electronic device displaying virtual objects and its control method { WAEARABLE ELECTRONIC DEVICE DISPLAYING VIRTUAL OBJECT AND METHOD FOR CONTROLLING THE SAME }

Embodiments of the present disclosure relate to a wearable electronic device displaying a virtual object and a control method thereof.

As electronic and communication technologies develop, electronic devices can be miniaturized and lightweight to the extent that they can be used without much inconvenience even when worn on the user's body. For example, wearable electronic devices such as head mounting devices (HMDs), smart watches (or bands), contact lens type devices, ring type devices, glove type devices, shoe type devices or clothing type devices are being commercialized. . Since the wearable electronic device is directly worn on the body, portability and user accessibility can be improved.

A head-mounted electronic device is a device worn on a user's head or face, and may provide augmented reality (AR) to the user. For example, a head-mounted device that provides augmented reality may be implemented in the form of glasses, and may provide information about objects in the form of images or text to the user in at least a part of the user's field of view.

In order to store the motion information of the user, an image including the user may be captured through a camera or a sensing value related to the motion may be obtained through a sensor included in a wearable electronic device worn by the user.

However, in order to store motion information related to a specific motion of the user, the user must input a manipulation for saving before the motion. If the manipulation is not input before the user's intended motion, the user's motion to be saved is lost. It can be.

Also, in an augmented reality environment, when trying to reproduce a stored motion of a user, the user has to check a plurality of stored motion information and then select motion information for reproduction.

Various embodiments of the present disclosure provide a wearable electronic device that stores motion information of a user without manipulation for storage and provides motion information related to a real space among a plurality of stored motion information in an augmented reality environment and a control method thereof .

According to an embodiment, a wearable electronic device includes a first sensor for detecting a location of the wearable electronic device, a second sensor for detecting surroundings of the wearable electronic device, a display, a memory, and at least one processor, wherein the at least One processor acquires location information of a real space where the wearable electronic device is located through the first sensor, and obtains object information related to a plurality of real objects included in the real space through the second sensor, , If the location information and feature point information of the user's movement related to the object information are stored in the memory, if the wearable electronic device is located in the real space, the virtual object notifying that there is motion information of the feature point is displayed. , and when a user input for displaying an image related to the motion information of the feature point is received through the virtual object, a 3D virtual object image is obtained based on the motion information of the feature point, and the 3D virtual object image is displayed on the virtual object. can be shown on the display.

According to an embodiment, a control method of a wearable electronic device includes an operation of acquiring location information of a real space where the wearable electronic device is located through a first sensor that senses a location of the wearable electronic device, and moving around the wearable electronic device. Obtaining object information related to a plurality of real objects included in the real space through a second sensor that senses, and if the location information and feature point information of the user's movement related to the object information are stored in the memory, the When the wearable electronic device is located in the real space, an operation of displaying a virtual object indicating that there is motion information of the feature point on a display, and receiving a user input for displaying an image related to the motion information of the feature point through the virtual object , an operation of obtaining a 3D virtual object image based on the motion information of the feature point and an operation of displaying the 3D virtual object image on the display may be included.

An electronic device according to various embodiments of the present disclosure may record a user's behavior without a user's manipulation by storing information about a user's movement that satisfies a set condition.

In addition, the electronic device according to various embodiments of the present disclosure may notify the user that there is a recorded user action and provide the recorded user action as a 3D virtual object image according to the user's manipulation.

1 is a block diagram of an electronic device in a network environment according to various embodiments.
2 is a perspective view of an electronic device according to an embodiment.
3A is a first perspective view illustrating an internal configuration of an electronic device according to an exemplary embodiment;
3B is a second perspective view illustrating an internal configuration of an electronic device according to an exemplary embodiment.
4 is an exploded perspective view of an electronic device according to an exemplary embodiment.
5 is another perspective view of a wearable electronic device according to various embodiments of the present disclosure.
6 is a flowchart illustrating an operation of storing motion information of a user of an electronic device according to an embodiment of the present disclosure.
7 is a flowchart illustrating an operation of storing motion information of a user of an electronic device according to an embodiment of the present disclosure.
8 is a flowchart illustrating an operation of determining whether motion information of a user satisfies a set condition according to an embodiment of the present disclosure.
9 is a diagram for explaining feature point information of a user's motion according to an embodiment of the present disclosure.
10 is a flowchart illustrating an operation of an electronic device to display a 3D virtual object image based on stored feature point information, according to an embodiment of the present disclosure.
11A is a diagram for explaining a case where an electronic device stores motion information of a user according to an embodiment of the present disclosure.
FIG. 11B is a diagram for explaining an operation of displaying a 3D virtual object image related to stored user motion information by an electronic device when the electronic device is located in a place where the user motion information of FIG. 11A is stored, according to an embodiment of the present disclosure. it is a drawing
12A is a diagram for explaining a case where an electronic device stores motion information of a user according to an embodiment of the present disclosure.
12B is a diagram for explaining an operation of displaying a 3D virtual object image related to stored user movement information by an electronic device when the electronic device is located in a place where the user movement information of FIG. 11A is stored according to an embodiment of the present disclosure. it is a drawing
13A is a diagram for explaining a case where an electronic device stores motion information of a user according to an embodiment of the present disclosure.
FIG. 13B is a diagram for explaining an operation of displaying a 3D virtual object image related to stored user motion information by an electronic device when the electronic device is located in a place where the user motion information of FIG. 11A is stored, according to an embodiment of the present disclosure. it is a drawing
14 is a diagram for explaining a communication connection form of an electronic device according to an embodiment of the present disclosure.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 may include a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 180 or communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor set to detect a touch or a pressure sensor set to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 is a peak data rate for eMBB realization (eg, 20 Gbps or more), a loss coverage for mMTC realization (eg, 164 dB or less), or a U-plane latency for URLLC realization (eg, Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a lower surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a perspective view of an electronic device 200 (eg, the electronic device 101 of FIG. 1 ) according to an embodiment.

Referring to FIG. 2 , the electronic device 200 is a wearable electronic device in the form of glasses, and a user can visually recognize surrounding objects or environments while wearing the electronic device 200 . For example, the electronic device 200 may be a head mounting device (HMD) or smart glasses capable of providing an image directly in front of the user's eyes. The configuration of the electronic device 200 of FIG. 2 may be entirely or partially the same as that of the electronic device 101 of FIG. 1 .

According to an embodiment, the electronic device 200 may include a housing 210 forming an exterior of the electronic device 200 . The housing 210 may provide a space in which components of the electronic device 200 may be disposed. For example, the housing 210 may include a lens frame 202 and at least one wearing member 203 .

According to an embodiment, the electronic device 200 may include at least one display member 201 capable of providing visual information to a user. For example, the display member 201 may include a module equipped with a lens, a display, a waveguide, and/or a touch circuit. According to one embodiment, the display member 201 may be formed to be transparent or translucent. According to an exemplary embodiment, the display member 201 may include glass made of a translucent material or a window member capable of adjusting light transmittance according to color concentration control. According to an embodiment, the display members 201 may be provided as a pair and disposed to correspond to the user's left and right eyes, respectively, while the electronic device 200 is worn on the user's body.

According to one embodiment, the lens frame 202 may accommodate at least a portion of the display member 201 . For example, the lens frame 202 may surround at least a portion of an edge of the display member 201 . According to an embodiment, the lens frame 202 may position at least one of the display members 201 to correspond to the user's eyes. According to one embodiment, the lens frame 202 may be the rim of a typical eyeglass structure. According to an embodiment, the lens frame 202 may include at least one closed curve surrounding the display member 201 .

According to one embodiment, the wearing member 203 may extend from the lens frame 202 . For example, the wearing member 203 extends from an end of the lens frame 202 and, together with the lens frame 202, may be supported or positioned on the user's body (eg, ear). According to one embodiment, the wearing member 203 may be rotatably coupled with respect to the lens frame 202 via a hinge structure 229 . According to one embodiment, the wearing member 203 may include an inner side surface 231c configured to face the user's body and an outer side surface 231d opposite to the inner side surface.

According to an embodiment, the electronic device 200 may include a hinge structure 229 configured to fold the wearing member 203 relative to the lens frame 202 . The hinge structure 229 may be disposed between the lens frame 202 and the wearing member 203 . In a state in which the electronic device 200 is not worn, the user may carry or store the wearing member 203 by folding the lens frame 202 such that a portion thereof overlaps with the lens frame 202 .

3A is a first perspective view illustrating an internal configuration of an electronic device according to various embodiments. 3B is a second perspective view illustrating an internal configuration of an electronic device according to various embodiments. 4 is an exploded perspective view of an electronic device according to various embodiments.

Referring to FIGS. 3A to 4 , the electronic device 200 includes components accommodated in a housing 210 (eg, at least one circuit board 241 (eg, a printed circuit board (PCB), a printed board assembly (PBA))) , FPCB (flexible PCB) or RFPCB (rigid-flexible PCB)), at least one battery 243, at least one speaker module 245, at least one power transmission structure 246, and camera module 250) can include The configuration of the housing 210 of FIGS. 3A and 3B may be the same in whole or in part as the configurations of the display member 201, the lens frame 202, the wearing member 203, and the hinge structure 229 of FIG. 2 . there is.

According to an embodiment, the electronic device 200 uses a camera module 250 (eg, the camera module 180 of FIG. 1 ) to view a user or a direction the electronic device 200 faces (eg, - Obtains and/or recognizes a visual image of an object or environment in the Y direction), and external electronic devices (eg, the first network 198 or the second network 199 in FIG. 1) through a network (eg, the first network 198 or the second network 199 in FIG. Information about objects or environments may be provided from the electronic devices 102 and 104 or the server 108 of FIG. 1 . In another embodiment, the electronic device 200 may provide the user with information about objects or environments in a sound or visual form. The electronic device 200 may provide information about objects or environments to the user through the display member 201 in a visual form using a display module (eg, the display module 160 of FIG. 1 ). For example, the electronic device 200 may implement augmented reality by implementing information about an object or environment in a visual form and combining it with a real image of a user's surrounding environment.

According to an embodiment, the display member 201 has a first surface F1 facing a direction in which external light is incident (eg, -Y direction) and a direction opposite to the first surface F1 (eg, +Y direction). direction) may include a second surface (F2) facing. While the user is wearing the electronic device 200, at least a portion of the light or image incident through the first surface F1 is directed to the second portion of the display member 201 disposed to face the left and/or right eyes of the user. It may pass through the surface F2 and enter the user's left eye and/or right eye.

According to one embodiment, the lens frame 202 may include at least two or more frames. For example, the lens frame 202 may include a first frame 202a and a second frame 202b. According to an embodiment, when the user wears the electronic device 200, the first frame 202a is a frame facing the user's face, and the second frame 202b is attached to the first frame 202a. It may be a part of the lens frame 202 spaced apart in the direction of the user's gaze (eg, -Y direction) with respect to the user.

According to one embodiment, the light output module 211 may provide images and/or videos to the user. For example, the light output module 211 includes a display panel (not shown) capable of outputting an image, and a lens (not shown) corresponding to the user's eyes and guiding the image to the display member 201. can do. For example, a user may acquire an image output from a display panel of the light output module 211 through a lens of the light output module 211 . According to one embodiment, the light output module 211 may include a device configured to display various types of information. For example, the light output module 211 may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode It may include at least one of (organic light emitting diode, OLED) or micro LED (micro light emitting diode, micro LED). According to an exemplary embodiment, when the light output module 211 and/or the display member 201 includes one of a liquid crystal display, a digital mirror display, and a silicon liquid crystal display, the electronic device 200 may generate light A light source radiating light to the display area of the output module 211 and/or the display member 201 may be included. According to another embodiment, when the light output module 211 and/or the display member 201 includes one of an organic light emitting diode and a micro LED, the electronic device 200 does not include a separate light source and provides information to the user. A virtual image can be provided.

According to one embodiment, at least a portion of the light output module 211 may be disposed within the housing 210 . For example, the light output module 211 may be disposed on the wearing member 203 or the lens frame 202 to correspond to the user's right eye and left eye, respectively. According to an embodiment, the light output module 211 may be connected to the display member 201 and provide an image to a user through the display member 201 . For example, an image output from the light output module 211 is incident on the display member 210 through an input optical member positioned at one end of the display member 201, and is positioned on at least a portion of the display member 210. It can be radiated toward the user's eyes through a waveguide and an output optical member. According to an embodiment, the waveguide may be made of glass, plastic, or polymer, and may include a nanopattern formed on an inner or outer surface, for example, a polygonal or curved grating structure. there is. According to an embodiment, the waveguide may include at least one of at least one diffractive element (eg, a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (eg, a reflective mirror). .

According to one embodiment, the circuit board 241 may include components for driving the electronic device 200 . For example, the circuit board 241 may include at least one integrated circuit chip, and may include the processor 120, memory 130, power management module 188, or communication module of FIG. 1 . At least one of 190 may be provided on the integrated circuit chip. According to one embodiment, the circuit board 241 may be disposed within the wearing member 203 of the housing 210 . According to one embodiment, the circuit board 241 may be electrically connected to the battery 243 through the power delivery structure 246 . According to one embodiment, the circuit board 241 is connected to the flexible printed circuit board 205, and electronic components (eg, the light output module 211, Electrical signals may be transmitted to the light emitting unit of the camera module 250. According to an embodiment, the circuit board 241 may be an interposer board.

According to various embodiments, the flexible printed circuit board 205 may extend from the circuit board 241 across the hinge structure 229 into the interior of the lens frame 202, and from the interior of the lens frame 202. It may be disposed on at least a part of the circumference of the display member 201 .

According to an embodiment, the battery 243 (eg, the battery 189 of FIG. 1 ) may be a component of the electronic device 200 (eg, the light output module 211, the circuit board 241, and the speaker module 245). , the microphone module 247, and/or the camera module 250) and can be electrically connected, and can supply power to components of the electronic device 200.

According to one embodiment, at least a part of the battery 243 may be disposed on the wearing member 203 . According to one embodiment, the battery 243 may be disposed adjacent to the ends 203a and 203b of the wearing member 203 . For example, the battery 243 may include a first battery 243a disposed at the first end 203a of the wearing member 203 and a second battery 243b disposed at the second end 203b. there is.

According to various embodiments, the speaker module 245 (eg, the audio module 170 or the sound output module 155 of FIG. 1 ) may convert an electrical signal into sound. At least a portion of the speaker module 245 may be disposed within the wearing member 203 of the housing 210 . According to one embodiment, the speaker module 245 may be positioned within the wearing member 203 to correspond to the user's ear. According to one embodiment (eg, FIG. 3A ), the speaker module 245 may be disposed on the circuit board 241 . For example, the speaker module 245 may be disposed between the circuit board 241 and an inner case (eg, the inner case 231 of FIG. 4 ). According to one embodiment (eg, FIG. 3B ), the speaker module 245 may be disposed next to the circuit board 241 . For example, the speaker module 245 may be disposed between the circuit board 241 and the battery 243 .

According to an embodiment, the electronic device 200 may include a speaker module 245 and a connection member 248 connected to the circuit board 241 . The connecting member 248 may transmit at least a part of sound and/or vibration generated by the speaker module 245 to the circuit board 241 . According to one embodiment, the connection member 248 may be integrally formed with the speaker module 245 . For example, a portion extending from the speaker frame of the speaker module 245 may be interpreted as a connecting member 248 . According to one embodiment (eg, FIG. 3A ), the connecting member 248 may be omitted. For example, when the speaker module 245 is disposed on the circuit board 241, the connecting member 248 may be omitted.

According to an embodiment, the power transmission structure 246 may transfer power of the battery 243 to an electronic component (eg, the light output module 211) of the electronic device 200 . For example, the power transmission structure 246 is electrically connected to the battery 243 and/or the circuit board 241, and the circuit board 241 outputs power received through the power transmission structure 246 to an optical output. It can be forwarded to module 211.

According to one embodiment, the power delivery structure 246 may be a configuration capable of transmitting power. For example, power delivery structure 246 may include a flexible printed circuit board or wire. For example, the wire may include a plurality of cables (not shown). In various embodiments, the shape of the power transmission structure 246 may be variously modified in consideration of the number and/or type of cables.

According to one embodiment, the microphone module 247 (eg, the input module 150 and/or the audio module 170 of FIG. 1 ) may convert sound into an electrical signal. According to one embodiment, the microphone module 247 may be disposed on at least a part of the lens frame 202 . For example, at least one microphone module 247 may be disposed at a lower end (eg, in a direction toward the -X axis) and/or an upper end (eg, in a direction toward the X axis) of the electronic device 200 . According to an embodiment, the electronic device 200 can more clearly recognize the user's voice using voice information (eg, sound) obtained from at least one microphone module 247 . For example, the electronic device 200 may distinguish voice information from ambient noise based on acquired voice information and/or additional information (eg, low-frequency vibration of the user's skin and bones). For example, the electronic device 200 can clearly recognize the user's voice and perform a function (eg, noise canceling) to reduce ambient noise.

According to an embodiment, the camera module 250 may capture still images and/or moving images. The camera module 250 may include at least one of a lens, at least one image sensor, an image signal processor, or a flash. According to one embodiment, the camera module 250 may be disposed within the lens frame 202 and disposed around the display member 201 .

According to one embodiment, the camera module 250 may include at least one first camera module 251 . According to an embodiment, the first camera module 251 may capture the trace of the user's eyes (eg, pupils) or gaze. For example, the first camera module 251 may capture a reflection pattern of light emitted from the light emitting unit to the user's eyes. For example, the light emitting unit may emit light in an infrared band for tracking a line of sight using the first camera module 251 . For example, the light emitting unit may include an IR LED. According to an embodiment, the processor (eg, the processor 120 of FIG. 1 ) may adjust the position of the virtual image projected on the display member 201 so that it corresponds to the direction in which the user's eyes gaze. According to an embodiment, the first camera module 251 may include a global shutter (GS) camera, and the user's eyes or The trajectory of the gaze can be traced.

According to various embodiments, the first camera module 251 periodically or aperiodically transmits information (eg, trajectory information) related to the trajectory of the user's eyes or gaze to a processor (eg, the processor 120 of FIG. 1 ). can be sent to According to another embodiment, when the first camera module 251 detects that the user's gaze has changed based on the trajectory information (eg, the eyes move more than a reference value while the head is not moving), the first camera module 251 converts the trajectory information to the processor can be sent to

According to one embodiment, the camera module 250 may include a second camera module 253. According to one embodiment, the second camera module 253 may capture an external image. According to an embodiment, the second camera module 253 may be a global shutter type camera or a rolling shutter (RS) type camera. According to an embodiment, the second camera module 253 may capture an external image through the second optical hole 223 formed in the second frame 202b. For example, the second camera module 253 may include a high resolution color camera and may be a high resolution (HR) or photo video (PV) camera. In addition, the second camera module 253 may provide an auto focus (AF) function and an optical image stabilizer (OIS) function.

According to various embodiments (not shown), the electronic device 200 may include a flash (not shown) located adjacent to the second camera module 253 . For example, a flash (not shown) may provide light for increasing the brightness (eg, illuminance) around the electronic device 200 when acquiring an external image of the second camera module 253, in a dark environment, It is possible to reduce difficulty in obtaining an image due to mixing of various light sources and/or reflection of light.

According to one embodiment, the camera module 250 may include at least one third camera module 255 . According to an embodiment, the third camera module 255 may capture a motion of the user through the first optical hole 221 formed in the lens frame 202 . For example, the third camera module 255 may capture a user's gesture (eg, hand motion). The third camera module 255 and/or the first optical hole 221 are both side ends of the lens frame 202 (eg, the second frame 202b), for example, the lens frame 202 in the X direction. (eg, may be disposed at both ends of the second frame 202b). According to an embodiment, the third camera module 255 may be a global shutter (GS) camera. For example, the third camera module 255 is a camera supporting 3 degrees of freedom (3DoF) or 6DoF to provide 360-degree space (eg, omnidirectional), location recognition, and/or movement recognition. can According to an embodiment, the third camera module 255 is a stereo camera, and uses a plurality of global shutter type cameras having the same standard and performance to perform simultaneous localization and mapping (SLAM) and user motion recognition. function can be performed. According to an embodiment, the third camera module 255 may include an infrared (IR) camera (eg, a time of flight (TOF) camera or a structured light camera). For example, the IR camera may operate as at least a part of a sensor module (eg, the sensor module 176 of FIG. 1 ) for detecting a distance to a subject.

According to an embodiment, at least one of the first camera module 251 and the third camera module 255 may be replaced with a sensor module (eg, the sensor module 176 of FIG. 1 ). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and/or a photodiode. For example, the photodiode may include a positive intrinsic negative (PIN) photodiode or an avalanche photodiode (APD). The photo diode may be referred to as a photo detector or a photo sensor.

According to an embodiment, at least one of the first camera module 251 , the second camera module 253 , and the third camera module 255 may include a plurality of camera modules (not shown). For example, the second camera module 253 is composed of a plurality of lenses (eg, wide-angle and telephoto lenses) and image sensors and is disposed on one side (eg, the side facing the -Y axis) of the electronic device 200. It can be. For example, the electronic device 200 may include a plurality of camera modules each having a different property (eg, angle of view) or function, and the angle of view of the camera module may be changed based on user selection and/or trajectory information. can be controlled to For example, at least one of the plurality of camera modules may be a wide-angle camera and at least the other may be a telephoto camera.

According to various embodiments, a processor (eg, the processor 120 of FIG. 1 ) acquires data using at least one of a gesture sensor, a gyro sensor, or an acceleration sensor of a sensor module (eg, the sensor module 176 of FIG. 1 ). Movement of the electronic device 200 using the information of one electronic device 200 and the user's motion acquired using the third camera module 255 (eg, the user's body approaching the electronic device 200) and/or determine the user's movement. According to an embodiment, the electronic device 200 may use a magnetic (geomagnetic) sensor capable of measuring a direction using a magnetic field and magnetic force in addition to the above-described sensor, and/or motion information (eg, movement) using the strength of a magnetic field. direction or moving distance) may be included. For example, the processor may determine the movement of the electronic device 200 and/or the movement of the user based on information obtained from a magnetic (geomagnetic) sensor and/or a Hall sensor.

According to various embodiments (not shown), the electronic device 200 may perform an input function capable of interacting with a user (eg, a touch and/or pressure sensing function). For example, a component configured to perform a touch and/or pressure sensing function (eg, a touch sensor and/or a pressure sensor) may be disposed on at least a portion of the wearing member 203 . The electronic device 200 may control a virtual image output through the display member 201 based on the information obtained through the components. For example, sensors related to touch and/or pressure sensing may be of the resistive type, capacitive type, electro-magnetic type (EM), or optical type. ) can be configured in a variety of ways. According to one embodiment, components configured to perform the touch and/or pressure sensing function may be all or partly the same as those of the input module 150 of FIG. 1 .

According to an embodiment, the electronic device 200 may include a reinforcing member 260 disposed in an inner space of the lens frame 202 and having a rigidity higher than that of the lens frame 202 .

According to an embodiment, the electronic device 200 may include a lens structure 270 . The lens structure 270 may refract at least a portion of light. For example, the lens structure 270 may be a prescription lens having a specified refractive power. According to one embodiment, the housing 210 may include a hinge cover 227 capable of concealing a portion of the hinge structure 229 . Another part of the hinge structure 229 may be accommodated or concealed between an inner case 231 and an outer case 233 to be described later.

According to various embodiments, the wearing member 203 may include an inner case 231 and an outer case 233 . The inner case 231 is, for example, a case configured to face the user's body or directly contact the user's body, and may be made of a material having low thermal conductivity, for example, synthetic resin. According to one embodiment, the inner case 231 may include an inner side facing the user's body (eg, the inner side 231c in FIG. 2 ). The outer case 233 includes, for example, a material capable of at least partially transferring heat (eg, a metal material), and may be coupled to face the inner case 231 . According to one embodiment, the outer case 233 may include an outer side opposite to the inner side 231c (eg, the outer side 231d of FIG. 2 ). In one embodiment, at least one of the circuit board 241 or the speaker module 245 may be accommodated in a space separated from the battery 243 within the wearing member 203 . In the illustrated embodiment, the inner case 231 includes a first case 231a including the circuit board 241 and/or the speaker module 245 and a second case 231b accommodating the battery 243. The outer case 233 may include a third case 233a coupled to face the first case 231a and a fourth case 233b coupled to face the second case 231b. can For example, the first case 231a and the third case 233a are coupled (hereinafter referred to as 'first case portions 231a and 233a') to accommodate the circuit board 241 and/or the speaker module 245. The second case 231b and the fourth case 233b may be coupled (hereinafter referred to as 'second case parts 231b and 233b') to accommodate the battery 243 .

According to one embodiment, the first case portions 231a and 233a are rotatably coupled to the lens frame 202 through a hinge structure 229, and the second case portions 231b and 233b are coupled to the connection structure 235 It may be connected to or mounted to the ends of the first case parts 231a and 233a through. In some embodiments, a portion of the connection structure 235 that comes into contact with the user's body may be made of a material with low thermal conductivity, for example, an elastic material such as silicone, polyurethane, or rubber. , A part that does not come into contact with the user's body may be made of a material with high thermal conductivity (eg, a metal material). For example, when heat is generated from the circuit board 241 or the battery 243, the connection structure 235 blocks the transfer of heat to a part in contact with the user's body and dissipates the heat through the part not in contact with the user's body. may be dispersed or released. According to one embodiment, a portion of the connection structure 235 configured to come into contact with the user's body may be interpreted as a part of the inner case 231, and a portion of the connection structure 235 that does not contact the user's body may be interpreted as an outer case ( 233) can be interpreted as part of According to one embodiment (not shown), the first case 231a and the second case 231b are integrally configured without a connection structure 235, and the third case 233a and the fourth case 233b are connected. It may be integrally configured without structure 235 . According to various embodiments, other components (eg, the antenna module 197 of FIG. 1) may be further included in addition to the illustrated components, and a network (eg, the first antenna module 197 of FIG. 1) may be used by using the communication module 190. Information about objects or environments may be provided from external electronic devices (eg, the electronic devices 102 and 104 of FIG. 1 or the server 108) through the first network 198 or the second network 199. .

5 is another perspective view of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 5 , the electronic device 400 may be a head mounting device (HMD) capable of providing an image in front of the user's eyes. The configuration of the electronic device 400 of FIG. 5 may be the same in whole or in part as the configuration of the electronic device 200 in FIG. 2 .

According to various embodiments, the electronic device 400 may include housings 410 and 420 that may form an exterior of the electronic device 400 and may provide a space in which parts of the electronic device 400 may be disposed. 430) can be included

According to various embodiments, the electronic device 400 may include a first housing 410 that may surround at least a portion of the user's head. According to an embodiment, the first housing 410 may include a first surface 400a facing the outside of the electronic device 400 (eg, -Y direction).

According to various embodiments, the first housing 410 may surround at least a portion of the inner space (I). For example, the first housing 410 may include a second surface 400b facing the inner space I of the electronic device 400 and a third surface 400c opposite to the second surface 400b. can According to one embodiment, the first housing 410 may be combined with the third housing 430 to form a closed curve shape surrounding the inner space (I).

According to various embodiments, the first housing 410 may accommodate at least some of the components of the electronic device 400 . For example, a light output module (eg, the light output module 211 of FIG. 3A ) and a circuit board (eg, the circuit board 241 and the speaker module 245 of FIG. 3A ) are the first housing 410 . can be placed within.

According to various embodiments, one display member 440 corresponding to the left and right eyes of the electronic device 400 may be included. The display member 440 may be disposed in the first housing 410 . The configuration of the display member 440 of FIG. 5 may be entirely or partially the same as that of the display member 201 of FIG. 2 .

According to various embodiments, the electronic device 400 may include a second housing 420 that may be seated on a user's face. According to one embodiment, the second housing 420 may include a fourth surface 400d that may at least partially face the user's face. According to an embodiment, the fourth surface 400d may be a surface in a direction toward the internal space I of the electronic device 400 (eg, +Y direction). According to one embodiment, the second housing 420 may be coupled to the first housing 410 .

According to various embodiments, the electronic device 400 may include a third housing 430 that may be seated on the back of the user's head. According to one embodiment, the third housing 430 may be coupled to the first housing 410 . According to an embodiment, the third housing 430 may accommodate at least some of the components of the electronic device 400 . For example, a battery (eg, the battery 243 of FIG. 3A ) may be disposed within the third housing 430 .

6 is a flowchart illustrating an operation of storing motion information of a user by an electronic device according to an embodiment of the present disclosure.

According to an embodiment, referring to FIG. 6 , in operation 610, an electronic device (eg, the electronic device 101 of FIG. 1 , the processor 120 of FIG. 1 , the electronic device 200 of FIG. 2 , or the electronic device 200 of FIG. 5 ) The electronic device 400 may acquire location information of a real space where the electronic device is located.

According to an embodiment, the electronic device may obtain location information of the electronic device through a sensor that detects the location of the electronic device (eg, the sensor module 176 of FIG. 1 ). For example, a sensor for detecting the position of an electronic device may be a GPS sensor. According to an embodiment, an electronic device may obtain location information by receiving a signal including location information from an external electronic device (eg, a beacon) disposed in a place.

According to an embodiment, in operation 620, the electronic device may acquire object information related to a plurality of real objects included in the real space.

According to an embodiment, the electronic device obtains a surrounding image through a camera (eg, the camera module 180 of FIG. 1 ), and based on the obtained image, the location and/or type of a real object disposed in the surrounding Related object information can be obtained. For example, the electronic device, based on the surrounding images obtained through the camera, based on the distance between the electronic device and the space (eg, wall) in the direction the electronic device faces and a real object disposed in the space, within the space. Object information including an actual object location (or arrangement) in may be obtained. According to an embodiment, the electronic device determines the type of real object (eg, what type of object is a key, desk, watch, or sofa) based on an image of a real object included in an acquired image and/or text included in the real object. object information related to recognition) may be obtained. According to an embodiment, the electronic device may include a sensor (eg, the sensor of FIG. 1 ) using at least one of a time of flight (TOF) method, a stereoscopic vision method, and a structured light pattern method. Object information related to the location of the real object may be obtained through module 176).

According to an embodiment, in operation 630, the electronic device may obtain motion information of the electronic device. For example, the electronic device may obtain motion information of the electronic device through a plurality of sensors included in the electronic device (eg, the sensor module 176 of FIG. 1 ).

For example, the electronic device may obtain motion information of the electronic device through a camera and/or a motion sensor such as an accelerometer, a gyroscope sensor, or a magnetic sensor. there is.

According to an embodiment, the electronic device may further obtain motion information of the user from an external device through a communication circuit (eg, the communication module 190 of FIG. 1 ). For example, the external device may be an electronic device capable of detecting a user's movement other than an external camera or an electronic device. According to an embodiment, the electronic device receives an image including a user's movement from an external camera (eg, a CCTV camera, a black box, or a camera included in a home appliance), and obtains user movement information based on the received image. can According to an embodiment, the electronic device may receive a sensing value related to the movement of the external electronic device and obtain motion information of the user based on the received sensing value. For example, the external electronic device may be a wearable electronic device worn by the user, such as a smart watch or smart band, held by the user, such as a smartphone, fixed to a part of the user's body, or in the user's clothes or bag. It may be an electronic device capable of detecting a user's movement.

According to an embodiment, in operation 640, when the motion information of the electronic device satisfies a set condition related to at least some of a plurality of real objects included in the real space, feature point information of the motion of a user wearing the electronic device is obtained. can For example, the electronic device may acquire feature point information of the user's movement based on the direction, distance, speed, and/or shaking of the movement or rotation of the electronic device obtained through a sensor (eg, a motion sensor). According to an embodiment, the feature point information of the user's motion may include motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.

According to an embodiment, the electronic device may identify that the knee is folded when the movement direction of the electronic device is downward in a vertical direction, and may identify that the waist is folded when the moving direction of the electronic device is in a downward parabolic direction. According to an embodiment, when the electronic device moves downward in a parabolic direction, the electronic device may identify that the neck is bent if the radius of the parabola is small, and identify that the waist is bent if the radius of the parabola is large.

According to an embodiment, the electronic device may further use an image obtained through a camera to obtain feature point information of the user's motion. For example, the electronic device may obtain feature point information including motion information of the user's movement path and parts of the user's body, such as the neck and waist, based on a real space included in an image obtained through a camera. According to an embodiment, the electronic device may obtain feature point information of the user's motion based on the motions of the user's hands and feet included in the image obtained through the camera.

According to an embodiment, when motion information of the user is further obtained from an external electronic device, the electronic device may further consider the received motion information to obtain feature point information of the motion of the user. For example, when an image including a user is obtained from an external electronic device, the electronic device may obtain feature point information of the user's movement by analyzing the user's movement included in the image. According to an embodiment, upon receiving motion information (eg, a sensing value of a motion sensor) from an external electronic device mounted on a part of the user's body, the electronic device obtains feature point information of the user's motion based on the received motion information. can

According to an embodiment, the set condition may be set based on at least one of location information of the electronic device, object information around the electronic device, and time information. For example, the set condition is a behavior that deviates from the user's existing behavior based on at least one of location information of the electronic device, object information around the electronic device, or time information, or a trigger for acquiring information on a characteristic point of the user's movement in advance. Can include designated movements.

For example, the time information may include time related to driving, performing activities such as fishing or diving, during rehabilitation, when a specific event occurs, when a specific schedule is scheduled, or when a user designates.

For example, the location information may include information on a location where the wearable electronic device is located and place information such as a house, a car, a hospital room, in the water, a hotel, a subway, a shop, a fishing spot, and a parking lot.

According to an embodiment, when the set condition is a behavior deviating from the user's existing behavior, a behavior different from the user's usual behavior (eg, a behavior of putting an object in a different location than usual) or a movement at the time of an accident, such as a fall. can include According to an embodiment, when the set condition is a motion designated in advance as a trigger for obtaining information on feature points of the user's motion, the set condition is when the user is located in a specific place or a specific action (eg, dance or pre-recorded action). can include

According to an embodiment, when a set condition is a behavior that deviates from a user's existing behavior, the electronic device is determined to satisfy the set condition when a difference between a motion pattern of the electronic device stored in a memory and motion information of the electronic device is equal to or greater than a set value. You can check. According to an embodiment, the movement pattern of the electronic device stored in the memory is acquired by the normal movement of a user wearing the electronic device, and is obtained by at least one of location information of the electronic device, object information around the electronic device, and time information. It may be related and learned. For example, the movement patterns of the electronic device stored in the memory may be learned by the electronic device or a server (eg, the server 108 of FIG. 1 ), based on a database associated with various movement patterns stored in the memory or the server. It can be obtained based on learning.

According to an embodiment, the movement pattern of the electronic device may be based on a user's normal movement related to at least some of a plurality of objects arranged in a specific place. According to an embodiment, the movement pattern of the electronic device may be a user's usual movement irrelevant to an object disposed in a specific place at a specific time. According to an embodiment, the movement pattern of the electronic device may be based on a user's usual movement related to at least some of a plurality of objects arranged in a specific place, at a specific time, and in a specific place.

According to an embodiment, the movement pattern of the electronic device stored in the memory is acquired by the movement of a user wearing the electronic device, and is associated with at least one of location information of the electronic device, object information around the electronic device, and time information. may have been learned.

According to an embodiment, an operation of determining whether a motion of an electronic device based on a user's motion satisfies a set condition will be described with reference to FIGS. 7 and 8 .

According to an embodiment, set conditions for obtaining information on feature points of user motion will be described with reference to [Table 1] and [Table 2] in the description of FIG. 10 .

According to an embodiment, the feature point information may include motion information of a plurality of joints of the user and motion information of bones connecting the plurality of joints. For example, the feature point information may include 3D movement information of some joints (eg, head, cervical spine, shoulder, elbow, wrist, lumbar spine, pelvis, knee) among a plurality of joints included in the user's body, and mutually adjacent joints of some joints. It may include 3D movement and rotation information of the bones connecting the . According to an embodiment, the feature point information may further include movement path information of the user based on movement information of joints or bones.

According to an exemplary embodiment, motion information of a plurality of joints and motion information of bones will be described with reference to FIG. 9 . In this way, by extracting feature point information rather than image or user motion information, resources required for data processing or storage may be reduced and user personal information may be protected.

According to an embodiment, in operation 650, the electronic device may store the obtained feature point information in a memory (eg, the memory 130 of FIG. 1 ). For example, the electronic device may store movement information of coordinates of feature points corresponding to a plurality of joints and movement information and angle information of a bone connecting a plurality of joints in a memory.

According to an embodiment, the electronic device may transmit acquired feature point information to a server through a communication circuit, and the server may store the feature point information received from the electronic device. In this way, when the minutiae point information is stored in the server, the minutiae point information is shared with other electronic devices than the electronic device from which the minutiae point information was obtained, and the other electronic devices can use the minutiae point information.

7 is a flowchart illustrating an operation of storing motion information of a user of an electronic device according to an embodiment of the present disclosure.

According to one embodiment, in operation 701, the database 700 (eg, the server 108 of FIG. 1 or the memory 130 of FIG. 1) stores pre-designated basic values, and continuously converts user logs into big data. can be processed with For example, a pre-designated basic value is a trigger for obtaining information on feature points of a user's movement, and may include information related to a pre-designated movement, and a continuous user log accumulates a specific user's usual behavior and manages it as big data. may include doing

In FIG. 7 , it has been illustrated and described that the database 700 stores preset default values and processes user logs as big data, but according to an embodiment, the electronic device 101 stores preset default values and continuously User logs can also be processed as big data.

According to an embodiment, in operation 702, the electronic device 101 (eg, the electronic device 101 of FIG. 1 , the processor 120 of FIG. 1 , or the electronic device 200 of FIG. 2 ) selects an external object as a camera. (eg, the camera module 180 of FIG. 1). For example, the electronic device 101 may be a wearable electronic device worn by a user, recognize a real object included in a space where the electronic device 101 is located, and obtain object information through a camera. For example, object information may include at least one of location information and type information of a real object.

According to an embodiment, in operation 703, the electronic device 101 may check whether the user is interacting with an external object. For example, the electronic device 101 may capture the motion of the user through a camera and determine whether the user is interacting with an external object (eg, the user puts a car key on a table). According to an embodiment, the electronic device 101 detects the user's motion through a sensor (eg, the sensor module 176 of FIG. 1 ) so that the user interacts with an external object (eg, the user sits on a sofa). You can check whether it is running or not.

According to an embodiment, if it is determined that the user is not interacting with an external object (Operation 703 - No), in Operation 704, the electronic device 101 may check whether the user's motion satisfies a set condition. According to an embodiment, the electronic device 101 may check whether a condition set based on a preset basic value stored in the database 700 or big data of a user is satisfied.

For example, the set condition may be set based on at least one of location information of the electronic device 101 , object information around the electronic device 101 , and time information. For example, the condition set in operation 704 is an action that deviates from the user's existing action based on at least one of location information and time information of the electronic device 101, or a trigger specified in advance as a trigger for obtaining information on feature points of the user's movement. movement may be included.

For example, if the set condition is a behavior that deviate from the user's existing behavior based on the user's big data, it may include behavior different from the user's usual behavior (e.g., moving while sleeping normally). there is. According to an embodiment, when a set condition is a movement (eg, a pre-designated default value) as a trigger for acquiring information on feature points of a user's movement, when the user is located in a specific place, or when the user takes a specific action (eg, dancing or pre-recorded actions).

According to an embodiment, when the set condition is a behavior that deviates from the user's existing behavior, the electronic device 101 determines the difference between the motion pattern of the electronic device 101 stored in the database 700 and the motion information of the electronic device as a set value. If it is above, it can be confirmed that the set condition is satisfied. According to an embodiment, the movement pattern of the electronic device 101 stored in the database 700 is acquired by the user's usual movement wearing the electronic device 100, and the location information of the electronic device 100, electronic It may be learned in association with at least one of object information or time information around the device 101 . For example, the motion patterns of the electronic device stored in the database 700 may be learned by the electronic device 101 or a server (eg, the server 108 of FIG. It may be learned by an artificial intelligence model stored in the memory 130) or a server.

According to an embodiment, if the user's motion does not satisfy the set condition (Operation 704 - No), in Operation 705, the electronic device 101 may discard the recorded material. For example, the recorded material may be information related to the movement of the electronic device 101 sensed by the sensor.

According to an embodiment, if the user's motion satisfies the set condition (Operation 704-Yes), in Operation 706, the electronic device 101 may store the recorded material. For example, the recorded material is information related to the movement of the electronic device 101 sensed by the sensor, and may include feature point information obtained based on the movement of the electronic device 101 . According to an embodiment, the feature point information includes motion information of a plurality of joints of a user wearing the electronic device 101 obtained based on motion of the electronic device 101 and motion information of bones connecting the joints. can include For example, the minutiae point information may include 3D movement information of a plurality of joints of the user, 3D movement information and rotation information of a bone. According to an embodiment, the feature point information may further include movement path information of the user based on movement information of joints or bones.

According to an embodiment, when it is confirmed that the user is interacting with an external object (Operation 703-Yes), in Operation 707, the electronic device 101 may determine whether the movements of the user and the external object satisfy a set condition. there is. According to an embodiment, the electronic device 101 may check whether a condition set based on a preset basic value stored in the database 700 or big data of a user is satisfied.

For example, the set condition may be set based on at least one of location information of the electronic device 101 , object information around the electronic device 101 , and time information. For example, the condition set in operation 707 is an action that deviate from the user's existing action based on at least one of location information of the electronic device 101, object information around the electronic device 101, and time information, or a characteristic point of the user's movement. A pre-specified motion may be included as a trigger for obtaining information of .

For example, if the set condition is a behavior that deviate from the user's existing behavior based on the user's big data, it includes behavior different from the user's usual behavior (e.g., behavior of putting things in a different location than usual). can do. According to an embodiment, when a set condition is a movement (eg, a pre-designated default value) as a trigger for acquiring information on feature points of a user's movement, when the user is located in a specific place, or when the user takes a specific action (eg, An action of getting out of a car in a parking lot and walking to the entrance of a building).

According to an embodiment, when the set condition is a behavior that deviates from the user's existing behavior, the electronic device 101 determines the difference between the motion pattern of the electronic device 101 stored in the database 700 and the motion information of the electronic device as a set value. If it is above, it can be confirmed that the set condition is satisfied. According to an embodiment, the movement pattern of the electronic device 101 stored in the database 700 is acquired by the user's usual movement wearing the electronic device 100, and the location information of the electronic device 100, electronic It may be learned in association with at least one of object information or time information around the device 101 . For example, the motion pattern of the electronic device stored in the database 700 may be learned by the electronic device 101 or a server (eg, the server 108 of FIG. It may be learned by an artificial intelligence model stored in the memory 130) or a server.

According to an embodiment, if the user's motion does not satisfy the set condition (Operation 707 - No), in Operation 705, the electronic device 101 may discard the recorded material. For example, the recorded material may be information related to the movement of the electronic device 101 sensed by the sensor.

According to an embodiment, if the user's movement satisfies the set condition (Operation 707 - Yes), in Operation 706, the electronic device 101 may store the recorded material. For example, the recorded material is information related to the movement of the electronic device 101 sensed by the sensor, and may include feature point information obtained based on the movement of the electronic device 101 . According to an embodiment, the feature point information includes motion information of a plurality of joints of a user wearing the electronic device 101 obtained based on motion of the electronic device 101 and motion information of bones connecting the joints. can include For example, the feature point information may include 3D movement information of a plurality of joints of the user, 3D movement information and rotation information of a bone. According to an embodiment, the feature point information may further include movement path information of the user based on movement information of joints or bones.

8 is a flowchart illustrating an operation of determining whether motion information of a user satisfies a set condition according to an embodiment of the present disclosure.

According to one embodiment, in operation 801, the server 108 (eg, the server 108 of FIG. 1 ) retrieves time, place, and behavior factors from a server that stores behavioral data (or predefined conditions) of the majority of people. You can search.

According to one embodiment, in operation 802, the server 108 may analyze the sampled data. For example, server 108 may calculate degrees of freedom, expected frequencies, test statistics, and associated P values. According to one embodiment, server 108 may analyze recurring patterns in the majority of people's behavioral data. For example, the server 108 may analyze a repeating pattern of behavioral elements of a large number of people at a specific time and a specific place, and obtain the repeating pattern as sample data. According to an embodiment, the server 108 may obtain degrees of freedom, expected frequencies, test statistics, and associated P-values by analyzing repeated patterns.

According to an embodiment, in operation 803, the electronic device 101 (eg, the electronic device 101 of FIG. 1 or the processor 120 of FIG. 1 ) or the electronic device 200 of FIG. 2 performs a user action can be detected and/or recorded. For example, the electronic device 101 detects and/or records a user's action through a sensor (eg, sensor module 176 in FIG. 1 ) and/or a camera (eg, camera module 180 in FIG. 1 ). can do.

According to an embodiment, in operation 804, the electronic device 101 may pre-process detected and/or recorded user behavior data. For example, the electronic device 101 may classify the user's behavior data into time, place, or action.

According to an embodiment, in operation 805, the electronic device 101 may generate a data subset (subset). For example, the electronic device 101 may generate a subset of user behavior data classified into time, place, or action by combining at least one of time, place, or action. For example, the electronic device 101 may generate a subset of user behavior data combining at least one of a specific time, a specific place, and a specific action.

According to an embodiment, the electronic device 101 receives sample data from the server 108 and generates a data subset by combining at least one of a specific time, a specific place, or a specific action based on the received sample data. can

According to an embodiment, in operation 806, the electronic device 101 may generate descriptive statistics. For example, the electronic device 101 may obtain at least one of an average or a scatter plot of the data subset. For example, at least one of averages or scatter plots of a plurality of subsets of data generated by combining at least one of a specific time, a specific place, and a specific action may be obtained.

According to an embodiment, in operation 807, the electronic device 101 may analyze the homogeneity distribution. For example, the electronic device 101 may compare the sample data received from the server 108 with the descriptive statistics of the data subset to determine whether a standard deviation exceeds 1.

According to an embodiment, if the standard deviation exceeds 1 (action 807-yes) by comparing the descriptive statistics of the sample data and the data subset, the electronic device 101 transmits the user's behavior data to the server 108 and , in operation 808, the server 108 may store the user's behavior data received from the electronic device 101. For example, the electronic device 101 compares the descriptive statistics of the sample data and the data subset, and if the standard deviation exceeds 1, it is determined that the user's behavior is different from the usual behavior pattern, and the user's behavior data can be saved.

According to an embodiment, if the standard deviation does not exceed 1 by comparing the descriptive statistics of the sample data and the data subset (Operation 807-No), in Operation 809, the electronic device 101 does not record the user's behavior data. may not be For example, the electronic device 101 compares the sample data and the descriptive statistics of the data subset, determines that the user's behavior is a normal behavior pattern if the standard deviation does not exceed 1, and does not record the user's behavior data. may not be

9 is a diagram for explaining feature point information of a user's motion according to an embodiment of the present disclosure.

According to an embodiment, referring to FIG. 9 , feature point information may include motion information of a plurality of joints 910 of a user and motion information of a bone 920 connecting the plurality of joints.

According to an embodiment, an electronic device (eg, the electronic device 101 of FIG. 1 , the processor 120 of FIG. 1 , or the electronic device 200 of FIG. 2 ) is a sensor (eg, the sensor module 176 of FIG. 1 ). ) obtained through motion information of the electronic device, a user image obtained through an external camera (eg, the electronic device 104 of FIG. 1 ), or an external electronic device (eg, the electronic device 104 of FIG. 1 ). Based on at least one of the acquired motion information of the user, motion information of the user's main joints 910 (eg, head, cervical spine, shoulder, elbow, wrist, lumbar spine, knee, ankle) may be obtained. For example, the motion information of the main joint 910 of the user may be 3D movement information of coordinates of the joint 910 .

According to an embodiment, the bone 920 may be obtained by connecting two adjacent major joints 910 based on the skeletal structure of the human body. For example, the bone 920 may connect the head and cervical vertebrae, the cervical vertebrae and shoulders, the shoulders and elbows, the elbows and wrists, the cervical vertebrae and lumbar vertebrae, the lumbar vertebrae and knees, and the knees and ankles, respectively.

According to an embodiment, the motion information of the bone 920 is motion information of a line connecting two points (eg, the joint 910), and may include at least one of 3D movement information and rotation information of the line.

In this way, by extracting feature point information rather than entire image or motion information of a user, resources required for data processing or storage may be reduced and personal information of the user may be protected.

10 is a flowchart illustrating an operation of an electronic device to display a 3D virtual object image based on stored feature point information, according to an embodiment of the present disclosure.

According to an embodiment, referring to FIG. 10 , in operation 1010, an electronic device (eg, the electronic device 101 of FIG. 1 , the processor 120 of FIG. 1 , the electronic device 200 of FIG. 2 , or the electronic device 200 of FIG. 5 ) The electronic device 400 may obtain location information of a real space where the electronic device is located.

According to an embodiment, the electronic device may obtain location information of the electronic device through a sensor that detects the location of the electronic device (eg, the sensor module 176 of FIG. 1 ). For example, a sensor for detecting the position of an electronic device may be a GPS sensor. According to an embodiment, an electronic device may obtain location information by receiving a signal including location information from an external electronic device (eg, a beacon) disposed in a place.

According to an embodiment, in operation 1020, the electronic device may acquire object information related to a plurality of real objects included in the real space.

According to an embodiment, the electronic device obtains a surrounding image through a camera (eg, the camera module 180 of FIG. 1 ), and based on the obtained image, the location and/or type of a real object disposed in the surrounding Related object information can be obtained. For example, the electronic device may obtain object information related to the type of the real object based on the image of the real object included in the obtained image or the text included in the real object. According to an embodiment, the electronic device may include a sensor (eg, the sensor of FIG. 1 ) using at least one of a time of flight (TOF) method, a stereoscopic vision method, and a structured light pattern method. The location information of the real object may be obtained through the module 176).

According to an embodiment, in operation 1030, if the electronic device stores feature point information of a user's movement related to location information and/or object information in a memory (eg, the memory 130 of FIG. 1 ), the feature point information is stored. A virtual object notifying that there is a virtual object may be displayed on a display (eg, the display module 160 of FIG. 1 ).

According to an embodiment, feature point information may be obtained by an operation of an electronic device as shown in FIG. 6 and stored in a memory or a server (eg, the server 108 of FIG. 1 ). For example, the feature point information may be stored in a memory or a server prior to a current point in time when the electronic device is located in a real space.

According to an embodiment, motion information of an electronic device obtained through a sensor (eg, the sensor module 176 of FIG. 1 ) in a real space, before the keypoint information is stored in a memory, is stored in a memory. If a set condition related to at least some of the included real objects is satisfied, information on feature points of motions of a user wearing an electronic device acquired based on motion information of the electronic device may be stored in the memory. For example, the feature point information may include motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints. For example, the minutiae point information may include 3D movement information of a plurality of joints of the user, 3D movement information and rotation information of a bone. According to an embodiment, the feature point information may further include movement path information of the user based on movement information of joints or bones.

According to an embodiment, feature point information may be acquired by an electronic device different from an electronic device and stored in a server. For example, an electronic device from which feature point information is acquired and a device that reproduces a 3D virtual object image based on the feature point information may be different devices.

According to an embodiment, the electronic device further obtains motion information of the user from the external electronic device through a communication circuit (eg, the communication module 180 of FIG. 1 ), and further considers the motion information of the user obtained from the external device. Thus, feature point information can be obtained. For example, the external electronic device may be an external camera (eg, a CCTV camera, a black box, or a camera included in home appliances), a wearable electronic device, or a terminal device such as a smartphone.

According to an embodiment, the feature point information may be obtained by further considering motion information of the user obtained based on an image including the motion of the user received from an external camera. According to an embodiment, the feature point information may be obtained by further considering motion information of the user obtained based on a sensing value related to the motion of the external electronic device received from the external wearable electronic device or the external terminal device.

According to an embodiment, feature point information may be stored in a memory or a server.

According to an embodiment, when feature point information of a user's motion related to a real space in which the electronic device is located is stored, the electronic device displays a virtual object informing that the feature point information exists on a part of the space or a real object related to the feature point information. You can control the display to For example, the virtual object may be in the form of a pop-up screen or icon, and may be a user interface (UI) that can be selected to display a 3D virtual object image based on feature point information.

According to an embodiment, in operation 1040, when a user input for displaying an image related to feature point information is received through a virtual object, the electronic device may obtain a 3D virtual object image based on the feature point information.

According to an embodiment, a 3D virtual object image may include a 3D virtual character capable of expressing body movements. According to an embodiment, the electronic device may obtain a 3D virtual object image including a 3D virtual character moving or moving in a space based on feature point information.

According to an embodiment, the 3D virtual object image may further include at least one virtual object respectively related to at least one of a plurality of real objects included in the real space. For example, the 3D virtual object image may further include a virtual object related to a real object disposed in a real space when feature point information is acquired together with a 3D virtual character. For example, when feature point information related to a user's action of putting a car key in a different location from a position where he normally puts it is stored, a 3D virtual object image obtained based on the feature point information includes a virtual object related to the car key and a car key related to the car key. It may include a 3D virtual character performing an action of placing the virtual object in a position different from a position in which the virtual object is normally placed.

According to an embodiment, a situation in which feature point information is obtained and an image displayed based on the feature point information are shown in [Table 1] below.

Significant behavior of the first user Reproduction of AR 3D characters 1st hour 1st place Behavior of the first user device that records behavior 2nd hour 3D Character Behavior device that reproduces the action purpose driving in the car perform wrong operation Camera, AR glass When you want to play later Avatar reproduces incorrect operation behavior In the Accident Investigator's AR glass Accident situation reproduction through driver behavior tracking, not mechanical record tracking in case of incident in the hospital room fall in bed camera When you want to play later Avatar fell out of bed In the nurse's AR glass Accident situation reproduction of patients with poor communication (elderly, infant, etc.) in case of incident at home Leaving the car key under the sofa instead of in the designated place AR glass When you want to play later Avatar puts the car key under the sofa In the family's AR glass Make shared items easier to find if misplaced in rehabilitation all space When you unconsciously do something that is not good for rehabilitation in your life AR glass When you want to play later Avatar reproduces the wrong behavior pattern. In the rehabilitation doctor's AR glass Correct rehabilitation is possible through the correction of wrong life behavior patterns. diving in the water Sending signals to co-workers AR glass real time Avatar reproduces hand signal Colleague's AR glass Enables hand signal communication when there is no visibility in the water or when a colleague is far away Entering the living room without taking off shoes (to go out in a hurry) Do not close the lid after using detergent or cooking ingredients On the birthday of friend A registered in the scheduler at home if you dance camera right after Avatar dances happy birthday dance In friend A's AR glass Send a happy birthday message to a friend with a dancing avatar when you want to record in the hotel room Point with your hand where you insert the key, the location of the refrigerator, the location of the safe, and the location of the welcome drink. AR glass When a guest checks into a hotel Avatar points to the key insertion location, refrigerator location, safe location, and welcome drink location. In your AR glass Carrying out easy-to-understand guidance instead of a hotel bellboy when you want to record at the subway ticket office Walk to the kiosk, insert money and buy a ticket AR glass When the customer wants Avatar walks to the kiosk, inserts money and buys a ticket In the AR glass of the user Users can easily know the location of the ticket kiosk and how to use it when getting out of the car in the parking lot Get out of the car and walk to the entrance of the building. AR glass 1When the user returns to the parking lot Avatar walks backwards from building entrance to car (rewind) 1In the user's AR glass You can easily know where you parked by following your avatar when fishing is successful at the fishing spot catch a fish camera when you want to play Avatar catches a fish 1 In the AR glass of the user or a third party who allowed sharing Easily record and share success stories when you want to record in store Assemble the furniture. camera when you want to play Avatar assembles furniture. In the AR glass of the customer who purchased the furniture You can easily assemble furniture by following the steps of the avatar.

According to an embodiment, data included in a 3D virtual object image may be as shown in [Table 2]. For example, the 3D virtual object image may include a virtual character to express a user's expression and movement, clothes worn by the character, and a virtual object corresponding to a real object.

division 3D Data Type explanation movement One Character model Character model to reproduce the behavior of the first user Displays the character model set by the user or designated by the system.
*When reproducing the motion, if the user is different, each emoji is reproduced. 2 Clothes Clothes worn by the character - clothes set by the user
- Clothes worn by the first user The character model wears the clothes set by the second user or clothes similar to the clothes worn by the first user. 3 Additional objects Types of 3D object-devices attached to the character body or displayed together in 3D space: mobile phones, AR glass, voice output devices (Buds, speakers, headsets), health aids (blood pressure monitors, weight scales), input devices (keyboards, joysticks) etc.
-Accessories: exercise equipment (yoga mat, ball, dumbbell), umbrella, car keys, keys, etc. The character model displays the object in the space as the first user wears and moves the object. 4 Facial motion Face movement imitated by the character to convey meaning during playback The character model performs the facial motion loaded from the server. 5 body motion Motion movement imitated by the character to convey meaning during playback The character model performs the body motion loaded from the server.

According to an embodiment, in operation 1050, the electronic device may display a 3D virtual object image on the display. For example, the electronic device may display a 3D virtual object image including a 3D virtual character and a virtual object related to a real object on a display.

According to an embodiment, when the real object is an external electronic device communicating with the electronic device, the electronic device may further display an application screen related to the real object together with a 3D virtual object image.

11A is a diagram for explaining a case where an electronic device stores motion information of a user according to an embodiment of the present disclosure. For example, the user shown in FIG. 11A may be wearing an electronic device (eg, the electronic device 101 of FIG. 1 ), and the image acquired by the electronic device is for explaining the user's behavior. It may not be. According to an embodiment, the image including the user of FIG. 11A may be obtained from an external camera device.

According to an embodiment, referring to (a) of FIG. 11A , the user 10 may normally place the car key 1110 on the rectangular table 1111 . According to an embodiment, the action of the user 10 placing the car key 1110 on the rectangular table 1111 in normal times may have been learned, and the electronic device may include location information of the space where the electronic device is located, information included in the space A user's usual behavior may be learned based on at least one of object information and time information of a real object.

According to an embodiment, referring to (b) of FIG. 11A , when the user 10 puts the car key 1110 on the circular table 1120, the electronic device behaves differently from the learned behavior of the user 10. It can be confirmed that the set condition for action is satisfied.

According to an embodiment, the electronic device obtains motion information of the user 10 placing the car key 1110 on the circular table 1120 from a sensor (eg, the sensor module 176 of FIG. 1 ), and obtains the obtained information. Feature point information may be obtained based on motion information. According to an embodiment, the electronic device receives an image in which the user's movement is captured from an external camera, or receives motion information of the user from the external electronic device, and based on the received image and motion information, information on feature points of the user's movement. can be obtained According to an embodiment, the feature point information may include motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.

FIG. 11B is a diagram for explaining an operation of displaying a 3D virtual object image related to stored user motion information by an electronic device when the electronic device is located in a place where the user motion information of FIG. 11A is stored, according to an embodiment of the present disclosure. it is a drawing

According to an embodiment, referring to (a) of FIG. 11B , after the feature point information is stored, when the electronic device 101 is located again in the space where the feature point information was obtained, the electronic device 101 is related to the space, A virtual object 1130 indicating that feature point information related to an object included in the space (eg, the rectangular table 1111) is stored may be displayed on a display (eg, the display module 160 of FIG. 1 ). For example, the virtual object 1130 notifying that feature point information is stored may be in the form of a pop-up screen or icon.

According to an embodiment, although one virtual object 1130 is shown to be displayed in (a) of FIG. 11B, when two or more feature point information related to a space is stored, two or more virtual objects may be displayed.

According to an embodiment, when the electronic device 101 receives a gesture input of the user 10 selecting the virtual object 1130 notifying that feature point information is stored, as shown in (b) of FIG. 11B, , a 3D virtual object image generated based on stored feature point information may be displayed.

According to an embodiment, referring to (b) of FIG. 11B, the electronic device 101 displays a 3D virtual character 1140 putting a car key on a round table 1120 instead of a square table normally placed there. An object image may be displayed on a display. According to an embodiment, the 3D virtual object image may further include a virtual object 1141 corresponding to a car key held in the hand of the 3D virtual character 1140 .

In this way, the electronic device may provide the stored specific behavior of the user to the user by displaying the stored motion information of the user through the 3D virtual object image.

12A is a diagram for explaining a case where an electronic device stores motion information of a user according to an embodiment of the present disclosure. For example, the user shown in FIG. 12A may be in a state wearing an electronic device (eg, the electronic device 101 of FIG. 1 ), and an image acquired by the electronic device is for explaining the user's behavior. It may not be. According to an embodiment, the image including the user of FIG. 12a may be obtained from an external camera device.

According to an embodiment, as shown in (a) of FIG. 12A, after the user 10 parks a car 1210 in a parking lot, as shown in (b) of FIG. 12A, the entrance 1220 of the building ) can be directed to According to an embodiment, when a condition set for obtaining feature point information of a user's motion is a specific action (eg, an action of entering a building after parking), the electronic device may obtain location information of the electronic device and information of recognized surrounding objects. Based on this, if the user's motion entering the building entrance 1220 after parking the car 1210 is confirmed, it can be confirmed that the user's motion satisfies the set condition.

According to an embodiment, the electronic device acquires location information where the car 1210 is parked and motion information of the user 10 related to a movement path while the user 10 goes to the building entrance 1220 after parking, Feature point information may be obtained based on the acquired motion information. According to an embodiment, the electronic device receives an image in which the user's movement is captured from an external camera, or receives motion information of the user from the external electronic device, and based on the received image and motion information, information on feature points of the user's movement. can be obtained According to an embodiment, the feature point information may include movement path information of the user, motion information of a plurality of joints of the user, and motion information of a bone connecting the plurality of joints.

FIG. 12B is a diagram for explaining an operation of displaying a 3D virtual object image related to stored user motion information by an electronic device when the electronic device is located in a place where the user motion information of FIG. 12A is stored, according to an embodiment of the present disclosure. it is a drawing

According to an embodiment, referring to (a) of FIG. 12B , after the feature point information is stored, if the electronic device 101 is located again in the space where the feature point information was obtained, the electronic device 101 is associated with the space, A virtual object 1230 indicating that feature point information related to the object 1220 (eg, a building entrance) included in the space is stored may be displayed on a display (eg, the display module 160 of FIG. 1 ). For example, the virtual object 1230 notifying that feature point information is stored may be in the form of a pop-up screen or icon.

According to an embodiment, although one virtual object 1230 is shown to be displayed in (a) of FIG. 12B, when two or more feature point information related to a space is stored, two or more virtual objects may be displayed.

According to an embodiment, when the electronic device 101 receives a gesture input of the user 10 for selecting the virtual object 1230 notifying that feature point information is stored, the electronic device 101 displays (b) and (c) of FIG. 12b. As shown, a 3D virtual object image generated based on stored feature point information may be displayed.

According to an embodiment, referring to (b) and (c) of FIG. 12B , the electronic device 101 includes a 3D virtual character 1240 moving from a building entrance to a parked car based on user feature point information. The 3D virtual object image may be displayed on the display.

As such, the electronic device may provide the user with information about the user's movement that satisfies the set condition by displaying the user's movement information stored through the 3D virtual object image.

13A is a diagram for explaining a case where an electronic device stores user motion information according to an embodiment of the present disclosure. For example, the user shown in FIG. 13A may be wearing an electronic device (eg, the electronic device 101 of FIG. 1 ), and an image acquired by the electronic device is for explaining the user's behavior. It may not be. According to an embodiment, the image including the user of FIG. 13A may be obtained from an external camera device.

According to an embodiment, as shown in (a) of FIG. 13A, the user 10 performs an operation of manipulating a control board 1310 capable of controlling room environments such as brightness and temperature, and FIG. 13A As shown in (b) of, an operation using the home appliance 1320 included in the room may be performed. According to an embodiment, when a set condition for obtaining feature point information of a user's motion is an operation after a user inputs a recording command, the electronic device acquires location information of the electronic device and information about a recognized surrounding object, and obtains the location information. And motion information of the user 10 may be obtained based on the object information. For example, movement information may be used to describe the location and usage of appliances (eg control boards, electric kettles or refrigerators) or items (eg welcome drinks, closets or amenities) placed in a particular space (eg hotel room). If necessary, the set condition may be an operation after a user inputs a recording command. According to an embodiment, (a) and (b) of FIG. 13a may be the continuous movement of the user 10.

According to an embodiment, the electronic device may obtain feature point information based on motion information of the user 10 . According to an embodiment, the electronic device receives an image in which the user's movement is captured from an external camera, or receives motion information of the user from the external electronic device, and based on the received image and motion information, information on feature points of the user's movement. can be obtained According to an embodiment, the feature point information may include movement path information of the user, motion information of a plurality of joints of the user, and motion information of a bone connecting the plurality of joints.

FIG. 13b illustrates an operation of displaying a 3D virtual object image related to the stored user motion information by the electronic device when the electronic device is located in the place where the user motion information of FIG. 13A (a) is stored, according to an embodiment of the present disclosure. It is a drawing for explaining.

According to an embodiment, referring to (a) of FIG. 13B , after feature point information is stored, if the electronic device 101 is located again in the space where the feature point information was acquired, the electronic device 101 is related to the space, A virtual object 1330 indicating that feature point information related to the object 1310 (eg, a control board) included in the space is stored may be displayed on a display (eg, the display module 160 of FIG. 1 ). For example, the virtual object 1330 notifying that feature point information is stored may be in the form of a pop-up screen or icon.

According to an embodiment, although one virtual object 1330 is shown to be displayed in (a) of FIG. 13B, when two or more feature point information related to a space is stored, two or more virtual objects may be displayed.

According to an embodiment, when the electronic device 101 receives a gesture input of the user 10 for selecting the virtual object 1330 notifying that feature point information is stored, as shown in (b) of FIG. 13B , a 3D virtual object image generated based on stored feature point information may be displayed.

According to an embodiment, referring to (b) of FIG. 13B, the electronic device 101 has a 3D virtual character 1340 guiding the location and manipulation method of the control board 1310 based on user feature point information. The included 3D virtual object image may be displayed on the display.

FIG. 13C illustrates an operation of displaying a 3D virtual object image related to stored user motion information by the electronic device when the electronic device is located in the place where the user motion information of FIG. 13A (b) is stored, according to an embodiment of the present disclosure. It is a drawing for explaining.

According to an embodiment, referring to (a) of FIG. 13C , after the feature point information is stored, if the electronic device 101 is located again in the space where the feature point information was obtained, the electronic device 101 is associated with the space, A virtual object 1350 indicating that feature point information related to the object 1320 (eg, home appliance) included in the space is stored may be displayed on a display (eg, the display module 160 of FIG. 1 ). For example, the virtual object 1350 notifying that feature point information is stored may be in the form of a pop-up screen or icon.

According to an embodiment, although one virtual object 1350 is shown to be displayed in (a) of FIG. 13C, when two or more feature point information related to a space is stored, two or more virtual objects may be displayed.

According to an embodiment, when the electronic device 101 receives a gesture input of the user 10 selecting a virtual object 1350 notifying that feature point information is stored, as shown in (b) of FIG. 13C , a 3D virtual object image generated based on stored feature point information may be displayed.

According to an embodiment, referring to (b) of FIG. 13C, the electronic device 101 includes a 3D virtual character 1340 for guiding the location and manipulation method of the home appliance 1320 based on user feature point information. The 3D virtual object image may be displayed on the display.

According to an embodiment, in FIGS. 13B and 13C , a virtual object indicating that feature point information is stored in each of the real objects (eg, the control board 1310 of FIG. 13B and the home appliance 1320 of FIG. 13C ) related to the stored feature point information is stored. Although the objects 1330 and 1350 are shown to be displayed, in actual implementation, as the user's motion for a plurality of real objects is continuous after inputting a recording command, FIGS. 13B(B) and 13C(B) show 3D virtual object images may be continuously reproduced.

As such, the electronic device may provide the user with information about the user's movement that satisfies the set condition by displaying the user's movement information stored through the 3D virtual object image.

14 is a diagram for explaining a communication connection form of an electronic device according to an embodiment of the present disclosure.

According to an embodiment, referring to FIG. 14 , the server 108 (eg, the server 108 of FIG. 1 ) of the 3D virtual object image providing system includes a plurality of electronic devices 101-1 and 101-2 (eg, the server 108). : It can communicate with the electronic device 101 of FIG. 1). For example, the plurality of electronic devices 101-1 and 101-2 may be wearable electronic devices such as smart glasses and HMD devices capable of reproducing 3D virtual object images.

According to an embodiment, the server 108 may include a memory in which the database DB is stored, a communication module for communicating with external devices, and at least one processor operatively connected to the memory and communication module.

According to an embodiment, the electronic devices 101-1 and 101-2 may include a plurality of sensors (eg, the sensor module 176 of FIG. 1), and the electronic devices 101-1 and 101-2 It is possible to obtain location information where is located, and since it is attached to the user's body, a sensing value related to the user's movement can be obtained.

According to an embodiment, the electronic devices 101-1 and 101-2 may include a camera (eg, the camera module 1180 of FIG. 1), and the electronic devices 101-1 and 101-2 are located. Object information may be obtained by recognizing an object included in the space.

According to an embodiment, the electronic devices 101-1 and 101-2 may communicate with a plurality of external electronic devices. For example, the electronic devices 101-1 and 101-2 may include external cameras 1411-1 and 1411-2, external electronic devices 1410-1 and 1410-2 (eg, smartphones), or external wearable devices. It may communicate with the electronic devices 1412-1 and 1412-2.

According to an embodiment, the electronic devices 101-1 and 101-2 may receive a captured image of a user or information (eg, a sensing value) related to a user's movement from a plurality of external electronic devices. For example, when the electronic device 101-1 or 101-2 confirms the user's motion that satisfies the set condition, the captured image or the acquired user's motion Information related to may be received from an external electronic device. According to an embodiment, the electronic devices 101-1 and 101-2 obtain feature point information of the user's movement based on a sensing value obtained by a sensor and information related to an image or user's movement received from an external electronic device. can do.

According to an embodiment, the electronic devices 101-1 and 101-2 may store feature point information in a memory (eg, the memory 130 of FIG. 1) or the server 108. According to an embodiment, when the feature point information is stored in the server 108, the 3D virtual object image obtained based on the feature point information obtained by the first electronic device 101-1 is stored in the second electronic device 101-1. 2) can also be indicated by

According to an embodiment, it has been described that the electronic devices 101-1 and 101-2 acquire feature point information by receiving images or motion information of the user from external electronic devices, but the external electronic devices 1410-1 and 1410- 2) (e.g., smart phone) sensed values acquired by the electronic devices 101-1 and 101-2, images acquired by the external cameras 1411-1 and 1411-2, or external wearable electronic devices 1412 Based on at least one of the user motion information acquired by steps -1 and 1412-2), feature point information of the user's motion may be obtained. According to an embodiment, the external electronic devices 1410-1 and 1410-2 (eg, smart phones) may include sensing values acquired by the electronic devices 101-1 and 101-2, external cameras 1411-1, 1411-2) or the user movement information obtained by the external wearable electronic devices 1412-1 and 1412-2 are transmitted to the server 108, and the server 108 generates feature point information of the user movement can be obtained.

According to an embodiment, when the electronic device 101-1 or 101-2 is located at a corresponding place after storing feature point information and receives a user input for displaying a 3D virtual object image related to the feature point information, the electronic device 101 -1, 101-2) receives feature point information from the external electronic devices 1410-1 and 1410-2 or the server 108, obtains a 3D virtual object image based on the received feature point information, and video can be displayed. According to an embodiment, when the electronic device 101-1 or 101-2 receives a 3D virtual object image from the external electronic device 1410-1 or 1410-2 or the server 108, the electronic device 101-1 , 101-2) may display the received 3D virtual object image.

According to an embodiment, sensing values obtained by the electronic devices 101-1 and 101-2, user movement information obtained by the external electronic devices 1410-1 and 1410-2 (eg, smartphones), At least one of the images obtained by the external cameras 1411-1 and 1411-2 and the user movement information obtained by the external wearable electronic devices 1412-1 and 1412-2 may be transmitted to the server 108. . According to an embodiment, the server 108 receives the sensing values obtained by the electronic devices 101-1 and 101-2 and the external electronic devices 1410-1 and 1410-2 (eg, smartphones). User movement based on at least one of the obtained user movement information, images obtained by the external cameras 1411-1 and 1411-2, and user movement information obtained by the external wearable electronic devices 1412-1 and 1412-2 Feature point information of can be obtained.

According to an embodiment, when the electronic device 101-1 or 101-2 is located at a corresponding place after storing feature point information and receives a user input for displaying a 3D virtual object image related to the feature point information, the electronic device 101 -1, 101-2) may receive feature point information from the server 108, obtain a 3D virtual object image based on the received feature point information, and display the 3D virtual object image. According to an embodiment, when the electronic device 101-1 or 101-2 receives a 3D virtual object image from the server 108, the electronic device 101-1 or 101-2 transmits the received 3D virtual object image. can be displayed

According to an embodiment, a wearable electronic device (eg, the electronic device 101 of FIG. 1 ) includes a first sensor (eg, the sensor module 176 of FIG. 1 ) detecting a position of the wearable electronic device, the wearable electronic device A second sensor (eg, the sensor module 176 of FIG. 1 ), a display (eg, the display module 160 of FIG. 1 , a memory (eg, the memory 130 of FIG. 1 )) that senses the surroundings of the electronic device, at least one A processor (eg, the processor 120 of FIG. 1 ), wherein the at least one processor obtains location information of a real space where the wearable electronic device is located through the first sensor, and uses the second sensor If object information related to a plurality of real objects included in the real space is acquired through this process, and feature point information of a user's movement related to the location information and the object information is stored in the memory, the wearable electronic device can detect the real object information. When located in the space, a virtual object indicating that motion information of the feature point is present is displayed on the display, and when a user input for displaying an image related to the motion information of the feature point is received through the virtual object, the motion of the feature point is received. A 3D virtual object image may be acquired based on the information, and the 3D virtual object image may be displayed on the display.

According to an embodiment, a third sensor (eg, the sensor module 176 of FIG. 1 ) for detecting motion of the wearable electronic device is further included, and the feature point information is stored before the feature point information is stored in the memory. If the motion information of the wearable electronic device acquired through the third sensor in the real space satisfies a set condition related to at least some of a plurality of real objects included in the real space, the at least one processor Thus, feature point information of the motion of the user wearing the wearable electronic device acquired based on the motion information of the wearable electronic device may be stored in the memory.

According to an embodiment, the feature point information may include motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.

According to an embodiment, the minutiae point information may further include movement path information of the user.

According to an embodiment, a communication circuit (eg, the communication module 190 of FIG. 1 ) is further included, and the feature point information is acquired by further considering motion information of a user acquired from an external electronic device through the communication circuit. may have been

According to an embodiment, the set condition may be set based on at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.

According to an embodiment, the at least one processor determines that the set condition is satisfied when a difference between a motion pattern of the wearable electronic device stored in the memory and motion information of the wearable electronic device is equal to or greater than a set value; The movement pattern of the wearable electronic device may be learned in association with at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.

According to an embodiment, the feature point information stored in the memory may be obtained from a wearable electronic device different from the wearable electronic device.

According to an embodiment, the 3D virtual object may include a character capable of expressing body movements based on the feature point information.

According to an embodiment, the 3D virtual object may further include at least one virtual object respectively related to at least one of the plurality of real objects.

According to an embodiment, a control method of a wearable electronic device includes an operation of acquiring location information of a real space where the wearable electronic device is located through a first sensor that senses a location of the wearable electronic device, and moving around the wearable electronic device. Obtaining object information related to a plurality of real objects included in the real space through a second sensor that senses, and if the location information and feature point information of the user's movement related to the object information are stored in the memory, the When the wearable electronic device is located in the real space, an operation of displaying a virtual object indicating that there is motion information of the feature point on a display, and receiving a user input for displaying an image related to the motion information of the feature point through the virtual object , an operation of obtaining a 3D virtual object image based on the motion information of the feature point and an operation of displaying the 3D virtual object image on the display may be included.

According to an embodiment, the feature point information is motion information of the wearable electronic device obtained through a third sensor for detecting motion of the wearable electronic device in the real space before the feature point information is stored in the memory. If a set condition related to at least some of the plurality of real objects included in the real space is satisfied, the feature point information of the motion of the user wearing the wearable electronic device acquired based on the motion information of the wearable electronic device is It may be stored in memory.

According to an embodiment, the feature point information may include motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.

According to an embodiment, the minutiae point information may further include movement path information of the user.

According to an embodiment, the feature point information may be obtained by further considering motion information of a user obtained from an external electronic device.

According to an embodiment, the set condition may be set based on at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.

According to an embodiment, the method further includes determining that the set condition is satisfied when a difference between a motion pattern of the wearable electronic device stored in the memory and motion information of the wearable electronic device is equal to or greater than a set value. The movement pattern of the wearable electronic device may be learned in association with at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.

According to an embodiment, the feature point information stored in the memory may be obtained from a wearable electronic device different from the wearable electronic device.

According to an embodiment, the 3D virtual object may include a character capable of expressing body movements based on the feature point information.

According to an embodiment, the 3D virtual object may further include at least one virtual object respectively related to at least one of the plurality of real objects.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component among the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

Claims (20)

In a wearable electronic device,
a first sensor for detecting a location of the wearable electronic device;
a second sensor that senses a periphery of the wearable electronic device;
display;
Memory;
at least one processor;
The at least one processor,
Obtaining location information of a real space where the wearable electronic device is located through the first sensor;
obtaining object information related to a plurality of real objects included in the real space through the second sensor;
If the location information and the information on the user's movement related to the object information are stored in the memory, if the wearable electronic device is located in the real space, a virtual object notifying that there is movement information on the feature point is displayed on the display. display,
When a user input for displaying an image related to the motion information of the feature point is received through the virtual object, a 3D virtual object image is obtained based on the motion information of the feature point;
A wearable electronic device displaying the 3D virtual object image on the display.
According to claim 1,
A third sensor for detecting motion of the wearable electronic device; further comprising;
The feature point information,
Before the feature point information is stored in the memory, the motion information of the wearable electronic device acquired through the third sensor in the real space is related to at least some of a plurality of real objects included in the real space. If is satisfied, the feature point information of the motion of the user wearing the wearable electronic device, obtained by the at least one processor based on the motion information of the wearable electronic device, is stored in the memory.
According to claim 2,
The feature point information,
The wearable electronic device comprising motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.
According to claim 3,
The feature point information,
The wearable electronic device further comprising the user's movement path information.
According to claim 2,
further comprising a communication circuit;
The feature point information,
The wearable electronic device obtained by further considering the motion information of the user obtained from the external electronic device through the communication circuit.
According to claim 2,
The set conditions are
The wearable electronic device, which is set based on at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.
According to claim 6,
The at least one processor,
When a difference between a motion pattern of the wearable electronic device stored in the memory and motion information of the wearable electronic device is equal to or greater than a set value, it is determined that the set condition is satisfied;
The movement pattern of the wearable electronic device,
The wearable electronic device is learned in association with at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.
According to claim 1,
The feature point information stored in the memory is
A wearable electronic device obtained from a wearable electronic device different from the wearable electronic device.
According to claim 1,
The 3D virtual object,
A wearable electronic device comprising a character capable of expressing body movements based on the feature point information.
According to claim 9,
The 3D virtual object,
The wearable electronic device further comprises at least one virtual object respectively related to at least one of the plurality of real objects.
In the control method of a wearable electronic device,
obtaining location information of a real space where the wearable electronic device is located through a first sensor that senses the location of the wearable electronic device;
obtaining object information related to a plurality of real objects included in the real space through a second sensor that detects a periphery of the wearable electronic device;
When the location information and the feature point information of the user's motion related to the object information are stored in the memory, when the wearable electronic device is located in the real space, displaying a virtual object indicating that there is movement information of the feature point on a display. movement;
obtaining a 3D virtual object image based on the motion information of the feature point when a user input for displaying an image related to the motion information of the feature point is received through the virtual object; and
A control method of a wearable electronic device comprising: displaying the 3D virtual object image on the display.
According to claim 11,
The feature point information,
Before the feature point information is stored in the memory, the motion information of the wearable electronic device acquired through the third sensor for detecting the motion of the wearable electronic device in the real space is stored in a plurality of real spaces included in the real space. If a set condition related to at least some of the objects is satisfied, control of the wearable electronic device, in which information on the characteristic point of the movement of the user wearing the wearable electronic device obtained based on the motion information of the wearable electronic device is stored in the memory. method.
According to claim 12,
The feature point information,
The method of controlling a wearable electronic device comprising motion information of a plurality of joints of the user and motion information of a bone connecting the plurality of joints.
According to claim 13,
The feature point information,
The control method of the wearable electronic device further comprising the user's movement path information.
According to claim 12,
The feature point information is
A control method of a wearable electronic device obtained by further considering motion information of a user acquired from an external electronic device.
According to claim 12,
The set conditions are
The control method of the wearable electronic device, which is set based on at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.
According to claim 16,
When the difference between the motion pattern of the wearable electronic device stored in the memory and the motion information of the wearable electronic device is equal to or greater than a set value, determining that the set condition is satisfied;
The movement pattern of the wearable electronic device,
The control method of the wearable electronic device, which is learned in association with at least one of location information of the wearable electronic device, object information around the wearable electronic device, and time information.
According to claim 11,
The feature point information stored in the memory is
A control method of a wearable electronic device obtained from a wearable electronic device different from the wearable electronic device.
According to claim 11,
The 3D virtual object,
A control method of a wearable electronic device comprising a character capable of expressing body movements based on the feature point information.
According to claim 19,
The 3D virtual object,
The control method of the wearable electronic device further comprising at least one virtual object respectively related to at least one of the plurality of real objects.

Images