KR20240062862A - Wearable device for executing application based on information obtained by tracking external object and method thereof - Google Patents

Abstract

According to one embodiment, the processor of the wearable device may identify an external object included in the display area of the display using a camera. The processor may display a first visual object representing an application corresponding to the external object based on the location of the external object in the display area. The processor obtains at least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information including at least one stroke drawn by the external object. can do. The processor may display a second visual object for executing the application using information selected from the first information or the second information, based on the location, according to the properties assigned to the external object. there is.

Description

Wearable device and method for executing an application based on information obtained by tracking an external object {WEARABLE DEVICE FOR EXECUTING APPLICATION BASED ON INFORMATION OBTAINED BY TRACKING EXTERNAL OBJECT AND METHOD THEREOF}

This disclosure (present disclosure) relates to a wearable device and method for executing an application based on information obtained by tracking an external object.

To provide an enhanced user experience, we provide augmented reality (AR) services that display computer-generated information in conjunction with external objects in the real-world. Electronic devices are being developed. The electronic device may be a wearable device that can be worn by a user. For example, the electronic device may be AR glasses and/or a head-mounted device (HMD).

According to an embodiment, a wearable device may include a display, a camera, and a processor. The processor may be configured to identify an external object included in a display area of the display using the camera. The processor may be configured to display a first visual object representing an application corresponding to the external object based on the location of the external object in the display area. The processor obtains at least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information including at least one stroke drawn by the external object. It can be configured to do so. The processor displays a second visual object for executing the application using information selected from the first information or the second information, based on the location, according to the attribute assigned to the external object. It can be configured.

According to one embodiment, a method of using a wearable device may include identifying an external object included in a display area of a display of the wearable device using a camera of the wearable device. The method may include displaying a first visual object representing an application corresponding to the external object based on the location of the external object in the display area. The method obtains at least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information comprising at least one stroke drawn by the external object. It may include actions such as: The method includes displaying a second visual object for executing the application using information selected from the first information or the second information, based on the location, according to the attribute assigned to the external object. may include.

According to one embodiment, a wearable device may include a display, memory, and a processor. The processor may be configured to display, within the display, a list of a plurality of applications stored in the memory. The processor executes the first application using the external object based on an input indicating that a visual object representing the first application included in the list is overlapped on an external object included in the display area. It can be configured to obtain properties for: The processor, after acquiring the attribute, obtains at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object, It can be configured. The processor may be configured to execute the first application using information selected by the attribute from among the first information or the second information.

According to one embodiment, a method of using a wearable device may include displaying a list of a plurality of applications stored in a memory of the wearable device on a display of the wearable device. The method is based on an input indicating that a visual object representing the first application included in the list is superimposed on an external object included in a display area of the display, using the external object to display the first application. It may include an operation to obtain properties for executing. The method includes, after obtaining the attribute, acquiring at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object. may include. The method may include executing the first application using information selected by the attribute from among the first information or the second information.

FIG. 1 illustrates an example of an operation in which a wearable device obtains information related to an external object, according to an embodiment.
Figure 2 shows an example of a flow chart for a wearable device, according to one embodiment.
Figure 3A shows an example of a perspective view of a wearable device, according to one embodiment.
FIG. 3B shows an example of one or more hardware deployed within a wearable device, according to one embodiment.
4A to 4B show an example of the appearance of a wearable device, according to an embodiment.
Figure 5 is a block diagram of a wearable device, according to one embodiment.
FIGS. 6A to 6D illustrate an example of an operation for identifying an input indicating that a wearable device registers an external object, according to an embodiment.
Figure 7 shows an example of a flow chart for a wearable device, according to one embodiment.
FIGS. 8A and 8B illustrate an example of a user interface (UI) displayed by a wearable device based on an external object, according to an embodiment.
9A to 9C illustrate an example of an operation in which a wearable device executes an application based on information obtained by tracking an external object, according to an embodiment.
FIG. 10 illustrates an example of an operation in which a wearable device switches a camera for tracking an external object, according to an embodiment.
FIG. 11 illustrates an example of a UI that a wearable device displays based on information obtained by tracking an external object, according to an embodiment.
FIGS. 12A and 12B illustrate an example of an operation in which a wearable device executes an application based on information obtained by tracking an external object, according to an embodiment.
FIG. 13 illustrates an example of an operation in which a wearable device executes an application based on information obtained by tracking an external object, according to an embodiment.
FIG. 14 illustrates an example of an operation in which a wearable device executes an application based on information obtained by tracking an external object, according to an embodiment.
FIGS. 15A and 15B illustrate an example of an operation in which a wearable device executes a plurality of applications based on information obtained by tracking an external object, according to an embodiment.
Figure 16 shows an example of a flow chart for a wearable device, according to one embodiment.
FIG. 17 illustrates an example of an operation in which a wearable device displays a UI for recommending at least one application based on information obtained by tracking an external object, according to an embodiment.
Figure 18 shows an example of a flow chart for a wearable device, according to one embodiment.
FIG. 19 illustrates an example of an operation in which a wearable device executes an application based on information acquired by an external electronic device, according to an embodiment.

Hereinafter, various embodiments of this document are described with reference to the attached drawings.

The various embodiments of this document and the terms used herein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various changes, equivalents, and/or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar components. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as "first", "second", "first" or "second" can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another. The components are not limited. When a component (e.g., a first) component is said to be "connected (functionally or communicatively)" or "connected" to another (e.g., second) component, it means that the component is connected to the other component. It may be connected directly to a component or may be connected through another component (e.g., a third component).

The term “module” used in this document includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

FIG. 1 illustrates an example of an operation in which a wearable device 101 acquires information related to an external object 130, according to an embodiment. Referring to FIG. 1 , a wearable device 101 according to an embodiment may include a head-mounted display (HMD) that is wearable on the head of the user 110. Although the outer appearance of the wearable device 101 in the form of glasses is shown, the embodiment is not limited thereto. An example of the structure of the wearable device 101 that can be worn on the head of the user 110 is described with reference to FIGS. 3A to 3B and/or 4A to 4B. One or more pieces of hardware included in the wearable device 101 are exemplarily described with reference to FIG. 5 .

According to one embodiment, the wearable device 101 may execute functions related to augmented reality (AR) and/or mixed reality (MR). Referring to FIG. 1 , when the user 110 is wearing the wearable device 101, the wearable device 101 may include at least one lens disposed adjacent to the eye of the user 110. The wearable device 101 may combine light emitted from the display of the wearable device 101 with ambient light passing through the lens. The display area of the display may be formed within a lens through which ambient light passes. Because the wearable device 101 combines the ambient light and the light emitted from the display, the user 110 can see a real object perceived by the ambient light and the light emitted from the display. You can see an image in which the formed virtual objects are mixed.

According to one embodiment, the wearable device 101 may execute functions related to video see-through (VST) and/or virtual reality (VR). Referring to FIG. 1 , when the user 110 is wearing the wearable device 101, the wearable device 101 may include a housing that covers the eyes of the user 110. In this state, the wearable device 101 may include a display disposed on a first side facing the eye. The wearable device 101 may include a camera disposed on a second side opposite to the first side. Using the camera, the wearable device 101 can obtain frames containing ambient light. The wearable device 101 may output the frames in a display disposed on the first side, allowing the user 110 to recognize the ambient light through the display. The display area of the display disposed on the first side may be formed by one or more pixels included in the display. The wearable device 101 may synthesize a virtual object within frames output through the display, allowing the user 110 to recognize the virtual object along with a real object recognized by ambient light.

Referring to FIG. 1 , an example in which the wearable device 101, according to one embodiment, identifies an external object 130 while worn by the user 110 is shown. In the example of FIG. 1 , the wearable device 101 may enable the user 110 to recognize an external space based on AR, MR, or VST. Within the external space, the wearable device 101 may identify an external object 130. Although the external object 130 having the shape of a pen is shown as an example, the embodiment is not limited thereto. The external object 130 may include an object that can be registered in the wearable device 101. For example, the external object 130 may have a shape extending in the longitudinal direction, such as a pen and/or a stick such as a tree branch. The embodiment is not limited to this, and the external object 130 may have the shape of a polyhedron or a curved body. An example of an operation performed by the wearable device 101 to register an external object 130 according to an embodiment is described with reference to FIGS. 6A to 6D and/or FIG. 7 .

Referring to FIG. 1 , an image shown by a user 110 wearing a wearable device 101 is shown. According to one embodiment, the wearable device 101 may identify an external object 130 mapped to a function of an application. The wearable device 101 may display a visual object 160 based on the location of the external object 130 within the display area of the display. The visual object 160 may have the form of an application icon mapped to the external object 130. The shape, size, and/or location of the visual object 160 displayed by the wearable device 101 is not limited to the embodiment of FIG. 1 .

According to one embodiment, the wearable device 101 may obtain information related to the motion of the external object 130. Referring to FIG. 1 , an exemplary state in which an external object 130 is moved by a hand 140 is shown. For example, using the camera (and/or depth sensor) of the wearable device 101 disposed toward the external object 130, the wearable device 101 provides first information related to the path of the external object 130. It can be obtained. The first information may include a path of an external object 130 contacted on a plane 150 (eg, paper and/or table). If the external object 130 is an object capable of creating a trace (e.g., at least one stroke) on the plane 150, such as a pen and/or pencil, the wearable device 101 may Second information including at least one stroke drawn by the external object 130 can be obtained using the camera. The wearable device 101 may perform optical character recognition (OCR) on the at least one stroke to obtain content (eg, text and/or image) generated by the motion of the external object 130.

According to one embodiment, the wearable device 101 is based on first information related to the path of the external object 130 and/or second information including at least one stroke drawn by the external object 130, An application corresponding to the object 130 may be executed. For example, the wearable device 101 may execute an application registered to the external object 130 and execute functions related to the first information and/or the second information. The wearable device 101 may execute the application in response to an input indicating selection of a visual object 160 including an icon representing the application. The operation of the wearable device 101 executing the application registered to the external object 130 is described with reference to FIGS. 8A to 8B, 9A to 9C, and 10 to 16. Depending on the application registered with the external object 130, the wearable device 101 may execute various functions based on the stroke drawn by the external object 130 and/or the path of the external object 130.

Referring to FIG. 1, according to one embodiment, the wearable device 101 may be connected to an external electronic device such as a terminal 120 and/or a server 122. The terminal 120 may include, for example, a personal computer (PC) such as a laptop and desktop, a smartphone, a smartpad, and/or a tablet PC. The server 122 may support a function (eg, OCR) for analyzing the path of the external object 130 and/or the stroke drawn by the external object 130. The wearable device 101, terminal 120, and/or server 122 may be connected to each other through a wired network and/or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network is a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, bluetooth low-energy (BLE), or a combination thereof. may include. Although the wearable device 101, terminal 120, and server 122 are shown as being directly connected, the wearable device 101, terminal 120, and server !22 are intermediate nodes. ) (e.g., one or more routers and/or AP (access point)).

As described above, the wearable device 101 according to one embodiment may track the path of the external object 130 and/or at least one stroke drawn by the external object 130. Based on the tracked path and/or stroke, the wearable device 101 may execute at least one application corresponding to the external object 130. At least one application matched to the external object 130 may be set by an attribute assigned to the external object 130. The wearable device 101 may execute the at least one application based on either the path or the stroke selected by the attribute. As the at least one application is executed, the wearable device 101 may support the user 110 controlling the wearable device 101 using the external object 130. Although external object 130 does not include circuitry for communicating with wearable device 101, such as a pen and/or pencil, wearable device 101 may receive information indicated by external object 130. You can visualize it.

Below, with reference to FIG. 2 , an example of the operation of the wearable device 101 according to an embodiment is described.

Figure 2 shows an example of a flow chart for a wearable device, according to one embodiment. The wearable device of FIG. 2 may include the wearable device 101 of FIG. 1 .

Referring to FIG. 2, within operation 210, according to one embodiment, the wearable device may identify an external object and a function mapped to the external object. For example, the wearable device may identify an external object seen by a user wearing the wearable device (eg, the user 110 of FIG. 1 ), such as the external object 130 of FIG. 1 . In order to identify external objects seen by a user wearing the wearable device, the wearable device may include at least one camera disposed facing a specified direction. Using the camera, a wearable device can identify external objects. If the external object is an external object registered to the wearable device, the wearable device can identify the function mapped to the external object. That an external object is registered in a wearable device may mean that properties assigned to the external object are stored in the wearable device. Based on the attributes, the wearable device can identify the function.

Referring to Figure 2, within operation 220, according to one embodiment, the wearable device determines the motion of the external object, or at least one stroke generated by the external object, based on whether the external object has been identified. You can obtain information about The motion of the external object may include the motion of the external object contacted on a plane (eg, plane 150 of FIG. 1) of external space including the external object. The at least one stroke may be created as a material (eg, ink and/or graphite) contained in an external object is adhered to the plane. The wearable device may selectively track either the motion or the at least one stroke based on an attribute assigned to an external object. To obtain the information, the wearable device may communicate with an external electronic device (eg, terminal 120 and/or server 122 of FIG. 1).

In one embodiment, in order to obtain information about the motion of an external object, a wearable device may detect a portion of the external object (e.g., an end, and/or a surface) of the external object. You can trace the route. Based on whether the portion of the external object is in contact with a plane, the wearable device may obtain first information related to the path of the external object in contact with the plane. In the state of acquiring the first information, the wearable device may display one or more lines having the shape of the path on the display of the wearable device. The portion of the external object tracked by the wearable device may be set by attributes assigned to the external object.

In one embodiment, in order to obtain information about at least one stroke drawn by an external object, the wearable device may control a camera disposed toward the external object, among different cameras. The wearable device can use the camera to identify at least one stroke drawn by an external object. The wearable device can identify handwriting drawn by an external object based on the letters indicated by the at least one stroke. The embodiment is not limited to this, and the wearable device can identify at least one figure indicated by the at least one stroke. The wearable device may communicate with an external electronic device to obtain text data (e.g., data encoded by a binary code such as Unicode) corresponding to the handwriting.

Referring to FIG. 2, within operation 230, a wearable device according to one embodiment may execute a function mapped to an external object based on acquired information. The function may be identified by an attribute assigned by an external object. The above function may be supported by at least one application installed on the wearable device. According to one embodiment, the wearable device can visualize the results of executing the above function to the user. The wearable device may execute the above function based on an input indicating selection of a visual object displayed in the display (eg, visual object 160 in FIG. 1). The visual object may be identified by the attribute. The wearable device may display the visual object based on the location of the external object within the display.

As described above, according to one embodiment, the wearable device may register an object that does not support the function of communicating with the wearable device and support control of the wearable device based on the registered object. The wearable device recognizes the motion of an external object, such as a pen, or recognizes at least one stroke drawn by the external object, and obtains electrical information (e.g., digital content) from non-electrical information about the external object. You can. Based on the electrical information, the wearable device can recognize an external object that cannot be directly connected to the wearable device and obtain digital content from the recognized external object.

Below, with reference to FIGS. 3A to 3B and/or 4A to 4B, an exemplary structure of a wearable device for recognizing an external object according to an embodiment is described.

Figure 3A shows an example of a perspective view of a wearable device, according to one embodiment. The wearable device 300 of FIGS. 3A and 3B may be an example of the wearable device 101 of FIG. 1 and/or the wearable device of FIG. 2 . 3A (as shown in), according to one embodiment, the wearable device 300 may include at least one display 350 and a frame supporting the at least one display 350. .

According to one embodiment, the wearable device 300 may be worn on a part of the user's body. The wearable device 300 provides the user wearing the wearable device 300 with augmented reality (AR), virtual reality (VR), or a mixed reality that combines augmented reality and virtual reality. , MR) can be provided. For example, the wearable device 300 outputs a virtual reality image to the user through at least one display 350 in response to the user's specified gesture acquired through the motion recognition camera 340-2 of FIG. 3B. can do.

According to one embodiment, at least one display 350 in the wearable device 300 may provide visual information to the user. For example, at least one display 350 may include a transparent or translucent lens. At least one display 350 may include a first display 350-1 and/or a second display 350-2 spaced apart from the first display 350-1. For example, the first display 350-1 and the second display 350-2 may be placed at positions corresponding to the user's left eye and right eye, respectively.

Referring to FIG. 3B, at least one display 350 forms a display area on the lens to display visual information included in external light passing through the lens to a user wearing the wearable device 300. It can provide other visual information that is distinct from visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The display area formed by at least one display 350 may be formed on the first surface 331 and the second surface 332 of the lens. When a user wears the wearable device 300, external light may be incident on the first surface 331 and transmitted through the second surface 332, thereby being transmitted to the user. As another example, at least one display 350 may display a virtual reality image to be combined with a real screen transmitted through external light. The virtual reality image output from at least one display 350 includes one or more hardware (e.g., optical devices 382 and 384, and/or at least one waveguides) included in the wearable device 300. )(333, 334)), it can be transmitted to the user's eyes.

According to one embodiment, the wearable device 300 includes waveguides 333 that diffract light transmitted from at least one display 350 and relayed by the optical devices 382 and 384 and deliver it to the user. , 334). The waveguides 333 and 334 may be formed based on at least one of glass, plastic, or polymer. A nanopattern may be formed on the outside or at least a portion of the inside of the waveguides 333 and 334. The nanopattern may be formed based on a polygonal and/or curved grating structure. Light incident on one end of the waveguides 333 and 334 may propagate to the other end of the waveguides 333 and 334 by the nanopattern. The waveguides 333 and 334 may include at least one of a diffractive element (eg, a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (eg, a reflective mirror). For example, the waveguides 333 and 334 may be disposed within the wearable device 300 to guide the screen displayed by at least one display 350 to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) generated within the waveguides 333 and 334.

According to one embodiment, the wearable device 300 analyzes objects included in the real-life image collected through the shooting camera 340-1, and selects an object that is the target of providing augmented reality from among the analyzed objects. Corresponding virtual objects can be combined and displayed on at least one display 350. The virtual object may include at least one of text and images for various information related to the object included in the real image. The wearable device 300 can analyze objects based on multi-cameras, such as stereo cameras. For the object analysis, the wearable device 300 may execute time-of-flight (ToF) and/or simultaneous localization and mapping (SLAM) supported by multi-cameras. A user wearing the wearable device 300 can watch images displayed on at least one display 350.

According to one embodiment, the frame may be made of a physical structure that allows the wearable device 300 to be worn on the user's body. According to one embodiment, the frame allows the first display 350-1 and the second display 350-2 to be positioned to correspond to the user's left eye and right eye when the user wears the wearable device 300. It can be configured so that The frame may support at least one display 350. For example, the frame may support the first display 350-1 and the second display 350-2 to be positioned at positions corresponding to the user's left and right eyes.

Referring to FIG. 3A according to one embodiment, when the user wears the wearable device 300, the frame may include an area 320 at least partially in contact with a portion of the user's body. For example, the area 320 of the frame in contact with a part of the user's body includes an area in contact with a part of the user's nose, a part of the user's ear, and a side part of the user's face that the wearable device 300 touches. can do. According to one embodiment, the frame may include a nose pad 310 that contacts a part of the user's body. When the wearable device 300 is worn by a user, the nose pad 310 may be in contact with a portion of the user's nose. The frame may include a first temple 304 and a second temple 305 that are in contact with another part of the user's body that is distinct from the part of the user's body.

In one embodiment, the frame includes a first rim 301 surrounding at least a portion of the first display 350-1 and a second rim 302 surrounding at least a portion of the second display 350-2. , a bridge 303 disposed between the first rim 301 and the second rim 302, a first pad disposed along a portion of the edge of the first rim 301 from one end of the bridge 303 ( 311), a second pad 312 disposed along a portion of the edge of the second rim 302 from the other end of the bridge 303, a first temple extending from the first rim 301 and fixed to a portion of the wearer's ear ( 304), and a second temple 305 extending from the second rim 302 and fixed to a portion of the ear opposite the ear. The first pad 311 and the second pad 312 may be in contact with a portion of the user's nose, and the first temple 304 and the second temple 305 may be in contact with a portion of the user's face and a portion of the ear. may come into contact with. The temples 304 and 305 may be rotatably connected to the rim via the hinge units 306 and 307 of FIG. 3B. The first temple 304 may be rotatably connected to the first rim 301 through a first hinge unit 306 disposed between the first rim 301 and the first temple 304. . The second temple 305 may be rotatably connected to the second rim 302 through a second hinge unit 307 disposed between the second rim 302 and the second temple 305. According to one embodiment, the wearable device 300 detects an external object (e.g., a user's fingertip) touching the frame using a touch sensor, a grip sensor, and/or a proximity sensor formed on at least a portion of the surface of the frame. fingertip)), and/or a gesture performed by the external object may be identified.

According to one embodiment, the wearable device 300 may include hardware (eg, hardware described above based on the block diagram of FIG. 6) that performs various functions. For example, the hardware includes a battery module 370, an antenna module 375, optical devices 382, 384, speakers 392-1, 392-2, and microphones 394-1, 394- 2, 394-3), a light emitting module (not shown), and/or a printed circuit board 390. Various hardware can be placed within the frame.

According to one embodiment, the microphones 394-1, 394-2, and 394-3 of the wearable device 300 may be disposed in at least a portion of the frame to obtain a sound signal. A first microphone 394-1 disposed on the nose pad 310, a second microphone 394-2 disposed on the second rim 302, and a third microphone disposed on the first rim 301. Although microphones 394-3 are shown in FIG. 3B, the number and placement of microphones 394 are not limited to the embodiment of FIG. 3B. If the number of microphones 394 included in the wearable device 300 is two or more, the wearable device 300 can identify the direction of the sound signal using a plurality of microphones disposed on different parts of the frame. there is.

According to one embodiment, the optical devices 382 and 384 may transmit a virtual object transmitted from at least one display 350 to the waveguides 333 and 334. For example, optical devices 382 and 384 may be projectors. The optical devices 382 and 384 may be disposed adjacent to the at least one display 350 or may be included within the at least one display 350 as part of the at least one display 350 . The first optical device 382 may correspond to the first display 350-1, and the second optical device 384 may correspond to the second display 350-2. The first optical device 382 can transmit the light output from the first display 350-1 to the first waveguide 333, and the second optical device 384 can transmit the light output from the first display 350-1 to the first waveguide 333. The light output from -2) can be transmitted to the second waveguide 334.

In one embodiment, the camera 340 may include an eye tracking camera (ET CAM) 340-1, a motion recognition camera 340-2, and/or an imaging camera 340-3. You can. The shooting camera 340-3, the eye tracking camera 340-1, and the motion recognition camera 340-2 may be placed at different positions on the frame and perform different functions. The gaze tracking camera 340-1 may output data representing the gaze of the user wearing the wearable device 300. For example, the wearable device 300 may detect the gaze from an image including the user's pupils obtained through the gaze tracking camera 340-1. An example in which the gaze tracking camera 340-1 is positioned toward the user's right eye is shown in FIG. 3B, but the embodiment is not limited thereto, and the gaze tracking camera 340-1 is positioned solely toward the user's left eye. It may be placed towards, or towards both eyes.

In one embodiment, the capturing camera 340-3 may capture a real image or background to be matched with a virtual image to implement augmented reality or mixed reality content. The capturing camera may capture an image of a specific object that exists at a location where the user is looking and provide the image to at least one display 350. At least one display 350 is one in which a real image or background information including the image of the specific object obtained using a photographing camera and a virtual image provided through the optical devices 382 and 384 are overlapped. The video can be displayed. In one embodiment, the imaging camera may be placed on the bridge 303 disposed between the first rim 301 and the second rim 302.

In one embodiment, the gaze tracking camera 340-1 tracks the gaze of a user wearing the wearable device 300, thereby tracking the user's gaze and visual information provided to at least one display 350. By matching them, a more realistic augmented reality can be realized. For example, when the user looks forward, the wearable device 300 may naturally display environmental information related to the user's front view on at least one display 350 at the location where the user is located. The gaze tracking camera 340-1 may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera 340-1 may receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera 340-1 may be placed at positions corresponding to the user's left and right eyes. For example, the eye tracking camera 340-1 may be placed within the first rim 301 and/or the second rim 302 to face the direction in which the user wearing the wearable device 300 is located. You can.

In one embodiment, the motion recognition camera 340-2 recognizes the movement of the entire or part of the user's body, such as the user's torso, hands, or face, and displays a specific event on the screen provided on at least one display 350. can be provided. The gesture recognition camera 340-2 may recognize a user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display 350. The processor may perform a designated function based on identifying a signal corresponding to the operation. In one embodiment, the motion recognition camera 340-2 may be disposed on the first rim 301 and/or the second rim 302.

In one embodiment, the camera 340 included in the wearable device 300 is not limited to the eye tracking camera 340-1 and the motion recognition camera 340-2 described above. For example, the wearable device 300 may identify an external object included within the user's FoV using the capturing camera 340-3 disposed toward the user's FoV. The wearable device 300 identifies an external object based on a sensor for identifying the distance between the wearable device 300 and the external object, such as a depth sensor and/or a time of flight (ToF) sensor. It can be. The camera 340 disposed toward the FoV may support an autofocus function and/or an optical image stabilization (OIS) function. For example, the wearable device 300 includes a camera 340 (e.g., a face tracking (FT) camera) disposed toward the face of a user wearing the wearable device 300 to obtain an image including the face. ) may include.

Although not shown, according to one embodiment, the wearable device 300 radiates light toward a subject (e.g., the user's eyes, face, and/or an external object within the FoV) captured using the camera 340. It may further include a light source (eg, LED). The light source may include an LED with an infrared wavelength. The light source may be placed in at least one of the frame and hinge units 306 and 307.

According to one embodiment, the battery module 370 may supply power to electronic components of the wearable device 300. In one embodiment, the battery module 370 may be disposed within the first temple 304 and/or the second temple 305. For example, the battery module 370 may be a plurality of battery modules 370 . A plurality of battery modules 370 may be disposed on the first temple 304 and the second temple 305, respectively. In one embodiment, the battery module 370 may be disposed at an end of the first temple 304 and/or the second temple 305.

In one embodiment, the antenna module 375 may transmit a signal or power to the outside of the wearable device 300, or may receive a signal or power from the outside. The antenna module 375 may be electrically and/or operationally connected to a communication circuit (e.g., a communication circuit 650 described later with reference to FIG. 6) within the wearable device 300. In one embodiment, the antenna module 375 may be disposed within the first temple 304 and/or the second temple 305. For example, the antenna module 375 may be placed close to one surface of the first temple 304 and/or the second temple 305.

In one embodiment, the speakers 392-1 and 392-2 may output sound signals to the outside of the wearable device 300. The sound output module may be referred to as a speaker. In one embodiment, the speakers 392-1 and 392-2 are disposed adjacent to the ears of the user wearing the wearable device 300, the first temple 304, and/or the second temple 305. ) can be placed within. For example, the wearable device 300 is disposed within the first temple 304 and the second speaker 392-2, which is disposed adjacent to the user's left ear, and is disposed within the second temple 305 to listen to the user's hearing. It may include a first speaker 392-1, which is disposed adjacent to the right ear.

In one embodiment, a light emitting module (not shown) may include at least one light emitting device. In order to visually provide information about the specific state of the wearable device 300 to the user, the light emitting module may emit light in a color corresponding to a specific state or emit light in an operation corresponding to the specific state. For example, when the wearable device 300 requires charging, it may repeatedly emit red light at designated times. In one embodiment, the light emitting module may be disposed on the first rim 301 and/or the second rim 302.

Referring to FIG. 3B , the wearable device 300 according to one embodiment may include a printed circuit board (PCB) 390. The PCB 390 may be included in at least one of the first temple 304 or the second temple 305. The PCB 390 may include an interposer disposed between at least two sub-PCBs. On the PCB 390, one or more hardware included in the wearable device 300 (eg, hardware shown by blocks described later with reference to FIG. 5) may be disposed. The wearable device 300 may include a flexible PCB (FPCB) for interconnecting the hardware.

According to one embodiment, the wearable device 300 includes a gyro sensor for detecting the posture of the wearable device 300 and/or the posture of a body part (e.g., head) of a user wearing the wearable device 300, It may include at least one of a gravity sensor and/or an acceleration sensor. The wearable device 300 can measure gravitational acceleration and acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other using a gravity sensor and an acceleration sensor, respectively. . A gyro sensor can measure the angular velocity of each of designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device 300 may identify a user's motion and/or gesture performed to execute or stop a specific function of the wearable device 300 based on the IMU.

4A to 4B show an example of the appearance of a wearable device 400, according to an embodiment. The wearable device 400 of FIGS. 4A and 4B may be an example of the wearable device 101 of FIG. 1 and/or the wearable device of FIG. 2 . An example of the appearance of the first side 410 of the housing of the wearable device 400, according to one embodiment, is shown in FIG. 4A, and the second side 420 is opposite to the first side 410. ) can be shown in Figure 4b.

Referring to FIG. 4A, according to one embodiment, the first surface 410 of the wearable device 400 may have a form attachable to a user's body part (e.g., the user's face). . Although not shown, the wearable device 400 may include a strap for securing to a portion of the user's body, and/or one or more temples (e.g., the first temple 304 of FIGS. 3A-3B , and/or the first temple 304 of FIGS. 3A-3B It may further include 2 temples (305). A first display 350-1 for outputting an image to the left eye of both eyes of the user, and a second display 350-2 for outputting an image to the right eye of the user's both eyes, have a first surface 410 It can be placed on top. The wearable device 400 is formed on the first surface 410 and emits light (e.g., external light (e.g., external light) different from the light emitted from the first display 350-1 and the second display 350-2. Rubber or silicone packing may be further included to prevent interference due to ambient light.

According to one embodiment, the wearable device 400 includes a camera for photographing and/or tracking both eyes of a user adjacent to each of the first display 350-1 and the second display 350-2. It may include (440-1, 440-2). The cameras 440-1 and 440-2 may be referred to as ET cameras. According to one embodiment, the wearable device 400 may include cameras 440-3 and 440-4 for photographing and/or recognizing the user's face. The cameras 440-3 and 440-4 may be referred to as FT cameras.

Referring to FIG. 4B, on the second side 420, which is opposite to the first side 410 of FIG. 4A, a camera (e.g., cameras 440) for acquiring information related to the external environment of the wearable device 400 -5, 440-6, 440-7, 440-8, 440-9, 440-10)), and/or a sensor (e.g., depth sensor 430) may be disposed. For example, the cameras 440-5, 440-6, 440-7, 440-8, 440-9, and 440-10 are configured to recognize external objects different from the wearable device 400. It may be placed on (420). For example, using the cameras 440-9 and 440-10, the wearable device 400 may obtain images and/or media to be transmitted to each of the user's eyes. The camera 440-9 will be disposed on the second side 420 of the wearable device 400 to acquire an image to be displayed through the second display 350-2 corresponding to the right eye among the two eyes. You can. The camera 440-10 may be disposed on the second side 420 of the wearable device 400 to acquire an image to be displayed through the first display 350-1 corresponding to the left eye among the two eyes. there is.

According to one embodiment, the wearable device 400 may include a depth sensor 430 disposed on the second surface 420 to identify the distance between the wearable device 400 and an external object. Using the depth sensor 430, the wearable device 400 acquires spatial information (e.g., depth map) about at least a portion of the FoV of the user wearing the wearable device 400. can do.

Although not shown, a microphone for acquiring sound output from an external object may be disposed on the second side 420 of the wearable device 400. The number of microphones may be one or more depending on the embodiment.

As described above, the wearable device 400 according to one embodiment may have a form factor to be worn on the user's head. The wearable device 400 may provide a user experience based on augmented reality, virtual reality, and/or mixed reality while worn on the head. The wearable device 400 can display a screen including an external object using the first display 350-1 and the second display 350-2 that are arranged to face each of the user's eyes. Within the screen, the wearable device 400 may register an external object or display the results of executing a function assigned to the external object.

Hereinafter, with reference to FIG. 5, a wearable device including the wearable device 300 of FIGS. 3A to 3B and/or the wearable device 400 of FIGS. 4A to 4B (e.g., the wearable device 101 of FIG. 1 )) An exemplary structure of one or more hardware included within is described.

Figure 5 is a block diagram of a wearable device 101, according to one embodiment. The wearable device 101 of FIG. 5 includes the wearable device 101 of FIG. 1, the wearable device of FIG. 2, the wearable device 300 of FIGS. 3A and 3B, and/or the wearable device 400 of FIGS. 4A and 4B. It can be included. The wearable device 101 may include at least one of a processor 510, memory 520, display 530, camera 540, or communication circuit 550. The processor 510, memory 520, display 530, camera 540, and communication circuit 550 are electrically and/or operationally connected to each other by electronic components such as a communication bus 502. Can be connected (electronically and/or operably coupled with each other). Hereinafter, hardware being operatively combined will mean that a direct connection or an indirect connection between the hardware is established, wired or wireless, such that the second hardware is controlled by the first hardware among the hardware. You can. Although shown based on different blocks, the embodiment is not limited thereto, and some of the hardware in FIG. 5 (e.g., the processor 510, the memory 520, and at least a portion of the communication circuit 550) are SoC (system It may be included in a single integrated circuit, such as on a chip. The type and/or number of hardware components included in the wearable device 101 are not limited to those shown in FIG. 5 . For example, wearable device 101 may include only some of the hardware components shown in FIG. 5 .

The processor 510 of the wearable device 101 according to an embodiment may include hardware components for processing data based on one or more instructions. Hardware components for processing data include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP). may include. The number of processors 510 may be one or more. For example, the processor 510 may have a multi-core processor structure such as dual core, quad core, or hexa core.

According to one embodiment, the memory 520 of the wearable device 101 may include a hardware component for storing data and/or instructions input and/or output to the processor 510. Memory 520 may include, for example, volatile memory such as random-access memory (RAM) and/or non-volatile memory such as read-only memory (ROM). there is. Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory includes, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid state drive (SSD), and embedded multi media card (eMMC). ) may include at least one of

According to one embodiment, within the memory 520 of the wearable device 101, one or more instructions (or commands) indicating operations and/or operations to be performed on data by the processor 510 may be stored. . A set of one or more instructions may be referred to as firmware, operating system, process, routine, sub-routine and/or application. For example, the wearable device 101 and/or the processor 510 executes a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, and/or application. , at least one of the operations of FIGS. 2, 7, 16, and/or 18 may be performed. Hereinafter, the fact that an application is installed on the wearable device 101 means that one or more instructions provided in the form of an application are stored in the memory 520 of the wearable device 101, and the one or more applications are installed in the wearable device 101. This may mean stored in an executable format (eg, a file with an extension specified by the operating system of the wearable device 101) by the processor 510. For example, the wearable device 101 executes one or more instructions to generate information related to the motion of an external object (e.g., the external object 130 of FIG. 1) (e.g., a path of a portion of the external object and/or the external object At least one stroke drawn by) can be obtained. The wearable device 101 may execute one or more applications installed in the memory 520 based on the acquired information. The wearable device 101 may execute the one or more applications to execute a function matched to the external object based on the obtained information.

According to one embodiment, the display 530 of the wearable device 101 may output visualized information to the user. The display 530 of FIG. 5 may include at least one display 350 of FIGS. 3A to 3B and/or the first display 350-1 to the second display 350-2 of FIGS. 4A to 4B. You can. The display 530 is controlled by a controller such as a GPU (graphic processing unit) and can output visualized information to the user. The display 530 may include a flat panel display (FPD) and/or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and/or one or more light emitting diodes (LED). The LED may include an organic LED (OLED).

According to one embodiment, the camera 540 of the wearable device 101 includes one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a CMOS (CMOS) sensor that generates an electrical signal representing the color and/or brightness of light. It may include a complementary metal oxide semiconductor (sensor). A plurality of optical sensors included in the camera 540 may be arranged in the form of a 2-dimensional array. The camera 540 may acquire electrical signals from a plurality of optical sensors substantially simultaneously and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional grid. For example, photo data captured using the camera 540 may mean one two-dimensional frame data obtained from the camera 540. For example, video data captured using the camera 540 may mean a sequence of a plurality of images acquired from the camera 540 according to a designated frame rate.

According to one embodiment, the wearable device 101 may include one or more cameras (eg, camera 540). When the wearable device 101 includes a plurality of cameras, the plurality of cameras can be distinguished by information to be obtained through the camera. For example, a camera for tracking the eyes of a user wearing the wearable device 101 may be referred to as an ET camera (eg, ET camera 340-1 in FIGS. 3A and 3B). For example, a camera for tracking the face of a user wearing the wearable device 101 may be referred to as an FT camera. For example, the wearable device 101 has a camera for tracking the user's hand (e.g., hand 140 in FIG. 1) and is referred to as a motion recognition camera (e.g., motion recognition camera 340-2 in FIG. 3B). It can be. Each of the plurality of cameras is distinguished by a field-of-view (FoV) formed by the direction the camera faces and/or the view angle while the user is wearing the wearable device 101. It can be. The external object is included within the FoV of the camera 540 and may include an object that is distinct from the wearable device 101.

According to one embodiment, the communication circuit 550 of the wearable device 101 transmits electrical signals between the wearable device 101 and an external electronic device (e.g., the terminal 120 and/or the server 122 of FIG. 1). May include hardware to support transmission and/or reception. The communication circuit 550 may include, for example, at least one of a modem (MODEM), an antenna, and an optical/electronic (O/E) converter. The communication circuit 550 includes Ethernet, local area network (LAN), wide area network (WAN), wireless fidelity (WiFi), Bluetooth, bluetooth low energy (BLE), ZigBee, long term evolution (LTE), It can support transmission and/or reception of electrical signals based on various types of protocols, such as 5G new radio (NR).

Although not shown, the wearable device 101 according to one embodiment may include an output means for outputting information in a form other than a visualized form. For example, the wearable device 101 may include a speaker for outputting an acoustic signal. For example, the wearable device 101 may include a motor to provide haptic feedback based on vibration.

According to one embodiment, the wearable device 101 may identify an external object using the camera 540. Attributes assigned to the external object may be stored in the memory 520 of the wearable device 101. Based on the properties, the wearable device 101 may track at least one of the path of the external object and/or at least one stroke drawn by the external object. Based on the tracked path and/or at least one stroke, the wearable device 101 may execute an application corresponding to the attribute. The wearable device 101 may execute the application and obtain digital content from the traced path and/or analog content indicated by at least one stroke. The embodiment is not limited to this, and the wearable device 101 may process information included in the tracked path and/or at least one stroke based on the application.

In one embodiment, the wearable device 101 may receive input for registering an identified external object using the camera 540. Below, with reference to FIGS. 6A to 6D and/or 7 , an example of an operation in which the wearable device 101 receives the input according to an embodiment is described.

FIGS. 6A to 6D illustrate an example of an operation of identifying an input indicating that the wearable device 101 registers an external object, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIGS. 6A to 6D. 6A to 6D, different states in which the wearable device 101 displays a user interface (UI) through the display area of the display (e.g., the display 530 in FIG. 5) according to an embodiment ( 601, 602, 603, 604) are shown. While the user 110 is wearing the wearable device 101, the UI can be viewed by the user 110.

Referring to FIG. 6A, according to one embodiment, the wearable device 101 displays a plurality of applications stored in the memory (e.g., memory 520 of FIG. 5) of the wearable device 101 within the display area formed by the display. An example state 601 is shown displaying a list 610 of . The list 610 is displayed within a portion of the display area and may have the form of an application tray. The wearable device 101 may display icons 611, 612, 613, and 614 representing each of the plurality of applications in the list 610. The number of icons 611, 612, 613, and 614 displayed by the wearable device 101 in the list 610 is not limited to the embodiment of FIG. 6A. If the number of applications installed on the wearable device 101 exceeds the specified upper limit that can be simultaneously displayed through the list 610, the wearable device 101 scrolls the list 610. In response to the indicated input, the icon displayed through the list 610 can be switched.

Within state 601 of FIG. 6A , according to one embodiment, the wearable device 101 may identify one or more external objects 621 , 622 , 623 , and 624 viewed by the user 110 . While the user 110 is wearing the wearable device 101, the wearable device 101 can identify external objects 621, 622, 623, and 624 included in the display area of the display. Based on identifying the external objects 621, 622, 623, 624, the wearable device 101 guides that the external objects 621, 622, 623, 624 have been identified by the wearable device 101. Bounding boxes 631, 632, 633, and 634 may be displayed within the display area. Although bounding boxes 631, 632, 633, and 634 in the form of square lines are shown, the embodiment is not limited thereto.

According to one embodiment, the wearable device 101 may identify whether the external object is registered based on the shape, size, and/or color of the external object. An external object is registered, meaning that it includes the shape, size, and/or color within the wearable device 101 and/or external electronic device (e.g., terminal 120 and/or server 122 of FIG. 1). This may mean that the properties of an external object are stored. Hereinafter, the operation of registering an external object in the wearable device 101 is not limited to the operation of storing the properties of the external object in the wearable device 101, and is stored in the external electronic device connected to the wearable device 101. Cases may be included. Within the case, according to one embodiment, the wearable device 101 may access the external electronic device and obtain properties of the external object stored in the external electronic device. The properties may include information about an external object (eg, motion of the external object and/or material output from the external object) that should be tracked by the wearable device 101 . The attribute may include an application matched to the external object and/or a function matched to the external object among a plurality of functions supported by the application. The embodiment is not limited to this, and the property may include a plurality of applications matched to the external object.

For example, when the external object 621 is registered in the wearable device 101, the wearable device 101 displays a bounding box 631 with a designated color (e.g., yellow) to guide registration of the external object 621. ) can be displayed overlapping with the external object 621 included in the display area. Referring to FIG. 6A, the wearable device 101 may display the name of the application (e.g., “1 APP”) matched to the external object 621 in the display area in connection with the external object 621. . If the external objects 622, 623, and 624 that are distinct from the external object 621 are not registered in the wearable device 101, the wearable device 101 overlaps the external objects 622, 623, and 624. The colors of the boxes 632, 633, and 634 may be displayed in a designated color (eg, blue) that is different from the designated color of the bounding box 631.

According to one embodiment, the wearable device 101 may enter state 601 of FIG. 6A in response to an input indicating registration of an external object with the wearable device 101. For example, the wearable device 101 may enter state 601 of FIG. 6A in response to an input indicating displaying a list 610. Within state 601, the wearable device 101 may display a visual object in the display area in the form of a button 640 for returning to another state before entering state 601.

According to one embodiment, the wearable device 101 may identify an input indicating mapping of an application in the list 610 as an external object included in the display area. The input may include a gesture that overlaps a visual object (eg, at least one of the icons 611, 612, 613, and 614) representing a specific application included in the list 610. Referring to FIG. 6B , an example state 602 is shown where wearable device 101 has identified the input included in list 610 . Within state 602, wearable device 101 may display icon 612 overlaid on external object 622 in response to the input. Within state 602, wearable device 101 displays an icon ( 612) can be identified. The gesture may include a drag-and-drop gesture for the icon 612. The gesture is performed in a portion of the external space corresponding to the icon 612 (e.g., an area visible to the user 110 wearing the wearable device 101, a space in which the icon 612 floats). It can be identified based on the hand (or finger) being moved. The gesture may be identified based on the hand approaching less than a specified distance from the portion. The gesture may be identified based on the hand remaining within a portion of external space corresponding to icon 612 for more than a specified period of time (e.g., approximately 1 to 1.5 seconds). Although the gesture is illustrated based on the hand, the embodiment is not limited thereto. For example, the wearable device 101 may perform the gesture based on the direction of the user's 110 gaze toward the icon 612 and/or the period during which the direction of the gaze of the size 110 is toward the icon 612. can be identified. For example, the wearable device 101 may respond to the user's 110 utterance (e.g., "Map which pencil, which application") and the user's 110 gesture within the time interval in which the utterance was received. Based on this, inputs that map a specific application to a specific external object can be identified. Based on the input, the wearable device 101 may allocate the application corresponding to the icon 612 to the external object 622 included in the display area.

In the state 602 of identifying the input dragged along the path 650 of FIG. 6B, the wearable device 101 uses the external object 622 to open an application (e.g., an application corresponding to the icon 612). Properties for execution can be obtained. Although the input for moving the icon 612 to the external object 622 along the path 650 is described as an example, the embodiment is not limited thereto. For example, the wearable device 101 moves one of the icons 611, 612, 613, and 614 to one of the external objects 621, 622, 623, and 624, based on another input, Attributes for executing the application corresponding to the icon moved by the other input can be obtained. For example, the wearable device 101 may display a visual object in the form of a window 660 for obtaining the properties within the display area. Window 660 may be referred to as a screen for receiving one or more properties related to the application. Within the window 660, the wearable device 101 may display information related to the application and/or the external object 622 selected by the input. Within the window 660, the wearable device 101 may display information indicating an executable function (eg, “function: schedule transmission”) using the external object 622. The function may be supported by the application. Within window 660, wearable device 101 can receive properties required to execute the function. Attributes required to execute the function may be set by the application selected by the input.

Referring to FIG. 6B, according to one embodiment, the wearable device 101 may receive an input for selecting information to be recognized by the wearable device 101 within the window 660. For example, wearable device 101 may display, within window 660, options for selecting whether to track the motion of a portion of external object 622 or a stroke drawn by external object 622. there is. For example, when the external object 622 is recognized as a pen containing ink, the wearable device 101 may display the above options in the window 660. Wearable device 101 may display, within window 660 , options for selecting a portion of external object 622 to be tracked by wearable device 101 . For example, when an option for tracking the motion of the external object 622 is selected among the options, the wearable device 101 may track an end of the external object 622. The one end of the external object 622 can be selected based on the position of the icon 612 dragged onto the external object 622 along a path 650 among both ends of the external object 622. there is. The embodiment is not limited to this, and the wearable device 101 may additionally receive an input for selecting a portion of the external object 622 to be tracked by the wearable device 101.

Referring to FIG. 6B, an example is shown in which the wearable device 101 displays a window 660 containing attributes related to the schedule application within a state 602 in which an icon 612 corresponds to the schedule application. Within state 602, the wearable device 101 tracks the external object 622 and displays information obtained by tracking the external object 622 (e.g., the path of a portion of the external object 622 and/or handwriting drawn by the external object 622). You can display text (e.g., “Calendar: Personal calendar”) to select the type of calendar to be added. Within state 602 , wearable device 101 may display options for selecting whether to track a tip of external object 622 or a stroke drawn by external object 622 . For example, if the option to track strokes drawn by external object 622 is selected, wearable device 101 may detect handwriting input entered through the substance (e.g., ink and/or graphite) of external object 622. can be identified. The one end of the external object 622 is a point of the external object 622 where the icon 612 dragged along the path 650 overlaps, among both ends of the external object 622 extending along the length direction. It can be selected as one adjacent group. The wearable device 101 may selectively display a window 660 according to the type of application selected in the list 610. Within state 602 of FIG. 6B, in response to input indicating selection of a button contained within window 660 (e.g., a button containing specified text such as “OK”), wearable device 101 may: Based on the properties included in the window 660, the external object 622 can be registered. In one embodiment that registers the external object 622 independently of the window 660, the wearable device 101, based on an input indicating dragging the icon 612 along the path 650, registers the external object ( 622) can be registered. Based on the registration of the external object 622, the wearable device 101 tracks the external object 622 and determines a function to be executed (e.g., a function to be executed based on the properties included in the window 660), and the external object ( 622).

Although the operation of the wearable device 101 acquiring at least one attribute set by the schedule application through the window 660 has been described, the embodiment is not limited thereto. According to one embodiment, the wearable device 101 may identify one or more properties required to execute an application corresponding to the external object 622 in a state 602 for registering the external object 622. . The one or more properties may be set by an application corresponding to the icon moved to the external object 622. For example, the one or more properties may be classified as shown in Table 1 based on applications installed on the wearable device 101.

application property schedule application Type of calendar in which the schedule will be saved, number of repetitions, reminder timing Note application Digital notes in which handwriting will be stored, style of strokes (e.g. color, thickness, transparency and/or pattern) Messenger application receiver Payment application currency, bank, recipient

Referring to Table 1, when matching the note application and the external object 622, the wearable device 101 stores a digital note containing at least one stroke drawn by the external object 622, and/ Alternatively, attributes for the style of the at least one stroke may be obtained. When matching the messenger application and the external object 622, the wearable device 101 can track the external object 622 and obtain properties of the counterpart who will receive the identified text message. When matching a payment application and an external object 622, the wearable device 101 tracks the external object 622 to obtain the currency for the identified amount, the other party to whom the amount is to be transferred, and/or the bank. You can. Independently of the properties included in Table 1, the wearable device 101 can obtain properties for whether to track a single end of the external object 622 or a stroke drawn by the external object 622. See Figure 6C. For reference, a state 603 in which the wearable device 101 has registered an external object 622 is shown. State 603 may be a state subsequent to state 602. Within the state 603, the wearable device 101 may display an icon 615 representing an application matched to the external object 622 included in the display area in association with the external object 622. In one embodiment in which the external object 622 is registered based on an input that moves the icon 612 in FIG. 6B to the external object 622 along the path 650, the icon 615 in FIG. 6C is the same as in FIGS. 6A to 6C. It may correspond to icon 612 in FIG. 6B. The examples are not limited thereto. As described with reference to the state 601 of FIG. 6A, within the state 603 after the external object 621 is registered with the wearable device 101, the wearable device 101 is connected to the external objects 621 and 622. ) The bounding boxes 631 and 632, each of which has a wrapping shape, may be displayed in a different color from the bounding boxes 633 and 634. The wearable device 101 displays the names of applications (e.g., “1 APP” and “2 APP”) matched to each of the external objects 621 and 622 in a portion of the display area adjacent to the bounding boxes 631 and 632. ) can be displayed. Within state 603 of FIG. 6C, the wearable device 101 can change analog content generated by the registered external object 622 into digital content using an application matched to the icon 612.

Although an embodiment of registering an external object using the list 610 of FIG. 6A has been described, the embodiment is not limited thereto. Referring to FIG. 6D, a camera (e.g., motion recognition camera 340-2 in FIG. 3B, cameras 440-9, 440- in FIG. 4B) having a FoV that at least partially overlaps the display area of the wearable device 101 An example state 604 is shown in which the wearable device 101 registers an external object 688 using 10) and/or camera 540 of FIG. 5. The wearable device 101 may display a list of one or more external objects registered with the wearable device 101 within a visual object in the form of a window 670 overlaid on at least a portion of the display area. Within window 670, wearable device 101 may display a visual object in the form of a button 672 containing designated text (e.g., “Add Pen”) for registering an external object. In response to an input indicating selection of the button 672, the wearable device 101 may display a UI for registering an external object within the display area.

Within state 604 of FIG. 6D, based on an input indicating selection of button 672, wearable device 101 may display, within the display area, a UI for identifying the user's selection of an external object. You can. For example, the wearable device 101 is a visual object in the form of a window 682 that includes designated text (e.g., “Please recognize the pen to register”) for guiding the selection of an external object within the display area. can be displayed. The wearable device 101 may display the bounding box 684 in association with the external object 688 included in the display area, based on identifying the external object 688 within the display area. The wearable device 101 may display a shutter 686 for recognizing an external object 688 included in the display area. Based on the input indicating selection of the shutter 686 and/or the bounding box 684, the wearable device 101 may display a window 690 for registering the external object 688.

Within state 604 of FIG. 6D , wearable device 101 may display an image for an external object 688 within a window 690 . The image may include an image obtained by controlling a camera by the wearable device 101 that identifies the gesture for the shutter 686. Within the image, wearable device 101 may display a portion of an external object 688 to be tracked by wearable device 101. Within window 690, wearable device 101 may display one or more properties for external object 688. One or more properties for external object 688 may include a name assigned to external object 688. The wearable device 101 may use the text box 691 included in the window 690 to identify an input indicating assignment of the name. One or more properties for external object 688 may include one or more applications matched to external object 688. The wearable device 101 may display a list 692 of applications installed in the wearable device 101 within a window 690. Through the list 692, the wearable device 101 can identify an input indicating matching one or more applications with the external object 688. The wearable device 101 may highlight the icon of an application selected by the input within the list 692. The wearable device 101 may emphasize the icon based on the thickness, color, and/or pattern of the outline of the icon.

According to one embodiment, the wearable device 101 may display properties related to at least one application selected through the list 692 within the portion 693 of the window 690. Within state 604 of Figure 6D, wearable device 101, in response to an input indicating selection of a drawing application for drawing an image within list 692, selects, within portion 693, the drawing application and the drawing application for drawing the image. Related properties can be displayed. For example, the wearable device 101 may display, within the portion 693, a visual object for adjusting the brush type corresponding to the external object 688. The wearable device 101 may perform object recognition on the external object 688 and identify the category of the external object 688 from among designated categories related to the pen. Within a state 604 in which the wearable device 101 has identified an external object 688 having the shape of a fountain pen, the wearable device 101 identifies a category of the external object 688 within portion 693 of the window 690. You can display text indicating the corresponding brush type (e.g., "fountain pen"). The properties displayed through the portion 693 are not limited to this, and the wearable device 101 displays other properties (e.g., stroke width, color, pattern) related to the drawing application within the portion 693. can do. The other properties may also be changed adaptively based on the category of the external object 688.

As described above, according to one embodiment, the wearable device 101 may register an external object while worn by the user 110, based on object recognition using a camera. The wearable device 101 may register an external object, such as a pen, that the user 110 uses to create analog content, and provide the user 110 with a user experience that changes the analog content into digital content. Within the state 604 of registering the external object 688, the wearable device 101 may change the properties of the external object 688 based on the characteristics of the external object 688.

Below, with reference to FIG. 7 , the operation of the wearable device 101 related to the states 601, 602, 603, and 604 of FIGS. 6A to 6D is described.

Figure 7 shows an example of a flow chart for a wearable device, according to one embodiment. The wearable device 101 of FIG. 5 may include the wearable device of FIG. 7 . For example, the operations of FIG. 7 may be performed by the wearable device 101 and/or the processor 510 of FIG. 5 . The operations of FIG. 7 may be related to at least one of the states 601, 602, 603, and 604 of FIGS. 6A to 6D. The operations of FIG. 7 may be related to at least one of the operations of FIG. 2 (eg, operation 210 of FIG. 2 ).

Referring to FIG. 7, within operation 710, according to one embodiment, the wearable device may display a list of one or more applications. Within state 601 of FIG. 6A, the wearable device may perform operation 719 to display a list of one or more applications installed in the wearable device (eg, list 610 of FIG. 6A). Within the list, the wearable device may display one or more icons representing the one or more applications.

Referring to FIG. 7 , within operation 720, according to one embodiment, the wearable device may identify an input indicating allocation of an application included in the list as an external object shown through the display. Based on the icon included in the list and selected by the user (e.g., user 110 in FIG. 1), for example, an input instructing to drag icon 612 along path 650 in FIG. 6B. Thus, the wearable device can identify an input indicating allocation of an application to an external object. Before identifying the input (720-No), the wearable device may maintain displaying the list based on the action (710).

Within the state of receiving an input indicating allocation of an application to an external object (720-Yes), based on operation 730, according to one embodiment, the wearable device requests to execute the application corresponding to the input. At least one attribute can be identified. As shown in Table 1, the at least one property may be set by an application assigned to an external object by input of operation 720. The embodiment is not limited to this, and the number of properties set by the application may be 0. If at least one attribute required to run the application does not exist (730-No), the wearable device may bypass performing operation 740. For example, if no attribute is required to run an application, the wearable device may not obtain at least one attribute based on operation 740.

Within the state of identifying at least one attribute (e.g., an attribute distinguished by the application in Table 1) required to run the application (730-Yes), based on the operation 740, according to one embodiment, the wearable A device can obtain at least one attribute. The at least one attribute may be set by the type of the external object and/or the category of the external object identified based on object recognition. The at least one attribute may be identified through a visual object in the form of a pop-up window displayed by the wearable device to receive the attribute, such as window 660 in FIG. 6B and/or window 690 in FIG. 6D. You can. For example, the wearable device may display a visual object to identify at least one attribute, such as window 660 in FIG. 6B and/or portion 693 of window 690 in FIG. 6D, based on movement 740. can be displayed.

Referring to FIG. 7, within operation 750, a wearable device according to one embodiment may store information for executing an application using an external object. The wearable device may store at least one attribute acquired based on the operation 740 in a memory (eg, memory 520 of FIG. 5) along with information for recognizing an external object. After operation 750, the wearable device records the motion of the external object (e.g., the path of a portion of the external object) and/or analog content generated by the external object (e.g., at least one In response to identifying the stroke), an application may be executed using the stored information based on operation 750.

As described above, according to one embodiment, a wearable device may register an external object, such as a pen, that does not support communication with the wearable device. Based on the registered external object, the wearable device can acquire digital content from the external object. Below, with reference to FIGS. 8A and 8B , an example of a UI displayed by a wearable device in a state in which different external objects registered as a wearable device are identified is described, according to an embodiment.

FIGS. 8A and 8B illustrate an example of a user interface (UI) displayed by the wearable device 101 based on an external object, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIGS. 8A to 8B. Referring to FIGS. 8A to 8D , according to one embodiment, the wearable device 101 displays a display based on different external objects 810 and 820 included in the display area of the display (e.g., the display 530 of FIG. 5). Thus, different states 801, 802, and 803 for displaying the UI are shown. It is assumed that each of the external objects 810 and 820 is registered with the wearable device 101 in response to an input indicating registration of the external object, as described above with reference to FIGS. 6A to 6D and/or 7. .

Referring to FIG. 8A, within state 801, according to one embodiment, the wearable device 101 uses a camera (e.g., motion recognition camera 340-2 of FIG. 3B) with a FoV that at least partially overlaps the display area. , external objects 810 and 820 can be identified using the cameras 440-9 and 440-10 of FIG. 4B and/or the camera 540 of FIG. 5. The wearable device 101 can identify whether each of the external objects 810 and 820 is an external object registered to the wearable device 101 based on the shape and/or size of each of the external objects 810 and 820. You can. Within the above assumption, the wearable device 101 can identify the attributes assigned to each of the external objects 810 and 820. Based on the identified attribute, the wearable device 101 may display one or more visual objects based on the attribute, overlapping with external objects 810 and 820 within the display area.

In the state 801 of FIG. 8A, when the note application for creating a digital note is assigned to the external object 810, the wearable device 101 is linked to the external object 810 and represents the note application. , the icon 832 can be displayed. Referring to FIG. 8A, the wearable device 101 may display an icon 832 representing the note application overlapping with an external object 810. The embodiment is not limited to this, and the wearable device 101 may display the icon 832 in a portion adjacent to the external object 810 within the display area. The wearable device 101 may display a figure 830 to guide the note application registered as the external object 810. The wearable device 101 may display a rectangular shape 830 having a location associated with an external object 810. The color of the shape 830 may have a specified transparency (transparency or opacity) and may at least partially include the color of the icon 832. For example, the color of the figure 830 may be a gradient of the color of the icon 832 and the color of the external object 810. A polygonal figure 830 such as a square is shown as an example, but the embodiment is not limited thereto, and the wearable device 101 may display a figure 830 in the form of a closed curve and/or line. The wearable device 101 may display text indicating properties assigned to the external object 810 overlapping with the external object 810 . Within the exemplary state 801 in which the note application is assigned to the external object 810, the wearable device 101 overlaps the external object 810 and displays the name of the digital note matched to the external object 810 (e.g., “K-Note”) can be displayed.

In the state 801 of FIG. 8A, when a messenger application for exchanging messages is assigned as an external object 820, the wearable device 101 overlaps the external object 820 and displays an icon representing the messenger application. (842) can be displayed. The wearable device 101 may use a shape 840 that overlaps the external object 820 to emphasize that the external object 820 is registered as the wearable device 101. The wearable device 101 may display the color of the figure 840 based on at least one of the color of the icon 842 and/or the color of the external object 820. The wearable device 101 may overlap the external object 820 and display attributes (eg, recipient) assigned to the external object 820 based on the messenger application.

According to one embodiment, the wearable device 101 may identify a designated body part, such as a hand 850, that is moved within the display area to select external objects 810 and 820. Referring to FIG. 8B , in response to identifying a hand 850 pointing to an external object 810 and/or a visual object (e.g., an icon 832) overlapping with the external object 810, the wearable device 101 ) may display a UI for changing properties assigned to the external object 810, such as the state 802. The UI may be displayed overlapping with the external object 810 within the display area. Within state 802, the wearable device 101 may display the UI for changing the properties assigned to the external object 810 based on the note application matched with the external object 810. The UI includes a button 834 for adjusting the style of at least one stroke to be added to the note application by an external object 810 (e.g., a button 834 containing designated text such as “pen settings”) may include. The UI may include a button 836 for selecting a digital note to which information obtained by tracking an external object 810 will be added (e.g., a button 836 containing designated text such as “note settings”). You can. In response to an input indicating selection of one of the buttons 834 and 836, the wearable device 101 may display a window in the display area for changing properties corresponding to the button selected by the input. there is.

Referring to FIG. 8B, in response to identifying the hand 850 pointing to the external object 820 among the external objects 810 and 820, the wearable device 101 displays the external object (850), such as state 803. 820), a UI for changing the assigned properties can be displayed. Gestures for displaying the UI are not limited to the gestures of the hand 850 described above. For example, the wearable device 101 uses data indicating the direction of the user's gaze (e.g., the user 110 in FIG. 1) to indicate that the user gazes at the external object 820 for more than a specified period of time. Based on identifying the UI, the UI can be displayed. The properties changed through the UI may be set by a messenger application matched to the external object 820. Within state 803, the wearable device 101 displays a button 844 (e.g., “change recipient”) for changing the counterpart to receive a message containing information obtained by tracking an external object 820. A button 844) containing specified text such as can be displayed. The wearable device 101 may display the button 844 overlapping with the external object 820. In response to an input indicating selection of the button 844, the wearable device 101 may display a UI for changing the other party, such as a contact list.

As described above, according to one embodiment, the wearable device 101 overlaps with an external object (e.g., external objects 810 and 820) registered as the wearable device 101 and is assigned to the external object. A visual object can be displayed based on one or more properties. Based on identifying an external object registered with the wearable device 101, such as external objects 810 and 820, the wearable device 101 may track the identified external object. Information obtained by tracking an external object may include information about a path of a portion of the external object selected by an attribute assigned to the external object, or at least one stroke drawn by the external object. Based on the information, the wearable device 101 may execute at least one application matched to the external object. Since the wearable device 101 executes the application based on the motion of an external object produced independently of the communication circuit, the wearable device 101 allows the user to use the wearable device 101 with various tools. It can be made possible to control it using .

Below, cases in which the wearable device 101 according to an embodiment executes at least one application matched to an external object based on information obtained by tracking the external object will be described. For example, with reference to FIGS. 9A to 9C and FIGS. 10 to 11 , an example in which the wearable device 101 executes a note application is described. For example, with reference to FIGS. 12A to 12B , an example in which the wearable device 101 executes a calendar application is described. For example, with reference to FIG. 13 , an example in which the wearable device 101 executes a messenger application is described. For example, with reference to FIG. 14 , an example in which the wearable device 101 executes a recording application is described. For example, with reference to FIGS. 15A and 15B , an example in which the wearable device 101 executes a plurality of applications assigned to one external object is described.

FIGS. 9A to 9C illustrate an example of an operation in which the wearable device 101 executes an application based on information obtained by tracking an external object 130, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIGS. 9A to 9C. Hereinafter, it is assumed that a note application for creating digital content such as a digital note combining text, image, and/or video is matched to the external object 130. The wearable device 101 may identify an input indicating registration of the note application as an external object 130 based on the operations of FIGS. 6A to 6D and/or 7 .

Referring to FIG. 9A , within state 901, the wearable device 101, according to one embodiment, may track an external object 130. Tracking of the external object 130 by the wearable device 101 may be performed based on an application executed by the wearable device 101 and/or the terminal 120. The wearable device 101 may include a camera (e.g., the motion recognition camera 340-2 in FIG. 3B, cameras 440-9, 440-10 in FIG. 4B), and/or a camera having a FoV that at least partially overlaps the display area. Using the camera 540 of FIG. 5, an external object 130 can be identified and/or tracked. The operation of identifying the external object 130 may include the operation of identifying one or more applications allocated to the external object 130. Within the state 901 in which the external object 130 is identified, the wearable device 101 displays an icon ( 920) can be displayed.

Referring to FIG. 9A, within the state 901 of tracking the external object 130, the wearable device 101 may display a visual object based on information obtained by tracking the external object 130 within the display area. there is. Within a state 901 in which the wearable device 101 tracks the path of a portion of an external object 130 (e.g., a point P corresponding to a pen tip), the wearable device 101 is positioned within the display area. A virtual object 930 representing the path may be displayed. The wearable device 101 determines the path of the external object 130 that is contacted on a plane (e.g., the plane formed by the note 910) based on the distance between the external object 130 and the wearable device 101. can be identified. For example, based on identifying contact between an external object 130 and the plane, wearable device 101 can track the path. The wearable device 101 may display a virtual object 930 representing the tracked path within the display area.

Referring to FIG. 9A , an exemplary state 901 in which a virtual object 930 in the form of a line and/or stroke is displayed by the wearable device 101 is shown, but the embodiment is not limited thereto. For example, the wearable device 101 may display a stroke included in the virtual object 930 based on the shape of the external object 130 and/or the shape of a portion of the external object 130 selected by the user 110. You can change its thickness, color and/or pattern. For example, the wearable device 101 may change the color of a stroke included within the virtual object 930 based on the properties assigned to the external object 130 (e.g., properties set through portion 693 in FIG. 6D), You can set at least one of transparency or thickness. For example, the color of the stroke included in the virtual object 930 is set to a specific color by an attribute assigned to the external object 130, or a first color adjacent to the external object 130 within the display area ( For example, it may be set to a second color based on the background color (e.g., a second color in which the first color is inverted). According to one embodiment, the wearable device 101 may change the color of the stroke based on the color distribution within the display area. For example, the wearable device 101 may identify a representative color from the color distribution by applying weights assigned to each portion of the display area to the color distribution. The wearable device 101 can set the color of the stroke by inverting the representative color. The wearable device 101 may invert colors based on brightness, saturation, and/or distance between colors in a color space.

Although an embodiment in which the wearable device 101 tracks the path of the point P of the external object 130 is described, the embodiment is not limited thereto. For example, the wearable device 101 may identify at least one stroke drawn by a material (eg, ink and/or graphite) output from the external object 130. The wearable device 101 can selectively track either the path of the point P or the material based on the attribute assigned to the external object 130.

Referring to FIG. 9A, within the state 901 of tracking the external object 130, the wearable device 101 uses information acquired by tracking the external object 130 (e.g., for displaying the virtual object 930). Information) can be used to display a button 940 for executing an application (e.g., a note application corresponding to the icon 920). Button 940 may include designated text (eg, “done”) for executing the application. The button 940 may be displayed based on the location of the external object 130 within the display area. In response to an input indicating selection of the button 940, the wearable device 101 may execute the note application. The embodiment is not limited to this, and the wearable device 101 executes the note application before the input (e.g., when the external object 130 is identified), and based on the input related to the button 940, You can execute the functions of the note application. In one embodiment, the wearable device 101 may transmit a signal to the terminal 120 in response to the input to run a note application installed on the terminal 120 based on the information.

Referring to FIG. 9 , an embodiment in which the wearable device 101 executes a note application corresponding to an external object 130 based on an input related to the button 940 is shown, but the embodiment is not limited thereto. For example, the wearable device 101 may track the path of the point P of the external object 130 independently of displaying the button 940. The wearable device 101 may execute the note application independently of the button 940 based on information obtained by tracking the external object 130.

In a state in which the note application is executed, the wearable device 101 uses properties assigned to the external object 130 (e.g., for the note application, a digital note in which handwriting drawn by the external object 130 is stored). Thus, among the different functions supported by the note application, at least one function matching the attribute can be executed. For example, the wearable device 101 may add an image and/or at least one character represented by the virtual object 930 to a digital note stored in the wearable device 101 by a note application. For example, the image and/or at least one character represented by the virtual object 930 is displayed on the wearable device 101 and/or the terminal 120 based on execution of the note application of the wearable device 101 and/or the terminal 120. /Or it may be displayed within the display area of the terminal 120. The wearable device 101, like the virtual object 930, stores information (e.g., digital content) generated by an external object 130, based on execution of the note application corresponding to the external object 130, as a wearable device. It can be stored in the device 101.

According to one embodiment, the wearable device 101 determines the color, thickness, or color of a stroke added by information obtained by tracking the external object 130, based on the posture of the hand 140 holding the external object 130. At least one of the transparency can be changed. Referring to FIG. 9B , example states 901, 902, and 903 in which the wearable device 101 changes properties of an external object 130 based on the posture of the hand 140, according to an embodiment. This is shown. Referring to FIG. 9B, within state 902 after state 901, the wearable device 101 may identify a posture in which the index finger included in the hand 140 is spread. The wearable device 101 may change the properties assigned to the external object 130 based on the location and/or trajectory of the point F within the fingertip of the index finger.

Within state 902 of FIG. 9B, the wearable device 101 displays a stroke extending by the path of the point P of the external object 130, based on the color of the portion adjacent to the point F within the display area. You can change the color and/or pattern of . For example, the posture of spreading the index finger may correspond to a tool (eg, eyedropper) for selecting the color of a stroke drawn by the external object 130. Based on identifying the posture of the hand 140 extending the index finger, the wearable device 101 displays an icon 922 (e.g., an icon 922 representing an eyedropper) to guide the tool within the display area. can do. The wearable device 101 may display an icon 922 in a portion of the display area adjacent to the hand 140.

According to one embodiment, the wearable device 101 may identify an input indicating changing the color of a stroke based on the point F of the fingertip. For example, the wearable device 101 may move the point F to a portion of the display area and then repeatedly move the fingertip. Based on identifying a fingertip that is repeatedly moved (e.g., a fingertip that is repeatedly moved along path T1, between points F1 and F2), the wearable device 101 selects a point within the display area. You can select the color of the part adjacent to (F). The wearable device 101 may change the color of the icon 922 (eg, the color of the outline of the icon 922) based on the selected color. Within state 902, after changing the color based on a gesture related to point F, wearable device 101 may add a stroke with the changed color along the path of external object 130.

In one embodiment, the operation performed by the wearable device 101 using a fingertip gesture is not limited to selecting and/or changing the color described above. Based on identifying the gesture of rotating the fingertip in the form of a circle, the wearable device 101 mixes the color indicated by the attribute assigned to the external object 130 and other colors adjacent to the point F within the fingertip. You can. The wearable device 101 has the shape of the path of the external object 130 and can add a stroke having the mixed colors. For example, the color of the stroke drawn by the path of the point P of the external object 130 (e.g., the color of the strokes included in the virtual object 930) is yellow, and at the point F If the other adjacent color is white, the wearable device 101 may gradually change the color of the stroke to white based on the number of times the fingertip is rotated in the shape of a circle. The wearable device 101 may mix colors differently based on the rotation direction of the fingertip. For example, when the wearable device 101 identifies a gesture of a fingertip rotating along a clockwise direction, the first color indicated by the attribute assigned to the external object 130, and the first color of the point F within the fingertip The mixing ratio of two colors can be changed so that the proportion of the second color increases. In the above example, when identifying a gesture of a fingertip rotating in a counterclockwise direction, the wearable device 101 may change the ratio so that the proportion of the first color increases.

In one embodiment, the posture of the index finger identified by the wearable device 101 is not limited to the posture of spreading the index finger. For example, the wearable device 101 may use the angles of joints included in the index finger to identify whether the index finger is unfolded or bent. Based on identifying the posture of the bent index finger (e.g., the index finger bent toward point F3), the wearable device 101 displays a pattern of a portion adjacent to point P of the external object 130 within the display area. You can run the function to extract . For example, after identifying the posture, the wearable device 101 may display a visual object indicating that the pattern extracting function is executed. The wearable device 101 may identify a portion of the display area including the point P of the external object 130 within the state in which the above function is executed. The wearable device 101 may add a stroke to the path of the point P of the external object 130 based on the identified portion.

Referring to FIG. 9B, within state 903, the wearable device 101 determines the point P of the external object 130 based on the attribute changed by the posture of the hand 140 identified within state 902. ) can display a stroke 932 expressing the path. If the wearable device 101 identifies an input that indicates selecting a color based on the extended index finger, within state 903, the color of the stroke 932 may have the color selected by the input. If the wearable device 101 identifies an input that indicates mixing colors based on a fingertip rotating in the shape of a circle, then within state 903, the color of the stroke 932 is the color mixed by the input. You can have If the wearable device 101 identifies an input that indicates selecting a pattern based on a bent index finger, within state 903, the color of the stroke 932 may have the pattern selected by the input.

In one embodiment, the wearable device 101 may identify the movement of the hand 140 that touches the external object 130 in order to change the attribute assigned to the external object 130. Referring to FIG. 9C, according to one embodiment, the wearable device 101 makes a gesture (e.g., rubbing) of the external object 130 along a direction 950 from a point Q on the surface of the external object 130. , a drag gesture), an example state 904 is shown. The gesture may be performed by a finger (eg, index finger) of the hand 140 holding the external object 130. Based on identifying the gesture, the wearable device 101 may change at least one attribute assigned to the external object 130. For example, wearable device 101 may increase saturation and/or transparency. Similarly, upon identifying a finger rubbing an external object 130 in a direction opposite to direction 950 on the surface of the external object 130, the wearable device 101 may reduce the saturation and/or the transparency. You can.

In one embodiment, the wearable device 101 identifying the movement of the hand 140 touching the external object 130 is not limited to a drag gesture performed along the direction in which the external object 130 extends. For example, based on identifying a gesture (e.g., long tap gesture) that touches a point Q on the external object 130 for more than a specified period (e.g., 1.5 seconds), the wearable device 101 A visual object for changing at least one property assigned to the object 130 may be displayed. Within the example state 904 of FIG. 9C , based on identifying the long tap gesture for point Q, the wearable device 101 displays buttons 962 within the display area for changing the at least one attribute. , 964, 966) can be displayed. Button 962 may correspond to a function for stopping the display of buttons 964 and 966. The button 964 may correspond to a function for changing the transparency of a stroke to be added along the path of the point P of the external object 130. The wearable device 101 may display designated text (eg, “opacity”) within the button 964 to guide changes in transparency. The button 966 can change the saturation of a stroke to be added along the path of the point P of the external object 130. The wearable device 101 may display designated text (eg, “saturation”) within the button 966 to guide changes in saturation.

As described above, the wearable device 101 according to one embodiment may track the path of the point P of the external object 130. Based on the tracked path, the wearable device 101 may display a virtual object (eg, virtual object 930 and/or stroke 932) representing the tracked path within the display area. The virtual object may be related to properties (eg, stroke thickness, color, and/or pattern) assigned to the external object 130 . The wearable device 101 may at least partially change the properties based on the posture of the hand 140 holding the external object 130. The wearable device 101 tracks the external object 130 and/or the hand 140 holding the external object 130, independently of electrical communication between the external object 130 and the wearable device 101, Digital content can be obtained. Within state 903 of FIG. 9B, wearable device 101, in response to an input indicating selection of button 940, creates a digital note including a virtual object 930 and a stroke 932, using the notes application. Based on the execution, it can be stored in memory (eg, memory 520 in FIG. 5).

With reference to FIGS. 9A to 9C , an embodiment of tracking the motion of an external object 130 using a camera having a FoV that at least partially overlaps the display area of the wearable device 101 has been described, but the embodiment does not follow this. It is not limited. Below, with reference to FIG. 10 , an example of an operation in which the wearable device 101 tracks the motion using another camera different from the camera, according to an embodiment, will be described.

FIG. 10 shows an example of an operation in which the wearable device 101 switches a camera for tracking an external object 130, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 10 . Referring to FIG. 10, according to one embodiment, the wearable device 101 includes a first FoV that overlaps the display area formed by the display (e.g., display 530 of FIG. 5) of the wearable device 101. It may include a camera 1012 and a second camera 1014 having a different direction and/or FoV than the first camera. The first camera 1012 and the second camera 1014 of FIG. 10 may be examples of the camera 540 of FIG. 5 .

Referring to FIG. 10 , an example state in which the wearable device 101 tracks an external object 130 using the first camera 1012 and the second camera 1014 is shown, according to an embodiment. For example, within time t1, based on identifying an external object 130 registered with the wearable device 101 using the first camera 1012 with a FoV overlapping the display area, the wearable device 101 ) can obtain information related to the external object 130 using the properties assigned to the external object 130. Within time t1, the wearable device 101 tracks the path of the point P of the external object 130 and creates a virtual object 1020 (e.g. , stroke) can be displayed.

According to one embodiment, the wearable device 101 may select one of the cameras within the wearable device 101 based on identifying an external object 130 that moves between the FoVs of the cameras. While a user (e.g., user 110 in FIG. 1) is wearing the wearable device 101, an external object 130 is detected in the FoV of the first camera 1012 based on the motion of the user's head. can be moved out. At time t2, the external object 130 may move out of the FoV of the first camera 1012 and be included in the FoV of a second camera 1014 that is different from the first camera 1012. According to one embodiment, the wearable device 101 determines that the external object 130 moves outside the display area and/or the FoV of the first camera 1012, based on frames obtained from the first camera 1012. can be identified. Based on identifying that the external object 130 moves outside the display area, the wearable device 101 tracks the path of the external object 130 using the second camera 1014 facing the external object 130. can do.

Referring to FIG. 10, within a time point t2 at which an external object 130 is tracked using the second camera 1014, the wearable device 101 displays the external object tracked by the second camera 1014 within the display area. Information related to (130) can be displayed. For example, the wearable device 101 may display a virtual object 1030 in which the path of the point P of the external object 130 is expressed within the display area. The virtual object 1030 is connected to the path of the point P of the external object 130 tracked by the second camera 1014, and to the first camera 1012 that is different from the second camera 1014 before time t2. The path of the point P of the external object 130 tracked by may include a combined line. The point at which the virtual object 1030 is displayed within the display area is not limited to the example shown in FIG. 10 .

As described above, according to one embodiment, the wearable device 101 moves outside the display area while tracking an external object 130 using the first camera 1012 overlapping the display area. Based on the external object 130, the external object 130 can be continuously tracked using a second camera 1014 that is different from the first camera 1012. Since the external object 130 is tracked using different cameras of the wearable device 101, the wearable device 101 can track the external object (130) independently of the positional relationship between the wearable device 101 and the external object 130. 130) can be traced. When the external object 130 moves outside the display area, the wearable device 101 may display the virtual object 1030 within the display area in order to visualize information obtained by tracking the external object 130. Using the virtual object 1030, the wearable device 101 can inform the user of the motion of the external object 130 that has moved outside the user's FoV.

Below, with reference to FIG. 11 , an example of an operation of the wearable device 101 that displays a virtual object 1030 in the display area based on the motion of an external object 130 moved outside the user's FoV is described. .

FIG. 11 illustrates an example of a UI that the wearable device 101 displays based on information obtained by tracking an external object 130, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 11 . The wearable device 101 of FIG. 11 may be an example of the wearable device 101 of FIG. 10 . For example, the first camera 1012 and the second camera 1014 of FIG. 11 may include the first camera 1012 and the second camera 1014 of FIG. 10.

Referring to FIG. 11 , an example state in which an external object 130 is tracked using a second camera 1014 having a different FoV than the display area of the wearable device 101 is shown. Within the state of FIG. 11 , the wearable device 101 may display information obtained by tracking an external object 130 within a portion 1120 of the display area. Within the portion 1120 , the wearable device 101 may display one or more strokes representing the path of the external object 130 contacted on the plane 1110 . According to one embodiment, the wearable device 101 may identify one or more characters from the path. For example, within time t1, the wearable device 101 may identify a plurality of characters arranged along a line, such as “hello.” The identification of the plurality of characters by the wearable device 101 may be performed by the wearable device 101 and/or an external electronic device (e.g., terminal 120 of FIG. 1) connected to the wearable device 101. . The wearable device 101 may display a virtual object 1131 representing the plurality of characters within a portion 1120 of the display area based on identifying the plurality of characters. The virtual object 1131 may have the shape of a path of an external object 130 contacted on the plane 1110.

Referring to FIG. 11 , the wearable device 101 may identify a gesture indicating adding a line in which a plurality of characters are arranged based on the motion of the external object 130. Within the example state of FIG. 11 identifying alphabets, the wearable device 101 can identify a plurality of characters written along a direction from left to right. Within the above state, the wearable device 101 uses a second line different from the first line on which the plurality of characters are arranged, based on identifying the external object 130 moved from right to left, to identify the external object. A new character expressed by the path of (130) can be recognized. Referring to time t2, the wearable device 101 may identify characters (“WORLD”) written within the plane 1110 along a direction from the left to the right of the plane 1110. The wearable device 101 displays a virtual object 1132 on which characters (“WORLD”) are placed within a line different from the line on which the characters (“hello”) identified at time t1 are placed within a time point t2, It can be displayed within a portion (1120). Similarly, between time t2 and time t3, the wearable device 101 displays a first object in which characters (“hello”) are placed based on identifying the external object 130 moving back to the left side of the plane 1110. Lines, and characters (“glasses”) obtained by tracking the point (P) of the external object 130 may be added within a line that is different from the second line where the characters (“WORLD”) are arranged. The virtual object 1133 displayed in a portion 1120 of the display area of the wearable device 101 at time t3 is composed of characters (“hello”, “WORLD”, and “glasses”) through three different lines. can be displayed.

As described above, the wearable device 101 may identify a gesture indicating adding a line in which a plurality of characters are arranged by comparing the direction in which the letters are arranged and the direction of the external object 130. For example, in the case of a language such as English, where characters are arranged from left to right within a line, the wearable device 101 performs the gesture based on detecting an external object 130 moving from right to left. can be identified. For example, in the case of a language in which letters are arranged from right to left, such as Arabic, the wearable device 101 can identify the gesture based on identifying the external object 130 moving from left to right. there is. Since the wearable device 101 uses the direction of the external object 130 to add a line in which the characters drawn by the external object 130 are arranged, the wearable device 101 creates a plane ( 1110), it is possible to support recognizing characters arranged along a plurality of lines.

FIGS. 12A and 12B illustrate an example of an operation in which the wearable device 101 executes an application based on information obtained by tracking an external object 130, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIGS. 12A to 12B. Hereinafter, it is assumed that the calendar application for storing the schedule is matched to the external object 130. The wearable device 101 may identify an input indicating registration of the calendar application as an external object 130 based on the operations of FIGS. 6A to 6D and/or 7 .

Referring to FIG. 12A, in the state 1201 of identifying the external object 130 matched to the calendar application, the wearable device 101 may display an icon 1210 representing the calendar application in the display area. . Together with the icon 1210, the wearable device 101 may display a visual object that overlaps the external object 130 and indicates the attribute assigned to the external object 130. Within the example state 1201 of FIG. 12A, the wearable device 101 displays text 1220 indicating an account matched to the external object 130 (e.g., “Account A”) among accounts registered in the calendar application. ) can be displayed overlapping with the external object 130.

Within state 1201 of FIG. 12A , the wearable device 101 is connected to the path of a portion of the external object 130 or to the material contained in the external object 130, based on the attributes assigned to the external object 130. At least one stroke drawn by can be traced. For example, within the state 1201 of identifying an attribute set to track ink extending from one end of the external object 130 (e.g., the pen tip of the external object 130), the wearable device 101 tracks the ink. At least one stroke drawn by can be identified. The wearable device 101 may identify a character expressed by the at least one stroke (in the example of FIG. 12A, “11/1 vacation”). The wearable device 101 may obtain a result of character recognition from the at least one stroke using the terminal 120 and/or the server 122 connected to the wearable device 101. For example, in order to reduce the amount of computation of the wearable device 101, the wearable device 101 may request the terminal 120 and/or the server 122 to recognize characters from strokes drawn with the ink. . By reducing the amount of calculation of the wearable device 101, heat emitted from the wearable device 101 can be reduced.

In the state 1201 of FIG. 12A in which a character drawn by ink output from the external object 130 is identified, the wearable device 101 uses a button 1230 to execute a calendar application matched to the external object 130. The input that indicates selection can be identified. In response to the input, the wearable device 101 may execute a calendar application based on characters drawn with the ink (eg, “11/1 vacation”). In one embodiment, the wearable device 101 may execute a function based on the characters using a calendar application executed before the input. Based on identifying the character drawn by the external object 130, the wearable device 101, prior to receiving the input associated with the button 1230, executes the calendar application to perform the function associated with the identified character. You can run . For example, while the calendar application is running, the wearable device 101 may add schedule data based on the information included in the text 1220 to the account indicated by the text 1220.

Referring to FIG. 12B, the wearable device 101 displays a window 1240 for adding schedule data based on an input indicating selection of the button 1230 in the state 1201 of FIG. 12A. (1202) is shown. Within state 1202, wearable device 101 may display a window 1240 for adding schedule data based on characters identified within state 1201. Within the window 1240, the wearable device 101 may display parameters (eg, schedule title, and time) required for generating the schedule data. The wearable device 101 may change the parameters based on information (eg, the characters) obtained by tracking the external object 130 within the state 1201. For example, a parameter contained within window 1240 and representing the name of the schedule data (e.g., a parameter with a field name of “Schedule Title”) may contain some of the characters identified within state 1201 (e.g., may include "vacation"). For example, a parameter contained within window 1240 and representing a time of schedule data (e.g., a parameter with a field name of “Time”) may be a portion of the characters identified within state 1201 (e.g., “ 11/1"). Matching of the characters identified within the parameters and states 1201 is an engine (e.g., a natural language engine (e.g., natural language engine) executed by the wearable device 101 and/or server 122 for natural language processing. It can be performed by a natural language unit (NLU)).

Referring to FIG. 12B, the wearable device 101 may display a button 1242 within the window 1240 for adding schedule data based on parameters displayed through the window 1240. Within button 1242, wearable device 101 may display designated text (e.g., “Save”) to guide adding and/or saving schedule data. Within window 1240, wearable device 101 may display a button 1244 to stop adding the schedule data (e.g., a button 1244 containing designated text, such as “Cancel”). .

Within state 1202 of FIG. 12B, in response to an input indicating selection of button 1242, wearable device 101 executes the calendar application to schedule a schedule based on the parameters displayed via window 1240. Data can be saved. Referring to FIG. 12B, the schedule data may be shared among different electronic devices (eg, wearable device 101 and terminal 120) used by the user 110. For example, the terminal 120 connected to the wearable device 101 may display a visual object 1252 based on the schedule data within the screen 1250 displayed through the display of the terminal 120. .

As described above, according to one embodiment, the wearable device 101 includes information for adding schedule data from the path of a portion of the external object 130 and/or at least one stroke drawn by the external object 130. can be obtained. Within states 1201 and 1202 in which the external object 130 and the calendar application are matched, the wearable device 101 may recognize characters drawn by the external object 130 and obtain the information. Since the wearable device 101 adds schedule data using the information, it can obtain the computer readable information independently of communicating with the external object 130.

FIG. 13 illustrates an example of an operation in which the wearable device 101 executes an application based on information obtained by tracking an external object 1330, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 13 .

Referring to FIG. 13, states 1301, 1302, 1303, and 1304 in which the wearable device 101 executes a messenger application using information obtained by tracking an external object 1330, according to one embodiment, are shown. do. Within state 1301, the wearable device 101 may display a list 1310 of applications installed within the wearable device 101, similar to states 601 and 602 of FIG. 6A. Among the icons 1311, 1312, and 1313 included in the list 1310, based on the icon 1311 dragged to the external object 1330 along the path 1320, the wearable device 101 selects the icon 1311. An input indicating matching the messenger application matched to the external object 1330 can be identified. Based on the input, wearable device 101 may transition from state 1301 to state 1302.

Within state 1302 of FIG. 13, the wearable device 101 may display a window 1340 for receiving at least one parameter required for execution of a messenger application. Within the window 1340, the wearable device 101 may display a list for selecting a counterpart to receive a message corresponding to information obtained by tracking the external object 1330. The list may be related to a contact list stored in the memory of the wearable device 101 (eg, memory 520 in FIG. 5). In state 1302 of FIG. 13 , it is assumed that the wearable device 101 has received an input indicating that user A is selected within the window 1340 . Within the above assumption, the wearable device 101 may store an attribute indicating that the external object 1330 matches the function of the messenger application for sending a message to user A.

Referring to FIG. 13, within the state 1303 after registering the external object 1330 based on the state 1302, the wearable device 101 can track the external object 1330 included in the display area. . Within state 1303, the wearable device 101 overlaps with the external object 1330 and indicates an icon 1311 representing an application matched with the external object 1330 and an attribute assigned to the external object 1330. The text 1350 can be displayed. Within the above assumption, the wearable device 101 may display text 1350 (e.g., “User A”) indicating the other party to receive a message containing information obtained by tracking an external object 1330. there is. The embodiment is not limited to this, and the wearable device 101 displays a shape based on the color of the icon 1311 and/or the color of the external object 1330, such as the shapes 832 and 834 of FIG. 8. It can be displayed by overlapping with the object 1330.

Within the example state 1303 of FIG. 13 , the wearable device 101 may identify one or more strokes drawn by ink output from point A (eg, handwriting representing “message”). The wearable device 101 may identify text corresponding to the one or more strokes based on OCR. Within state 1303, the wearable device 101 may display a button 1355 for executing an application (eg, a messenger application) using information obtained by tracking an external object 1330. In response to an input indicating selection of button 1355, wearable device 101 may transition from state 1303 to state 1304.

Referring to FIG. 13, within state 1304, the wearable device 101 may display a window 1360 provided from a messenger application corresponding to an external object 1330. Within the window 1360, the wearable device 101 may display information obtained by tracking the external object 1330. For example, the wearable device 101 may send text (e.g., “message”) identified from strokes drawn by the external object 1330 to the other party (e.g., “message”) indicated by the attribute assigned to the external object 1330. It can be guided using the window 1360 that the information will be transmitted to “User A”). The wearable device 101 includes the window 1360, a button 1372 for transmitting a message based on information included in the window 1360 within the display area, and a button 1375 for stopping transmission of the message. ) can be displayed. Within button 1372, wearable device 101 may display designated text (e.g., “Send”) to guide transmission of the message. Within button 1374, wearable device 101 may display designated text (e.g., “Cancel”) to guide cessation of transmission of the message. For example, in response to an input indicating selection of button 1372, wearable device 101 may send a message, among functions supported by the messenger application, based on information contained within window 1360. functions can be executed. The wearable device 101 may control a communication circuit (eg, the communication circuit 550 of FIG. 5) to transmit a message.

As described above, according to one embodiment, the wearable device 101 can track an external object 1330 matched to a messenger application and obtain information used to transmit a message using the messenger application. For example, the wearable device 101 may obtain text to be included in the message from at least one character drawn by the external object 1330. The wearable device 101 may execute a function for transmitting a message including the text to the other party indicated by the attribute assigned to the external object 1330 using the communication circuit of the wearable device 101.

FIG. 14 illustrates an example of an operation in which the wearable device 101 executes an application based on information obtained by tracking an external object 1410, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 14 . Hereinafter, it is assumed that a recording application for recording an audio signal is matched to the external object 1410. The wearable device 101 may identify an input indicating registration of the recording application as an external object 1410 based on the operations of FIGS. 6A to 6D and/or 7 .

Referring to FIG. 14 , different states 1401 and 1402 in which the wearable device 101 tracks an external object 1410 are shown. Within the state 1401, the wearable device 101 that has identified the external object 1410 overlaps the external object 1410 within the display area and displays an icon 1412 representing the recording application matched with the external object 1410. ) and/or a button 1414 for recording an audio signal may be displayed based on execution of the recording application. Within state 1401, in response to an input indicating selection of button 1414, wearable device 101 may execute a recording application. Based on execution of the recording application, the wearable device 101 may initiate recording of the audio signal. To record audio signals, the wearable device 101 may include a microphone.

Referring to FIG. 14, in a state 1402 of recording an audio signal based on execution of a recording application, the wearable device 101 overlaps an external object 1410 included in the display area and records the recorded audio signal. Text 1420 indicating the name of the file containing can be displayed. Within state 1402, wearable device 101 may track an external object 1410 matched to the recording application. For example, the wearable device 101 may identify handwriting (“bookmark”) drawn by an external object 1410. Information obtained by the wearable device 101 by tracking the external object 1410 may include text corresponding to the handwriting. For example, the information may include the result of performing OCR on the handwriting.

Within state 1402 of FIG. 14 , based on identifying handwriting drawn by an external object 1410, the wearable device 101 may add a bookmark within the audio signal recorded by execution of the recording application. For example, the wearable device 101 may add a bookmark to a section corresponding to the time when the handwriting was drawn within the audio signal recorded by the recording application. The bookmark may be stored in metadata corresponding to the audio signal. The name of the bookmark may include text corresponding to the handwriting.

14, an example of a screen 1430 displayed during playback of an audio signal acquired within state 1402 is shown. The screen 1430 may be displayed by the wearable device 101 in response to an input indicating reproduction of the audio signal. In one embodiment, the wearable device 101 may perform speech-to-text (STT) on the audio signal to obtain text representing the speech included in the audio signal. Within a portion 1432 of the screen 1430, the wearable device 101 may display the obtained text. The wearable device 101 selects text (e.g., "I will write with an object. ") included in the section corresponding to the point in time when the handwriting in the state 1402 was drawn, among the text displayed on the screen 1430, to the handwriting. It can be displayed with a visual object 1434 indicating the bookmark added. For example, the visual object 1434 may have the form of a bounding box to distinguish text recognized from a portion of the audio signal corresponding to the section (e.g., “I will write with an object.”) from other text. You can. Based on the input indicating selection of the visual object 1434, the wearable device 101 may display the properties of the bookmark (eg, the name of the bookmark) corresponding to the visual object 1434 in the display area.

As described above, according to one embodiment, the wearable device 101 may track an external object 1410 matched to a recording application and execute different functions of the recording application. For example, the wearable device 101 may execute a recording function based on the recording application based on the button 1414 displayed overlapping with the external object 1410. For example, the wearable device 101 may execute a function for adding a bookmark to an audio signal recorded by the recording application based on information obtained by tracking the external object 1410.

FIGS. 15A and 15B illustrate an example of an operation in which the wearable device 101 executes a plurality of applications based on information obtained by tracking an external object 1510, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIGS. 15A to 15B. Hereinafter, it is assumed that a plurality of applications (eg, a note application and a recording application) are matched to the external object 1510. The wearable device 101 may identify an input indicating registration of the plurality of applications as an external object 1510 based on the operations of FIGS. 6A to 6D and/or 7 .

Referring to FIG. 15A , different states 1501 and 1502 in which the wearable device 101 tracks an external object 1510 are shown. Within the state 1501, the wearable device 101 that has identified the external object 1510 displays each of a plurality of applications matched to the external object 1510 by overlapping with the external object 1510 included in the display area. Icons 1512 and 1514 can be displayed. For example, the wearable device 101 may display an icon 1512 representing a note application and an icon 1514 representing a recording application. Within state 1501, wearable device 101 can identify a curve drawn by external object 1510 from point P1 to point P2. The curve may be the wearable device 101 visualizing a partial path of the external object 1510. The curve may be drawn with ink within the external object 1510.

Referring to FIG. 15A , in a state 1501 in which a curve drawn by an external object 1510 is identified, the wearable device 101 may execute the plurality of applications matched to the external object 1510. For example, the wearable device 101 can run a note application and a recording application. Based on execution of the recording application, the wearable device 101 may start recording the audio signal. Based on execution of the note application, the wearable device 101 may identify digital content based on the identified curve. For example, based on execution of a recording application, within state 1501, wearable device 101 may obtain an audio signal including utterance 1521.

Referring to FIG. 15A , within state 1502 , similar to state 1501 , wearable device 101 may maintain tracking a curve drawn by external object 1510 . Wearable device 101 may maintain recording within state 1502 based on execution of the recording application. Wearable device 101 may identify, within state 1502, another curve drawn by external object 1510 from point P2 to point P3. In state 1502, the wearable device 101 may acquire an audio signal including the utterance 1522 based on the recording application.

According to one embodiment, the wearable device 101 may acquire first media content including curves moved along the points P1, P2, and P3 based on the states 1501 and 1502. The first media content may be obtained based on execution of a note application among a plurality of applications matched to the external object 1510. The wearable device 101 may obtain second media content in which the remarks 1521 and 1522 are recorded based on the states 1501 and 1502. The second media content may be obtained based on execution of a recording application among a plurality of applications matched to the external object 1510. The first media content and the second media content may be stored in the memory (eg, memory 520 of FIG. 5) of the wearable device 101.

According to one embodiment, the wearable device 101 may synchronize first media content and second media content obtained by executing each of a plurality of applications matched to the external object 1510. Referring to FIG. 15A, after states 1501 and 1502, the wearable device 101 may display a screen 1530 for synchronizing first media content and second media content. The screen 1530 may be displayed through a display area formed by a display device (eg, display 530 in FIG. 5) of the wearable device 101. The screen 1530 may be provided from at least one of a note application or a recording application matched to the external object 1510.

Referring to FIG. 15A, within the screen 1530, the wearable device 101 may display a visual object 1540 corresponding to the first media content obtained based on execution of the note application. For example, the visual object 1540 may be a line representing a partial path of the external object 1510 moved along the points P1, P2, and P3 within the states 1501 and 1502. Based on the screen 1530 on which the visual object 1540 corresponding to the first media content is displayed, the wearable device 101 may execute a function for synchronizing the first media content and the second media content. The function for synchronizing the first media content and the second media content simultaneously synchronizes the line included in the first media content (e.g., a line expressing the path of the external object 1510) and the audio signal included in the second media content. It may include a playback action.

Referring to FIG. 15A, within the screen 1530, the wearable device 101 may display visual objects 1551 and 1552 corresponding to different sections of the second media content. The visual object 1551 may correspond to the first section in which the speech 1521 is recorded within the second media content. The visual object 1552 may correspond to a second section in which the utterance 1522 is recorded within the second media content. The wearable device 101 may display a visual object 1551 corresponding to the first section within a line included in the visual object 1540, adjacent to a portion of the line drawn within the first section. . The wearable device 101 may display a visual object 1552 corresponding to the second section within a line included in the visual object 1540, adjacent to a portion of the line drawn within the second section. As described above, according to one embodiment, the wearable device 101 may synchronize the line included in the first media content and the audio signal included in the second media content within the screen 1530 based on the section. .

According to one embodiment, the wearable device 101 may play first media content and second media content simultaneously. For example, while playing a first animation that draws a curve along points P1 and P2 within screen 1530, wearable device 101 may record an audio signal (e.g., utterance (e.g., utterance) corresponding to visual object 1551. 1521) can be played. After playing the first animation, while playing the second animation drawing a curve along the points P2 and P3 within the screen 1530, the wearable device 101 generates an audio signal corresponding to the visual object 1552. (e.g., an audio signal including the utterance 1522) may be played.

As described above, while a plurality of applications are matched with the external object 1510, the wearable device 101 can acquire a plurality of media contents using each of the plurality of applications. Within example states 1501 and 1502 of FIG. 15A , wearable device 101 may display different types of media content (e.g., first media content comprising visualized strokes, and first media content comprising audio signals). second media content) can be obtained. The wearable device 101 may substantially synchronize the plurality of media contents acquired simultaneously.

Referring to FIG. 15B , an example state 1503 in which the wearable device 101 simultaneously executes a plurality of applications matched to an external object 1510 is shown. The wearable device 101 may register the plurality of applications as external objects 1510 based on the operations of FIGS. 6A to 6D and/or 7 . Hereinafter, it is assumed that the plurality of applications include a note application corresponding to the icon 1512 and a recording application corresponding to the icon 1514.

Referring to FIG. 15B, the wearable device 101 may execute a plurality of applications matched to the external object 1510 in a state 1503 that identifies a curve drawn by the external object 1510. For example, within the state 1503 of identifying the path of a portion of the external object 1510, or the curve drawn by a material (e.g., ink and/or graphite) included in the external object 1510, the wearable device 101 may execute the plurality of applications in response to an input indicating selection of a visual object 1560. Referring to FIG. 15B, within the display area of the wearable device 101, a visual object 1560 having the form of a button containing designated text such as “done” is exemplarily shown, but the embodiment does not apply to this. It is not limited.

Within state 1503 of FIG. 15B, the wearable device 101 may display a multi-window in which screens 1571 and 1572 provided from each of the plurality of applications assigned to the external object 1510 are arranged. . Screens 1571 and 1572 may be displayed on at least a portion of the display area of the wearable device 101. Referring to FIG. 15B , an example case in which the wearable device 101 displays screens 1571 and 1572 combined in parallel in the vertical direction is shown. The embodiment is not limited to this, and the wearable device 101 may display the screens 1571 and 1572 along a direction (eg, horizontal direction) different from the vertical direction. The wearable device 101 may display screens 1571 and 1572 on portions spaced apart from each other within the display area.

Referring to FIG. 15B, based on the above assumption, the wearable device 101 may display a screen 1571 corresponding to the recording application and a screen 1572 corresponding to the note application. The wearable device 101 may acquire an audio signal that was acquired while tracking the external object 1510 or a curve drawn by the external object 1510 within the screen 1571. In the example case where the audio signal includes the utterance 1523, the wearable device 101 may display text for the utterance 1523 on the screen 1571. The wearable device 101 may display an image representing the curve within the screen 1572.

As described above, according to one embodiment, the wearable device 101 may substantially simultaneously execute a plurality of applications assigned to the external object 1510 based on multitasking. The wearable device 101 may display a multi-window in which screens 1571 and 1572 provided from the plurality of applications are arranged. The wearable device 101 may use the multi-window to execute one or more functions for acquiring digital content based on an external object 1510 that does not include a communication circuit.

Below, with reference to FIG. 16 , an operation in which the wearable device 101 uses an external object to execute applications matched to the external object, according to an embodiment, will be described.

Figure 16 shows an example of a flow chart for a wearable device, according to one embodiment. The wearable device 101 of FIG. 5 may include the wearable device of FIG. 16 . For example, the operations of FIG. 16 may be performed by the wearable device 101 and/or the processor 510 of FIG. 5 . The operations of FIG. 16 may be related to the operations of the wearable device 101 described above with reference to FIGS. 8A to 8B, 9A to 9C, and 10 to 15A to 15B. The operations of FIG. 16 may be related to at least one of the operations of FIG. 2 (e.g., operation 220 of FIG. 2) and/or the operations of FIG. 7.

Referring to FIG. 16, within operation 1610, the wearable device according to one embodiment may identify an external object included in the display area. The wearable device can identify external objects registered as a wearable device based on information stored in memory (eg, memory 520 in FIG. 5). The information may include data indicating the shape, color, and/or size of an external object registered as a wearable device. The wearable device includes a camera (e.g., the motion recognition camera 340-2 in FIG. 3B, cameras 440-9, 440-10 in FIG. 4B, and/or FIG. 5) that at least partially overlaps the display area of the wearable device. Based on frames obtained from the camera 540, the external object can be identified. Identifying an external object by a wearable device may include identifying at least one attribute assigned to the external object.

Referring to FIG. 16, within operation 1620, according to one embodiment, the wearable device displays a first visual object representing an application corresponding to the external object based on the location of the external object in the display area. You can. The first visual object may be displayed to highlight an external object identified based on the action 1610. The first visual object may include, for example, the shapes 830 and 840 and icons 832 and 842 of FIG. 8A. The first visual object may include text indicating the application (eg, a name assigned to the application).

Referring to FIG. 16, within operation 1630, according to one embodiment, the wearable device displays at least one of first information related to the motion of an external object or second information including an image drawn by the external object. It can be obtained. For example, the wearable device may obtain the first information that at least partially includes the path of a portion of the external object (eg, point P in FIG. 9A) indicated by an attribute assigned to the external object. The first information may include a path of a portion of an external object contacted on a plane (eg, a plane formed by the note 910 in FIG. 9A). For example, the wearable device may obtain the second information including at least one stroke drawn by a material (e.g., ink and/or graphite) output from the external object, based on the attribute assigned to the external object. there is. For example, the at least one stroke may be referred to as at least one stroke drawn by an external object.

Referring to FIG. 16, in operation 1640, according to one embodiment, the wearable device selects either first information or second information based on an attribute assigned to an external object, and selects the selected information. A second visual object for executing an application can be displayed using . The second visual object may include the button 940 of FIGS. 9A to 9C. According to one embodiment, a wearable device may selectively obtain either the first information or the second information of operation 1630 using attributes assigned to an external object. After operation 1640, the wearable device may execute an application based on the selected information in response to an input indicating selection of the second visual object. Based on the execution of the application, the wearable device may execute the function of the application based on the selected information.

As described above with reference to FIGS. 6A to 6B and FIGS. 7 to 16, according to one embodiment, a wearable device requests an input to register an external object in order to execute an application based on the external object. You can. However, the embodiment is not limited thereto. Hereinafter, with reference to FIGS. 17 and 18, an operation in which a wearable device, according to an embodiment, identifies an external object independently of the input and executes at least one application corresponding to the identified external object is described. An example is explained.

FIG. 17 illustrates an example of an operation in which the wearable device 101 displays a UI for recommending at least one application based on information obtained by tracking an external object 130, according to an embodiment. . The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 17 .

Referring to FIG. 17 , a state 1701 is shown in which the wearable device 101 has identified an external object 1705 included in the display area, according to one embodiment. Hereinafter, state 1701 is a state before the wearable device 101 receives an input indicating that it registers the external object 1705, described above with reference to FIGS. 6A to 6D and/or 7. It is assumed that For example, within state 1701, wearable device 101 may not be able to identify information that matches the shape, color, and/or size of external object 1705 included within the display area. In a state 1701 in which the external object 1705 is not registered with the wearable device 101, the wearable device 101 may identify the category of the external object 1705. If the external object 1705 is classified into a designated category corresponding to a pen containing a substance such as ink and/or graphite, the wearable device 101 identifies at least one stroke drawn by the external object 1705. can do. For example, the wearable device 101 may identify analog content drawn by an external object 1705 on the note 1710. For example, as the external object 1705 is moved on the note 1710 by the hand 140 holding the external object 1705, the wearable device 101 is glued on the note 1710, and the external object 1705 ( 1705), traces of printed ink can be identified.

Within example state 1701 of FIG. 17 , wearable device 101 can identify text 1712 drawn on note 1710 by external object 1705 . The wearable device 101 may identify the text 1712 based on an image obtained from a camera (eg, camera 540 in FIG. 5) and including the note 1710. For example, the wearable device 101 may identify text 1712 by performing OCR on the image. According to one embodiment, the wearable device 101 executes at least one of a plurality of applications installed on the wearable device 101 based on information included in the text 1712 generated by the external object 1705. , can be recommended to the user 110. For example, the wearable device 101 may select at least one application to be recommended to the user 110 based on whether the text 1712 includes characters arranged according to the format included in Table 2. . '#' in Table 2 may mean a number between 0 and 9.

format Applications to be recommended ####-##-###Month #Day
Date format such as ####year #month #day etc. calendar application ## Hour ## Minute or ##:## Reminder (or alarm) application Formulas containing arithmetic operators calculator application Numbers containing monetary units Payment application Non-textual information, including pictures Note application Textual information that does not correspond to the above format Note application

Referring to Table 2, the wearable device 101 may recommend execution of the calendar application based on identifying handwriting written according to conventional notation for recording dates. The wearable device 101 may recommend execution of a reminder application based on handwriting written according to a customary notation for recording time. Referring to FIG. 17, because the text 1712 has the format of a formula including the addition operator (+), the wearable device 101 can recommend a calculator application based on the text 1712. In FIG. 17 Within state 1701, the wearable device 101 may display an icon 1721 representing a calculator application, overlapping with an external object 1705 included in the display area. The wearable device 101 may display an icon 1721 corresponding to the calculator application, as well as other icons representing other applications installed within the wearable device 101. For example, the wearable device 101 may display one or more icons representing applications recently executed by the wearable device 101, along with an icon 1721 of an application (e.g., a calculator application) selected based on the text 1712. Icons can be displayed sequentially.

Referring to FIG. 17 , an example state 1701 is shown in which the wearable device 101 sequentially displays a plurality of icons 1721, 1722, 1723, and 1734. Among the icons 1721, 1722, 1723, and 1724, the wearable device 101 selects the icon 1721 corresponding to the calculator application, which is recommended based on the text 1712, as part of the external object 1705 (e.g. , such as a pen tip, ink can be placed closest to the part where the ink is output from the external object 1705.

According to one embodiment, the order in which the wearable device 101 displays the icons 1721, 1722, 1723, and 1724 on the external object 1705 may be related to the order executed by the wearable device 101. there is. For example, the order of the icons 1722, 1723, and 1724, excluding the icon 1721 corresponding to the calculator application recommended based on the text 1712, is the order of the applications recently executed by the wearable device 101. You can respond to the order. For example, when the wearable device 101 does not recommend any applications, the wearable device 101 may display icons for a plurality of recently executed applications.

Referring to FIG. 17 , the wearable device 101 may align icons 1721, 1722, 1723, and 1724 based on the shape of the external object 1705 shown through the display area. For example, the icons 1721, 1722, 1723, and 1724 may be arranged along the direction in which the external object 1705 extends within the display area. The number of icons 1721, 1722, 1723, and 1724 overlapping on the external object 1705 is determined by the length of the external object 1705 shown through the display area, and the icons 1721, 1722, 1723, and 1724. It can be related to each size. The number of icons displayed together with the external object 1705 may have a specified upper limit or may be adaptively changed based on the length of the external object 1705 shown through the display area.

Referring to FIG. 17, in a state 1701 in which applications are recommended using icons 1721, 1722, 1723, and 1724, the wearable device 101 indicates executing at least one of the recommended applications. input can be identified. For example, the wearable device 101 may execute a calculator application corresponding to the icon 1721 in response to an input indicating selection of the icon 1721. The wearable device 101 may execute a calculator application based on the formula included in the text 1712 and output the result of the formula. The results may be displayed visually within the display area of the wearable device 101 and/or output in the form of an audio signal through a speaker of the wearable device 101.

Referring to FIG. 17, in the state 1702 after executing the calculator application, the wearable device 101 may display a window 1740 for registering an external object 1705. Within the window 1740, the wearable device 101 may display text (e.g., “Do you want to register a calculator to this pen?”) to guide whether to register a calculator application. Within the window 1740, the wearable device 101 may display a button 1742 for registering the calculator application and a button 1746 for stopping display of the window 1740. In response to an input indicating selection of the button 1742, the wearable device 101 may register the external object 1705. In the state of registering the external object 1705, the wearable device 101 may store information for executing a calculator application based on the external object 1705.

As described above, according to one embodiment, the wearable device 101 can identify analog content drawn by an external object 1705, within the state 1701 of identifying the external object 1705, such as a pen. there is. If the external object 1705 is not registered, the wearable device 101 may recommend execution of at least one application based on digital content converted from the analog content. The wearable device 101 may enhance the user experience associated with an external object 1705, such as a pen, by recommending execution of an application.

Figure 18 shows an example of a flow chart for a wearable device, according to one embodiment. The wearable device 101 of FIG. 5 may include the wearable device of FIG. 18 . For example, the operations of FIG. 18 may be performed by the wearable device 101 and/or the processor 510 of FIG. 5 . The operations of FIG. 18 may be related to the operations of the wearable device 101 described above with reference to FIG. 17 . The operations of FIG. 18 may be related to the operations of FIG. 2, FIG. 7, and/or FIG. 16.

Referring to FIG. 18, within operation 1810, according to one embodiment, the wearable device may identify an external object for drawing one or more strokes. The external object may be an external object different from the external object registered in the wearable device, such as the external object 1705 of FIG. 17 . The wearable device may identify the external object using a camera within the wearable device (eg, camera 540 in FIG. 5). For example, a wearable device can identify external objects, including a pen.

Referring to FIG. 18, within operation 1820, the wearable device, according to one embodiment, may identify media content indicated by one or more strokes adjacent to an external object. A wearable device may track the motion of an external object and identify one or more strokes drawn on the external object. The wearable device may acquire an image including the one or more strokes using a camera within the wearable device. The wearable device can identify the media content by performing OCR on the acquired image. The media content identified by the wearable device may include text and/or images indicated by the one or more strokes.

Referring to FIG. 18, in operation 1830, the wearable device, according to one embodiment, may determine whether media content of a specified type has been identified. For example, the wearable device can identify whether the media content includes text arranged according to the specified format in Table 2. A designated type of operation 1830 may be associated with a designated format in Table 2. For example, the designated type may be set to identify at least one application corresponding to media content identified by the wearable device.

When media content of the specified type is identified (1830 - Yes), based on operation 1840, the wearable device may display at least one icon, including an icon representing an application corresponding to the specified type. For example, like the icon 1721 in FIG. 17, the wearable device may display an icon to guide the execution of an application corresponding to a specified type. The wearable device may display icons representing applications corresponding to the specified type, as well as icons representing other applications that have been recently executed by the wearable device. The icons displayed by the wearable device may be displayed overlapping with external objects within the display area. Because the icons are displayed overlapping with the external object, the wearable device can prevent media content from being obscured by the icons. For example, a wearable device may maintain visibility of media content by displaying the icons in an overlapping manner on the external object.

If media content of the specified type is not identified (1830-No), the wearable device may perform operation 1850 and display at least one icon based on the application execution history. For example, a wearable device can display icons representing a specified number of applications that have recently been executed by the wearable device. Within the display area, the wearable device may display the icons overlapping with external objects. When the external object has a shape that extends in the length direction, such as a pen, the wearable device can arrange the icons along the length direction within the display area.

As described above, according to one embodiment, the wearable device may recognize the external object independently of the operation of electrically communicating with the external object. Based on the recognition of the external object, the wearable device may execute a function of generating digital content based on the external object. The embodiment is not limited to this, and the wearable device may communicate with an external object to obtain information used to execute an application installed on the wearable device. Below, with reference to FIG. 19, an example of an operation in which a wearable device communicates with an external object according to an embodiment is described.

FIG. 19 illustrates an example of an operation in which the wearable device 101 executes an application based on information acquired by the external electronic device 1920, according to an embodiment. The wearable device 101 of FIG. 5 may include the wearable device 101 of FIG. 19 .

Referring to FIG. 19, states 1901, 1902, and 1903 after the wearable device 101 identifies the external electronic device 1920, according to one embodiment, are shown. Hereinafter, an external electronic device 1920 in the form of a smart watch will be described as an example, but the embodiment is not limited thereto. For example, the wearable device 101 may be connected to an external electronic device 1920, such as a smart ring or stylus. The wearable device 101 can identify the external electronic device 1920 based on a wireless communication protocol such as Bluetooth. The wearable device 101 may identify the external electronic device 1920 using a camera (e.g., camera 540 in FIG. 5) having a FoV that at least partially overlaps the display area.

Within state 1901 of FIG. 19 , the wearable device 101 may display a list 1910 of applications installed within the wearable device 101 , similar to the states 601 and 602 of FIG. 6A . Among the icons 1911, 1912, and 1913 included in the list 1910, the wearable device 101 can identify the icon 1913 that is dragged onto the external electronic device 1920 along the path 1930. there is. Based on the input of dragging the icon 1913 along the path 1930, the wearable device 101 may allocate a note application corresponding to the icon 1913 to the external electronic device 1920.

Referring to FIG. 19, a state 1902 is shown after the wearable device 101 assigns the note application to the external electronic device 1920. Within state 1902, the wearable device 101 may display an icon 1913 representing the note application assigned to the external electronic device 1920 in a portion of the display area adjacent to the external electronic device 1920. there is. Within state 1902, the wearable device 101 may track the motion of the external electronic device 1920 using a camera. For example, the wearable device 101 may identify the motion of the external electronic device 1920 moving along the path 1940. The wearable device 101 can identify the two-dimensional motion of the external electronic device 1920 from the camera frames. According to one embodiment, the wearable device 101 can electrically communicate with an external electronic device 1920 and identify the motion in three dimensions. For example, within state 1902, wearable device 101 may request information identified by a sensor (e.g., an inertial measurement unit (IMU)) within external electronic device 1920. You can. The information may include three-dimensional motion of the external electronic device 1920 moved along the path 1940. Based on the information received from the external electronic device 1920, the wearable device 101 can identify the three-dimensional motion of the external electronic device 1920.

Referring to FIG. 19 , a state 1903 after executing a note application based on information obtained by tracking an external electronic device 1920 is shown. Within state 1902, the wearable device 101 may switch from state 1902 to state 1903 based on an input for selecting a button 1950 for executing the note application. Within state 1903, the wearable device 101 may execute a note application and display a screen 1960 for storing digital content based on information obtained by tracking the external electronic device 1920. Within the screen 1960, the wearable device 101 may display a visual object 1962 representing the motion of the external electronic device 1920 moved along the path 1940. Because the wearable device 101 uses the sensor of the external electronic device 1920 to identify the three-dimensional motion of the external electronic device 1920 moving along the path 1940, the wearable device 101 detects the visual object 1962 can be displayed three-dimensionally.

In the example state 1903 of FIG. 19, the wearable device 101 displays a screen for browsing a visual object 1962 in three dimensions based on axes (x-axis, y-axis, and z-axis). (1960) can be displayed. For example, based on the axes, the wearable device 101 may identify an input for rotating the visual object 1962. Based on the input, the wearable device 101 may rotate or move the visual object 1962 to provide a user experience of viewing the visual object 1962 within a three-dimensional spatial coordinate system. The wearable device 101 may display a button 1964 for storing digital content corresponding to a visual object 1962 within the screen 1960. Based on the input indicating selection of the button 1964, the wearable device 101 may store information for three-dimensionally displaying the path 1940 as a visual object 1962 in the memory.

As described above, the wearable device 101 according to one embodiment may support registering an external electronic device 1920. Using a sensor included in the external electronic device 1920, the wearable device 101 can track the external electronic device 1920 and enhance information obtained. For example, using a sensor included in the external electronic device 1920, the wearable device 101 can enhance information acquired using a camera.

In one embodiment, there may be a need for a wearable device to recognize the motion of an external object or analog content drawn by an external object to execute an application installed in the wearable device. As described above, according to one embodiment, a wearable device (e.g., wearable device 101 of FIGS. 1 and/or 5) includes a display (e.g., display 530 of FIG. 5) and a camera. It may be configured to include a processor (e.g., the camera 540 of FIG. 5) and a processor (e.g., the processor 510 of FIG. 5). The processor may be configured to identify an external object (eg, external object 130 in FIG. 1) included in the display area of the display using the camera. The processor is configured to display a first visual object (e.g., icon 160 in FIG. 1) representing an application corresponding to the external object based on the location of the external object in the display area. It can be. The processor may be configured to display at least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information including at least one stroke drawn by the external object. It can be configured to obtain one. The processor creates a second visual object (e.g., Figure 2) for executing the application using information selected from the first information or the second information, according to an attribute assigned to the external object. The button 940 of 9a may be configured to display based on the location. According to one embodiment, a wearable device may execute an application installed within the wearable device using information obtained by tracking an external object.

For example, the processor may be configured to obtain the first information by tracking the path based on whether a portion of the external object indicated by the attribute is in contact with the plane. there is.

For example, the processor may be configured to display within the display one or more lines having the form of the tracked path.

For example, the processor may be configured to identify that the external object moves out of the display area due to a motion of the wearable device while tracking the path using the camera, which is the first camera. The processor may be configured to track the path of the external object using a second camera facing the external object that has moved outside the display area. The processor may be configured to display, within the display, a line having the shape of the path tracked by the second camera.

For example, the processor may be configured to display, in the display, the one or more lines having at least one of color, transparency, or thickness set by the attribute.

For example, the processor is selected to track the at least one stroke drawn by ink extending from one end of the external object, based on the attribute, among the first information or the second information. It may be configured to obtain the second information.

For example, the processor may be configured to execute the application using the information selected according to the attribute in response to an input indicating selection of the second visual object.

For example, the processor may be configured to execute the application based on the second visual object and display a screen including an image indicated by the information selected according to the attribute on the display. .

For example, the electronic device may further include a communication circuit. The processor executes the application based on the second visual object to send a message containing text indicated by the information selected according to the attribute, indicated by the second visual object through the communication circuit. Can be configured to transmit to an external electronic device.

For example, the processor may be configured to execute the application based on the second visual object and obtain schedule data based on text indicated by the information selected according to the attribute.

As described above, the method of using a wearable device according to an embodiment may include an operation of displaying a list of a plurality of applications stored in a memory of the wearable device on a display of the wearable device. The method is based on an input indicating that a visual object representing the first application included in the list is superimposed on an external object included in a display area of the display, using the external object to display the first application. It may include an operation to obtain properties for executing. The method includes, after obtaining the attribute, acquiring at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object. may include. The method may include executing the first application using information selected by the attribute from among the first information or the second information.

For example, the operation of acquiring the property may include the operation of displaying a screen for receiving one or more properties related to the first application within the display area in response to the input.

For example, the operation of acquiring the attribute may include selecting one end adjacent to the first visual object superimposed on the external object by the input among both ends of the external object extending in the longitudinal direction. It may include an identifying operation.

For example, acquiring at least one of the first information or the second information may include obtaining the first information related to the identified set of paths.

For example, the executing operation may include executing the application based on text recognized from information selected by the attribute among the first information or the second information.

As described above, the method of using a wearable device according to an embodiment may include an operation of identifying an external object included in a display area of a display of the wearable device using a camera of the wearable device. The method may include displaying a first visual object representing an application corresponding to the external object based on the location of the external object in the display area. The method obtains at least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information comprising at least one stroke drawn by the external object. It may include actions such as: The method includes displaying a second visual object for executing the application using information selected from the first information or the second information, based on the location, according to the attribute assigned to the external object. may include.

For example, the acquiring operation may include acquiring the first information by tracking the path based on whether a part of the external object indicated by the attribute is in contact with the plane. .

For example, the operation of obtaining the first information may include the operation of displaying one or more lines in the form of the tracked path on the display.

For example, the operation of displaying one or more lines may identify that the external object is moved out of the display area by the motion of the wearable device while tracking the path using the camera, which is the first camera. Can include actions. The method may include tracking the path of the external object using a second camera facing the external object that has moved outside the display area. The method may include displaying a line having the shape of the path tracked by the second camera on the display.

For example, the method may include executing the application using the information selected according to the attribute in response to an input indicating selection of the second visual object.

For example, the executing operation may include displaying a screen including an image indicated by the information selected according to the attribute on the display.

For example, the executing operation includes transmitting a message containing text indicated by the information selected according to the attribute to an external electronic device indicated by the second visual object through a communication circuit of the wearable device. It may include actions such as:

For example, the executing operation may include obtaining schedule data based on text indicated by the information selected according to the attribute.

As described above, according to one embodiment, a wearable device (e.g., wearable device 101 of FIGS. 1 and/or 5) includes a display (e.g., display 530 of FIG. 5), memory (e.g., FIG. It may include a memory 520 in Figure 5) and a processor (eg, processor 510 in Figure 5). The processor may be configured to display, within the display, a list of a plurality of applications stored in the memory (eg, list 610 in FIG. 6). The processor, based on an input indicating that a visual object representing the first application included in the list is superimposed on an external object (e.g., external object 130 in FIG. 1) included in the display area, It may be configured to obtain properties for executing the first application using the external object. The processor, after acquiring the attribute, obtains at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object, It can be configured. The processor may be configured to execute the first application using information selected by the attribute among the first information or the second information.

For example, the processor may be configured to display a screen for receiving one or more properties related to the first application within the display area in response to the input.

For example, the processor may be configured to identify, among both ends of the external object extending in the longitudinal direction, one end adjacent to the first visual object overlaid on the external object by the input.

For example, the processor may be configured to obtain the first information related to the identified set of paths.

For example, the processor may be configured to execute the application based on text recognized from information selected by the attribute among the first information or the second information.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for the purpose of being interpreted by or providing instructions or data to the processing device. there is. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various other software, or servers.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (28)

In the wearable device 101,
display 530;
camera 540; and
Includes a processor 510, where the processor:
Identifying an external object 130 included in a display area of the display using the camera;
displaying a first visual object representing an application corresponding to the external object based on the location of the external object in the display area;
At least one of first information related to a path of the external object moved on a plane while the first visual object is displayed, or second information including at least one stroke drawn by the external object Acquire one; and
According to an attribute assigned to the external object, a second visual object for executing the application using information selected from the first information or the second information is displayed based on the location. to, composed,
Wearable devices. The method of claim 1, wherein the processor:
configured to obtain the first information by tracking the path based on whether a portion of the external object indicated by the attribute is in contact with the plane,
Wearable devices. The method of any one of claims 1 to 2, wherein the processor:
configured to display within the display one or more lines having the form of the tracked path,
Wearable devices. The method of any one of claims 1 to 3, wherein the processor:
While tracking the path using the camera, which is a first camera, identify that the external object moves outside the display area due to the motion of the wearable device;
tracking the path of the external object using a second camera facing the external object moved outside the display area; and
configured to display in the display a line having the shape of the path tracked by the second camera,
Wearable devices. The method of any one of claims 1 to 4, wherein the processor:
configured to display within the display the one or more lines having at least one of color, transparency, or thickness set by the attribute,
Wearable devices. The method of any one of claims 1 to 5, wherein the processor:
Based on the attribute selected to track the at least one stroke drawn by ink extending from one end of the external object, obtain the second information from among the first information or the second information, composed,
Wearable devices. The method of any one of claims 1 to 6, wherein the processor:
configured to, in response to an input indicating selection of the second visual object, execute the application using the information selected according to the attribute,
Wearable devices. The method of any one of claims 1 to 7, wherein the processor:
configured to execute the application based on the second visual object and display a screen including an image indicated by the information selected according to the attribute within the display,
Wearable devices. The method of any one of claims 1 to 8, further comprising a communication circuit,
The processor,
Executes the application based on the second visual object and sends a message including text indicated by the information selected according to the attribute to an external electronic device indicated by the second visual object through the communication circuit. configured to transmit,
Wearable devices. The method of any one of claims 1 to 9, wherein the processor:
configured to execute the application based on the second visual object and obtain schedule data based on text indicated by the information selected according to the attribute,
Wearable devices. In the method of the wearable device,
Displaying, on a display of the wearable device, a list of a plurality of applications stored in a memory of the wearable device;
Based on an input indicating overlapping a visual object representing the first application included in the list on an external object included in the display area of the display, for executing the first application using the external object An operation to acquire a property;
After acquiring the attribute, acquiring at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object; and
Comprising the operation of executing the first application using information selected by the attribute among the first information or the second information,
method. The method of claim 11, wherein the operation of acquiring the property is:
In response to the input, displaying a screen for receiving one or more properties related to the first application within the display area,
method. The method according to any one of claims 11 to 12, wherein the operation of acquiring the property includes:
Among both ends of the external object extending in the longitudinal direction, an operation of identifying one end adjacent to the first visual object superimposed on the external object by the input,
method. The method of any one of claims 11 to 13, wherein the operation of acquiring at least one of the first information or the second information includes:
Comprising the operation of obtaining the first information related to the identified set of paths,
method. The method of any one of claims 11 to 14, wherein the executing operation is:
Comprising an operation of executing the application based on text recognized from information selected by the attribute among the first information or the second information,
method. In the method of the wearable device,
Identifying an external object included in a display area of a display of the wearable device using a camera of the wearable device;
Displaying a first visual object representing an application corresponding to the external object based on the location of the external object in the display area;
While the first visual object is displayed, obtaining at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object; and
Including an operation of displaying a second visual object for executing the application using information selected from the first information or the second information, based on the location, according to the attribute assigned to the external object,
method. The method of claim 16, wherein the acquiring operation is:
Comprising an operation of acquiring the first information by tracking the path based on whether a part of the external object indicated by the attribute is in contact with the plane,
method. The method of any one of claims 16 to 17, wherein the operation of acquiring the first information includes:
Including displaying in the display one or more lines having the form of the tracked path,
method. The method of any one of claims 16 to 18, wherein the operation of displaying one or more lines includes:
An operation of identifying that the external object moves outside the display area due to a motion of the wearable device while tracking the path using the camera, which is a first camera;
An operation of tracking the path of the external object using a second camera facing the external object that has moved out of the display area; and
Including an operation of displaying a line having the shape of the path tracked by the second camera on the display,
method. According to any one of claims 16 to 19,
In response to an input indicating selection of the second visual object, further comprising executing the application using the information selected according to the attribute,
method. The method of any one of claims 16 to 20, wherein the executing operation is:
Comprising the operation of displaying a screen including an image indicated by the information selected according to the attribute on the display,
method. The method of any one of claims 16 to 21, wherein the executing operation is:
An operation of transmitting a message containing text indicated by the information selected according to the attribute to an external electronic device indicated by the second visual object through a communication circuit of the wearable device,
method. The method of any one of claims 16 to 22, wherein the executing operation is:
Comprising the operation of obtaining schedule data based on text indicated by the information selected according to the attribute,
method. In the wearable device 101,
display 530;
memory(520); and
Includes a processor 510, where the processor:
within the display, displaying a list 610 of a plurality of applications stored in the memory;
Executing the first application using the external object based on an input indicating overlapping a visual object representing the first application included in the list on the external object 130 included in the display area acquire properties for;
After acquiring the attribute, obtain at least one of first information related to a path of the external object moved on a plane or second information including at least one stroke drawn by the external object; and
configured to execute the first application using information selected by the attribute among the first information or the second information,
Wearable devices. 25. The method of claim 24, wherein the processor:
configured to, in response to the input, display a screen for receiving one or more properties related to the first application within the display area,
Wearable devices. The method of any one of claims 24 to 25, wherein the processor:
Configured to identify, among both ends of the external object extending in the longitudinal direction, one end adjacent to the first visual object superimposed on the external object by the input,
Wearable devices. The method of any one of claims 24 to 26, wherein the processor:
configured to obtain the first information related to the identified set of paths,
Wearable devices. The method of any one of claims 24 to 27, wherein the processor:
configured to execute the application based on text recognized from information selected by the attribute among the first information or the second information,
Wearable devices.

Images