WO2022050604A1

WO2022050604A1 - Electronic device and method for processing in-air gesture in electronic device

Info

Publication number: WO2022050604A1
Application number: PCT/KR2021/011026
Authority: WO
Inventors: 데그티야렌코일리야; 아자로프헨나디; 클레쉬취키릴; 이바노프올렉산드르
Original assignee: 삼성전자 주식회사
Priority date: 2020-09-07
Filing date: 2021-08-19
Publication date: 2022-03-10
Also published as: KR20220032343A

Abstract

An electronic device and a method for processing an in-air gesture in the electronic device, according to various embodiments of the present document, are provided, the electronic device comprising: a communication module; at least one processor; a display module; and a memory, wherein the memory may store instructions that, when executed, cause the processor to receive in-air gesture and button selection information from a digital pen connected through the communication module, determine the type of gesture on the basis of the received in-air gesture and button selection information, recognize the in-air gesture according to the determined gesture type, and control the display module to output the recognized in-air gesture. Other various embodiments are possible.

Description

How to handle in-air gestures on electronic devices and electronic devices

Various embodiments of the present document relate to an electronic device and a method for recognizing and processing an in-air gesture in the electronic device.

Recently, a user-friendly data input interface is required for various electronic devices (eg, augmented reality (AR) devices, virtual reality (VR) devices, or smart TVs). To input data to these devices, a voice recognition-based or gesture-based input method may be used.

Among gesture-based input methods, a method using an input form such as handwriting is one of the most natural methods for interaction between a human and a machine. The gesture recognition technology may be based on a video tracking technology.

As a voice recognition-based or gesture-based input method is used, various disadvantages, such as low reliability, inappropriate input of personal information, or lack of opportunity to change recognized data, may occur.

In addition, all handwriting data and control gestures can be recorded as one stroke without a special mechanism, and in the case of free-form input to content (eg, control gestures, text, figures, sketches, etc.), the handwriting data and gestures It can be very complicated to separate the s into different gestures or strokes.

An electronic device and a method of processing an in-air gesture in an electronic device according to various embodiments of the present disclosure include detecting pen up/down in an in-air gesture input from the air based on a sensor, classifying the type of gesture, and performing the gesture input. can be processed

According to various embodiments of the present disclosure, an electronic device may include a communication module; at least one processor; display module; and a memory; wherein, when executed, the processor receives in-air gesture and button selection information from a digital pen connected through the communication module, and the received in-air Stores instructions for determining a gesture type based on gesture and button selection information, recognizing the in-air gesture according to the determined gesture type, and controlling the display module to output the recognized in-air gesture. can

According to various embodiments of the present disclosure, a method of processing an in-air gesture in an electronic device includes: receiving in-air gesture and button selection information from a digital pen connected through a communication module of the electronic device; determining a gesture type based on the received in-air gesture and button selection information; recognizing the in-air gesture according to the determined gesture type; and outputting the recognized in-air gesture.

An electronic device and a method of processing an in-air gesture in the electronic device according to various embodiments of the present document classify and process the user's in-air gesture according to types, so that the user can intuitively control the electronic device .

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure;

2A is a diagram illustrating an example of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

2B is a diagram illustrating an example of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

2C is a diagram illustrating an example of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

2D is a diagram illustrating an example of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

2E is a diagram illustrating an example of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

3A is a block diagram illustrating an example of a configuration of an electronic device and a digital pen according to various embodiments of the present disclosure;

3B is a diagram illustrating various examples of input and output of an in-air gesture according to various embodiments of the present disclosure.

4 is a flowchart illustrating an example of an operation of classifying types of in-air gestures according to various embodiments of the present disclosure.

5 is a diagram illustrating an example of an operation of filtering a pen up/down in a user's in-air gesture, according to various embodiments of the present disclosure;

6 is a flowchart illustrating an example of an operation of outputting a filtered gesture as text, according to various embodiments of the present disclosure.

7 is a diagram illustrating an example of an operation of processing an in-air gesture in a text form in an electronic device, according to various embodiments of the present disclosure;

8 is a diagram illustrating an example of an operation of processing an in-air gesture in the form of a figure in an electronic device, according to various embodiments of the present disclosure;

9 is a diagram illustrating an example of an operation of selecting a display object using an in-air gesture in an electronic device, according to various embodiments of the present disclosure;

10 is a diagram illustrating an example of an operation of controlling a display object using an in-air gesture in an electronic device, according to various embodiments of the present disclosure;

11 is a diagram illustrating an example of an operation of performing 3D modeling using an in-air gesture in an electronic device, according to various embodiments of the present disclosure;

12 is a diagram illustrating an example of an operation of controlling a smart TV using an in-air gesture, according to various embodiments of the present disclosure.

13 is a diagram illustrating an example of an operation of processing a user's in-air gesture according to a type in an electronic device, according to various embodiments of the present disclosure;

14 is a flowchart illustrating an example of an operation of processing an in-air gesture in an electronic device, according to various embodiments of the present disclosure.

15 is a flowchart illustrating an example of an operation of processing an in-air gesture between an electronic device and a digital pen, according to various embodiments of the present disclosure;

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. Examples and terms used therein are not intended to limit the technology described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like components. The singular expression may include the plural expression unless the context clearly dictates otherwise. In this document, expressions such as “A or B” or “at least one of A or B” may include all possible combinations of items listed together. Expressions such as "first," "second," "first," or "second," can modify the corresponding elements, regardless of order or importance, and to distinguish one element from another element. It is used only and does not limit the corresponding components. When an (eg, first) component is referred to as being “(functionally or communicatively) connected” or “connected” to another (eg, second) component, that component is It may be directly connected to the component or may be connected through another component (eg, a third component).

In this document, "configured (or configured to)" means "suitable for," "having the ability to," "modified to," depending on the context, for example, hardware or software. ," "made to," "capable of," or "designed to" may be used interchangeably. In some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C" refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device. , may refer to a general-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

The electronic device according to various embodiments of the present disclosure may include, for example, a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a PMP ( portable multimedia player), an MP3 player, a medical device, a camera, or a wearable device. A wearable device may be an accessory (e.g., watch, ring, bracelet, anklet, necklace, eyewear, contact lens, or head-mounted-device (HMD)), a textile or clothing integral (e.g. electronic garment); It may include at least one of a body-worn (eg, skin pad or tattoo) or bioimplantable circuit In some embodiments, the electronic device may include, for example, a television, digital video disk (DVD) player, audio , refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set-top box, home automation control panel, security control panel, media box (eg Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ); It may include at least one of a game console (eg, Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include various medical devices (eg, various portable medical measuring devices (eg, a blood glucose monitor, a heart rate monitor, a blood pressure monitor, or a body temperature monitor), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), CT (computed tomography), imager, or ultrasound machine, etc.), navigation device, global navigation satellite system (GNSS), event data recorder (EDR), flight data recorder (FDR), automotive infotainment device, marine electronic equipment (e.g. navigation devices for ships, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or household robots, drones, ATMs in financial institutions, point of sale (POS) in stores of sales) or IoT devices (eg, light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.). In some embodiments, the electronic device is a piece of furniture, a building/structure or a vehicle, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (eg, water, electricity, gas, or a radio wave measuring device). In various embodiments, the electronic device may be flexible or a combination of two or more of the various devices described above. The electronic device according to the embodiment of the present document is not limited to the above-described devices. In this document, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Hereinafter, an electronic device and an electronic device and method for service discovery according to various embodiments of the present disclosure will be described with reference to the accompanying drawings. In this document, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 may be included. In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120 . It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in the volatile memory 132 , and may process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 may use less power than the main processor 121 or may be set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The auxiliary processor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the co-processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used in a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . A sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a LAN (local area network) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses the subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 may support various requirements specified in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 is to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2A to 2E are diagrams illustrating various examples of a situation in which an in-air gesture is input to an electronic device, according to various embodiments of the present disclosure;

According to various embodiments of the present disclosure, the user 202 may use the digital pen 201 (eg, a stylus pen) to input an in-air gesture. For example, the digital pen 201 may include at least one button 201a, and when the button 201a is selected, the electronic device 101 may perform a predetermined function for the in-air gesture. According to various embodiments, in the following description, a state in which the button 201a is pressed may be referred to as a pen down state, and a state in which the button 201a is released is referred to as a pen up state. ) can be referred to as a state.

The digital pen (eg, S Pen ^TM ) according to various embodiments of the present document may be equipped with an inertial system (eg, an acceleration sensor, a gyro sensor) that can be used to track gestures and, in particular, to input handwriting and hand-drawn objects. there is.

Referring to FIG. 2A , the electronic device 101 may be a wearable device and may be wearable on a body part (eg, head) of a user 202 . The user 202 may input handwriting data as an in-air gesture by using the digital pen 201 . For example, the user 202 may input a handwriting corresponding to "hello" using the digital pen 201 .

Referring to FIG. 2B , the electronic device 101 is a tablet personal computer (PC) and may include a display module 160 that detects a contact of the digital pen 201 .

According to various embodiments of the present disclosure, the user 202 may write handwriting on the display module 160 while holding the digital pen 201 . For example, the electronic device 101 may track a point where the digital pen 201 touches the display module 160 , and may identify a trajectory made up of the points as the handwriting data 221 .

Referring to FIG. 2C , the user 202 may control the function of the electronic device 101 using the digital pen 201 . For example, the electronic device 101 may execute a camera application.

According to various embodiments of the present disclosure, the electronic device 101 may control a function in the camera application according to an input from the digital pen 201 . For example, when an image acquired through the camera module 180 is displayed on the screen of the electronic device 101 and the button 201a of the digital pen 201 is selected, the electronic device at the time when the button 201a is selected. The screen displayed at 101 may be photographed.

Referring to FIG. 2D , the electronic device 101 may provide a content editing function using the digital pen 201 . For example, the user of the electronic device 101 selects a photographed image 204 , and uses the digital pen 201 to display text (eg, 'JOSH') or various displays input to the image 204 . can be entered. The electronic device 101 may synthesize text or marks in the image 204 and display it on a screen or store it in a memory.

Referring to FIG. 2E , the electronic device 101 may be a monitor type device. For example, various executable applications may be displayed on the electronic device 101 .

According to various embodiments of the present disclosure, the electronic device 101 may remotely detect an input of the digital pen 201 and perform a predetermined function with respect to an application corresponding to the sensed input among the applications. For example, the electronic device 101 may identify the detected input as a command for selecting and executing a corresponding application.

Referring to FIG. 3A , the electronic device 101 is communicatively connected to the digital pen 201 to receive information corresponding to an in-air gesture from the digital pen 201 .

According to various embodiments of the present disclosure, the electronic device 101 may include a processor 120 , a memory 130 , a display module 160 , a sensor module 176 , and a communication module 190 . The processor 120 may include an in-air gesture classification module 310 , a handwriting recognition module 320 , a sketch and doodle recognition module 330 , a figure and diagram recognition module 340 or an official recognition module 350 . there is.

According to various embodiments of the present disclosure, the electronic device 101 is a wearable device, and may be an HMD device in the form of glasses/helmet, a headset, a head-up display device, or a holographic display device. In addition, various types of display devices may be possible.

According to various embodiments of the present document, although the processor 120 has been described as including various modules, some of the components may be omitted or deleted, and the processor 120 may be configured to perform the operation of the module. can

According to various embodiments of the present document, the in-air gesture classification module 310 may classify the types of in-air gestures remotely input through the digital pen 202 . For example, the type of the in-air gesture may include handwriting, a sketch, a doodle, a shape, a diagram, or a formula, and may include various other classification items. there is.

According to various embodiments of the present document, the handwriting recognition module 320 may process data of the in-air gesture classified as handwriting from the in-air gesture classification module 310 . For example, the handwriting recognition module 320 may recognize a text corresponding to the in-air gesture and output the in-air gesture in the form of text on the screen through the display module 160 .

According to various embodiments of the present disclosure, the sketch and doodle recognition module 330 may process data of the in-air gesture classified into sketches or doodles from the in-air gesture classification module 310 . For example, the sketch and doodle recognition module 330 may extract a feature from the in-air gesture and output a photo or an image of an object corresponding to the extracted feature on the screen through the display module 160 .

According to various embodiments of the present disclosure, the figure and diagram recognition module 340 may process data of an in-air gesture classified as a figure or diagram from the in-air gesture classification module 310 . For example, the figure and diagram recognition module 340 may recognize a figure or diagram corresponding to the in-air gesture, and output the in-air gesture on the screen through the display module 160 in the form of the recognized figure or diagram. .

According to various embodiments of the present document, the formula recognition module 350 may process the in-air gesture data classified as a formula by the in-air gesture classification module 310 . For example, the formula recognition module 350 may extract numbers, formula symbols, operators, etc. from the data of the in-air gesture, and output the in-air gesture in the form of a formula on the screen through the display module 160 .

According to various embodiments of the present disclosure, when the memory 130 is executed, the processor 120 (or the auxiliary processor 123 ) receives an in-air gesture and a button from a digital pen connected through the communication module 190 . Receive selection information, determine a gesture type based on the received in-air gesture and button selection information, recognize the in-air gesture according to the determined gesture type, and output the recognized in-air gesture You can store instructions that control the module.

According to various embodiments of the present disclosure, the display module 160 may display data of the in-air gesture output through the processor 120 . For example, the data of the in-air gesture may be output in the form of handwriting, sketch, doodle, figure, diagram, or formula and displayed on the display module 160 .

According to various embodiments of the present document, the sensor module 176 may detect an in-air gesture. For example, the sensor module 176 may include a gesture sensor, an acceleration sensor, or a proximity sensor, and various other sensors for detecting or classifying an in-air gesture may be included.

According to various embodiments of the present disclosure, the communication module 190 may perform communication connection between the electronic device 101 and the digital pen 201 . For example, the communication module 190 connects the electronic device 101 and the digital pen 201 through a short-distance communication method (eg, BLE) to input or print a button 201a from the digital pen 201 . Information related to the air gesture may be received.

Referring to FIG. 3B , the user 202 may input various gestures using the digital pen 201 . For example, as an in-air gesture to which the gesture input is remotely input, the electronic device 101 may control to classify the in-air gesture type through the in-air gesture classification module 310 .

According to various embodiments of the present document, the user 202 may input a gesture 320a corresponding to 'Hello'. The in-air gesture classification module 310 may classify the type of the input gesture 320a as handwriting, and allow the handwriting recognition module 320 to process the gesture 320a.

According to various embodiments of the present document, the handwriting recognition module 320 may identify a language corresponding to the gesture 320a and identify a character corresponding to the gesture 320a in the language. The confirmed character may be output in the form of text data 320b.

According to various embodiments of the present disclosure, the user 202 may input a gesture 330a corresponding to a picture of a cat. The in-air gesture classification module 310 may classify the type of the input gesture 330a into sketches or doodles, and allow the sketch and doodle recognition module 330 to process the gesture 330a.

According to various embodiments of the present document, the sketch and doodle recognition module 330 may identify a photo or object corresponding to the gesture 330a. The identified photo or object may be output in the form of image data 330b.

According to various embodiments of the present document, the user 202 may input a gesture 340a in the form of a diagram. The in-air gesture classification module 310 may classify the type of the input gesture 340a into a figure or diagram, and allow the figure and diagram recognition module 340 to process the gesture 340a.

According to various embodiments of the present document, the figure and diagram recognition module 340 checks a diagram structure (eg, ratio, color, and numerical value) corresponding to the gesture 340a, and according to the diagram structure, the gesture 340a ) can be output in the form of a diagram object 340b.

According to various embodiments of the present disclosure, the user 202 may input a gesture 350a corresponding to 'E=mc ² '. The in-air gesture classification module 310 may classify the type of the input gesture 350a into a formula, and allow the formula recognition module 350 to process the gesture 350a.

According to various embodiments of the present disclosure, the formula recognition module 350 extracts a number, a formula symbol, an operator, etc. from the gesture 350a, and converts the extracted number, formula symbol or operator into the form of the formula 350b can be printed out.

Referring back to FIG. 3A , the digital pen 201 includes a button 201a , a processor 220 , a memory 230 , a resonance circuit 287 , a charging circuit 288 , a battery 289 , and a communication circuit 290 . ), an antenna 297 , a trigger circuit 298 and/or a sensor 299 . In some embodiments, the processor 220 of the digital pen 201, at least a portion of the resonant circuit 287, and/or at least a portion of the communication circuit 290 may be configured on a printed circuit board or in the form of a chip. there is. The button 201a , the processor 220 , the resonant circuit 287 and/or the communication circuit 290 includes a memory 230 , a charging circuit 288 , a battery 289 , an antenna 297 , and a trigger circuit 298 . ) and/or the sensor 299 may be electrically connected.

The button 201a according to various embodiments of the present document may be mounted on the outside of the digital pen 201 in the form of a physical button. For example, when the button 201a is selected, the processor 220 may transmit, to the electronic device 101 , viewpoint information and in-air gesture information while the button 201a is selected.

According to various embodiments of the present disclosure, the electronic device 101 identifies the in-air gesture input while the button 201a is selected as a gesture in the pen-down (or selects the button 201a) state, and the checked The gesture in the pen-down state may be processed separately from the gesture in the pen-up state.

The processor 220 according to various embodiments of the present document may include a generic processor configured to execute a customized hardware module or software (eg, an application program). The processor 220 includes at least one of various sensors included in the digital pen 201, a data measurement module, an input/output interface, a module for managing the state or environment of the digital pen 201, or a communication module. It may include a component (function) or a software element (program). The processor 220 may include, for example, one or a combination of two or more of hardware, software, and firmware. According to an embodiment, the processor 220 includes information indicating a pressed state of a button (eg, button 337), sensing information acquired by the sensor 299, and/or information calculated based on the sensing information ( For example, information related to the position of the digital pen 201 may be set to be transmitted to the electronic device 101 through the communication circuit 290 .

The resonance circuit 287 according to various embodiments of the present disclosure may resonate based on an electromagnetic field signal generated from a digitizer (eg, the display device 160 ) of the electronic device 101 , and by resonance, an electromagnetic resonance method It can radiate (electro-magnetic resonance, EMR) input signal (or magnetic field and). The electronic device 101 may identify the position of the digital pen 201 on the electronic device 101 by using the electromagnetic resonance method input signal. For example, the electronic device 101, in each of a plurality of channels (eg, a plurality of loop coils) in the digitizer, based on the magnitude of an induced electromotive force (eg, output current) generated by an electromagnetic resonance input signal Thus, the position of the digital pen 201 may be confirmed. Meanwhile, in the above-mentioned bar, although it has been described that the electronic device 101 and the digital pen 201 operate based on the EMR method, this is merely an example, and the electronic device 101 is electrically coupled resonance (ECR) method. It is also possible to generate a signal based on an electric field based on The resonance circuit of the digital pen 201 may be resonated by an electric field. The electronic device 101 may check a potential in a plurality of channels (eg, electrodes) due to resonance in the digital pen 201 , and may also check a position of the digital pen 201 based on the potential . Those skilled in the art will understand that the digital pen 201 may be implemented in an active electrostatic (AES) method, and there is no limitation in the type of implementation. In addition, the electronic device 101 may detect the digital pen 201 based on a change in capacitance (self capacitance or mutual capacitance) associated with at least one electrode of the touch panel. In this case, the digital pen 201 may not include a resonance circuit. In the present disclosure, “panel” or “sensing panel” may be used as a term encompassing a digitizer and a touch screen panel (TSP).

According to various embodiments, a signal having a pattern may be received through the resonance circuit 287 . The processor 220 may analyze a pattern of a signal received through the resonance circuit 287 and perform an operation based on the analysis result. The digital pen 201 according to various embodiments may perform one first communication through the resonance circuit 287 and may perform a second communication through the communication circuit 290 . For example, when the digital pen 201 is inserted into the electronic device 101 , the digital pen 201 may receive information from the electronic device 101 through the resonance circuit 287 . For example, the digital pen 201 may receive a communication signal through the communication circuit 290 when it is removed from the electronic device 101 , but receives a communication signal through the communication circuit 290 even when inserted. This may be possible.

The memory 230 according to various embodiments of the present document may store information related to the operation of the digital pen 201 . For example, the information may include information for communication with the electronic device 101 and frequency information related to an input operation of the digital pen 201 . In addition, the memory 230 includes a program (or application, algorithm, or processing loop). The memory 230 may store a communication stack of the communication circuit 290 . Depending on the implementation, communication circuitry 290 and/or processor 220 may include dedicated memory.

The resonance circuit 287 according to various embodiments of the present document may include a coil (or an inductor) and/or a capacitor. The resonance circuit 287 may resonate based on an input electric field and/or magnetic field (eg, an electric field and/or magnetic field generated by a digitizer of the electronic device 101 ). When the digital pen 201 transmits a signal by the EMR method, the digital pen 201 sets a resonance frequency based on an electromagnetic field generated from an inductive panel of the electronic device 101 . It is possible to generate a signal containing When the digital pen 201 transmits a signal using the AES method, the digital pen 201 may generate a signal using the electronic device 101 and capacitive coupling. When the digital pen 201 transmits a signal by the ECR method, the digital pen 201 transmits a signal including a resonant frequency based on an electric field generated from a capacitive device of an electronic device. can create According to an embodiment, the resonance circuit 287 may be used to change the intensity or frequency of the electromagnetic field according to a user's manipulation state. For example, the resonance circuit 287 may provide various frequencies for recognizing a hovering input, a drawing input, a button input, or an erasing input. For example, the resonant circuit 287 may provide various resonant frequencies according to a connection combination of a plurality of capacitors, or may provide various resonant frequencies based on a variable inductor and/or a variable capacitor.

When the charging circuit 288 according to various embodiments of the present disclosure is connected to the resonance circuit 287 based on the switching circuit, the resonance signal generated in the resonance circuit 287 is rectified into a DC signal and provided to the battery 289 can do. According to an embodiment, the digital pen 201 may determine whether the digital pen 201 is inserted into the electronic device 101 by using the voltage level of the DC signal detected by the charging circuit 288 . Alternatively, the digital pen 201 may check whether the digital pen 201 is inserted by checking a pattern corresponding to the signal checked by the charging circuit 288 .

The battery 289 according to various embodiments of the present document may be configured to store power required for the operation of the digital pen 201 . The battery 289 may include, for example, a lithium-ion battery or a capacitor, and may be rechargeable or replaceable. According to an embodiment, the battery 289 may be charged using power (eg, a DC signal (direct current power)) provided from the charging circuit 288 .

The communication circuit 290 according to various embodiments of the present disclosure may be configured to perform a wireless communication function between the digital pen 201 and the communication module 190 of the electronic device 101 . According to an embodiment, the communication circuit 290 may transmit state information, input information, and/or information related to a location of the digital pen 201 to the electronic device 101 using a short-distance communication method. For example, the communication circuit 290 may include direction information (eg, motion sensor data) of the digital pen 201 acquired through the trigger circuit 298 , voice information input through a microphone, or the remaining amount of the battery 289 . Information may be transmitted to the electronic device 101 . For example, the communication circuit 290 may transmit sensing data obtained from the sensor 299 and/or information related to the position of the digital pen 201 identified based on the sensing data to the electronic device 101 . can For example, the communication circuit 290 may transmit information on the state of the button 201a included in the digital pen 201 to the electronic device 101 . For example, the short-range communication method may include at least one of Bluetooth, Bluetooth low energy (BLE), NFC, and Wi-Fi direct, but there is no limitation on the type.

The antenna 297 according to various embodiments of the present document may be used to transmit or receive a signal or power to the outside (eg, the electronic device 101). According to an embodiment, the digital pen 201 may include a plurality of antennas 297 , and among them, at least one antenna 297 suitable for a communication method may be selected. Through the selected at least one antenna 297 , the communication circuit 290 may exchange a signal or power with an external electronic device.

The trigger circuit 298 according to various embodiments of the present document may include at least one button or a sensor circuit. According to an embodiment, the processor 220 may check the input method (eg, touch or press) or type (eg, EMR button or BLE button) of the button 201a of the digital pen 201 . . According to an embodiment, the trigger circuit 298 may transmit a trigger signal to the electronic device 101 using an input signal of the button 201a or a signal through the sensor 299 .

The sensor 299 according to various embodiments of the present document may include an acceleration sensor, a gyro sensor, and/or a geomagnetic sensor. The acceleration sensor may sense information about the linear motion of the digital pen 201 and/or the acceleration of the digital pen 201 in three axes. The gyro sensor may sense information related to rotation of the digital pen 201 . The geomagnetic sensor may sense information about a direction in which the digital pen 201 faces within an absolute coordinate system. According to an embodiment, the sensor 299 is not only a sensor for measuring movement, but also a sensor capable of generating an electrical signal or data value corresponding to an internal operating state or external environmental state of the digital pen 201 . , for example, may include at least one of a remaining battery level sensor, a pressure sensor, an optical sensor, a temperature sensor, and a biometric sensor. According to various embodiments, the processor 220 may transmit information obtained from the sensor 299 to the electronic device 101 through the communication circuit 290 . Alternatively, the processor 220, based on the information obtained from the sensor 299 , information related to the position of the digital pen 201 (eg, coordinates of the digital pen 201 and/or the displacement of the digital pen 201 ) ) may be transmitted to the electronic device 101 through the communication circuit 290 .

Referring to FIG. 4 , in operation 410 , the electronic device 101 may acquire sensor data transmitted from the digital pen 201 through the communication module 190 . For example, the sensor data may include direction information (eg, motion sensor data) of the digital pen 201 and selection information of the button 201a of the digital pen 201 .

In operation 420 , the electronic device 101 may form a three-dimensional (3D) gesture from the received gesture information. For example, the electronic device 101 may form a three-dimensional trajectory of the gesture as a 3D gesture by using the acquired sensor data.

In operation 421, the electronic device 101 may extract characteristic information of the 3D gesture. For example, the electronic device 101 may extract specific information such as depth information or scale information from the 3D gesture.

In operation 430 , the electronic device 101 may select an informative two dimensional plane valid in the 3D gesture. For example, in the 3D gesture, the electronic device 101 may select a plane capable of identifying the user's gesture as the valid 2D plane.

In operation 440 , the electronic device 101 may filter the selected valid 2D plane gesture based on the pen up/down state information. The processor 120 of the electronic device 101 may identify a gesture input while the button 201a of the digital pen 201 is selected as a pen-down (or selection of the button 201a) gesture. The electronic device 101 may filter the gesture input in the confirmed pen-down state, and transmit a stroke including the filtered gesture in the pen-down state to the in-air gesture classification module 310 .

In operation 441 , the electronic device 101 may extract 2D characteristic information from a valid 2D plane gesture. For example, the characteristic information of the 2D plane may include a change amount (eg, x or y) or pen-up/down information on the x-axis or y-axis of the gesture.

According to various embodiments of the present disclosure, if 3D plane characteristic information and 2D plane characteristic information are extracted as a result of performing

operation

421 or 441 , in operation 450 , the electronic device 101 performs in-air operation based on a recurrent neural network (RNN). Gesture classification can be performed. The processor 120 of the electronic device 101 may transmit the in-air gesture classification result to the in-air gesture classification module 310 .

According to various embodiments of the present disclosure, the electronic device 101 may classify the in-air gesture based on a combination of the extracted 3D plane characteristic information and 2D plane characteristic information. For example, when the 2D plane characteristic information for classification of the in-air gesture is not sufficiently extracted, the electronic device 101 may classify the in-air gesture by further considering the extracted 3D plane characteristic (eg, depth information or scale information) information. can As the electronic device 101 classifies the in-air gesture by further considering the 3D characteristic of the in-air gesture input from the air, accurate classification may be possible according to the characteristic of the in-air gesture.

According to various embodiments of the present disclosure, the electronic device 101 may store information related to characteristic information of a 2D gesture and characteristic information of a 3D gesture corresponding to a specific gesture type. For example, the electronic device 101 may classify the in-air gesture by determining whether the extracted characteristic information of the 2D gesture and the characteristic information of the 3D gesture correspond to characteristic information corresponding to a specific gesture type.

In operation 460 , the in-air gesture classification module 310 of the electronic device 101 may check the transmitted filtered stroke or a result of the in-air gesture classification and select a recognition module. For example, the recognition module may include an in-air gesture classification module 310 , a handwriting recognition module 320 , a sketch and doodle recognition module 330 , a figure and diagram recognition module 340 or an official recognition module 350 . Also, according to the result of the gesture classification, the electronic device 101 transmits the 2D plane characteristic information extracted for the filtered stroke or the in-air gesture to the selected recognition module so that the corresponding recognition module processes the stroke or gesture. .

Referring to FIG. 5 , the electronic device 101 may identify the original gesture 500 input by the user using the digital pen 201 .

According to various embodiments of the present disclosure, the electronic device 101 may filter the pen up/down from the original gesture 500 to distinguish the movement gesture 510 and the valid gesture 520 . For example, the movement gesture 510 may be an input in which the user moves the digital pen 201 in an up or down direction to prepare for data input. The valid gesture 520 is a gesture in which the user inputs actual data, and may include the form of a word or sentence, number, figure, or image according to a preset language.

According to various embodiments of the present disclosure, the electronic device 101 determines that the button 201a of the digital pen 201 is pressed as the pen-down state, and the button 201a in the original gesture 500 of the user Only gestures in a pressed state (eg, a pen-down state) can be filtered. A gesture input in the pen-down state (eg, a state in which the button 201a is pressed) is classified as a valid gesture, and a gesture input in a pen-up state (eg, a state in which the button 201a is released) is classified as a movement gesture. can do. For example, the user may input a gesture while pressing the button 201a for a desired gesture input, and the electronic device 101 performs a movement gesture based on the pen up/down state in the gesture input input during the corresponding period. By filtering 510 , a gesture input that the actual user wants to input may be identified as a valid gesture 520 .

According to various embodiments of the present disclosure, the electronic device 101 may select data corresponding to the valid gesture 520 from the original gesture 500 as a result of the pen-up/down filtering operation.

Referring to FIG. 6 , in operation 610 , the electronic device 101 may identify a pen-up/down-filtered gesture (or stroke) from a gesture input using the digital pen 201 .

In operation 620, the electronic device 101 may perform pre-processing on the filtered gesture.

In operation 621 , the electronic device 101 may extract the filtered characteristic of the gesture.

In operation 631 , the electronic device 101 may detect a language for the filtered gesture. For example, the electronic device 101 may detect a language for a gesture based on external data (eg, language setting or location).

In operation 640 , the electronic device 101 may set a language based on the knowledge. For example, the electronic device 101 may perform operation 641 of checking the RNN weight and operation 642 of checking the language model.

As a result of performing the above-described

operations

621 and 641 , if a characteristic is extracted from the filtered gesture and an RNN weight is checked, in operation 622 , the electronic device 101 may apply a long short term memory (LSTM) RNN algorithm.

As a result of performing

operations

622 and 642 described above, if the LSTM RNN algorithm is applied and the language model is checked, the electronic device 101 may perform decoding in operation 623 .

In operation 650 , the electronic device 101 may output the recognized text as a result of decoding. For example, text corresponding to 'Hello World' may be extracted from the filtered gesture and outputted.

Referring to FIG. 7 , the electronic device 101 is an HMD device, and a user inputs an in-air gesture using a digital pen 201 communicatively connected to the electronic device 101 while wearing the electronic device 101 . can do.

According to various embodiments of the present document, the electronic device 101 may display the VR image 700 . The VR image 700 may include a gesture input keyboard 710 in which the user inputs an in-air gesture input in the form of text, and a gesture display area 720 in which the input text is displayed. The gesture input area 710 may include a search window 711 for searching by inputting text in various forms and a button 712 for deleting the display of the gesture input keyboard 710 .

According to various embodiments of the present disclosure, the electronic device 101 is an electronic device (eg, an HMD device, an AR/VR device, or a smart TV) that is in an environment in which it is important to use a keyboard for text input or is inconvenient to input text or a figure. can

According to various embodiments of the present disclosure, a user inputs a desired text as an in-air gesture as if writing with a pen, and the electronic device 101 extracts text from the input in-air gesture and displays it in the gesture display area 720 . can be displayed The electronic device 101 may extract texts according to various hieroglyphic-type languages (eg, Korean, Chinese, or Japanese). According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

Referring to FIG. 8 , the electronic device 101 is an HMD device, and when the first user 801 wears the electronic device 101 , an AR image 800 may be displayed. The second user 802 inputs the in-air gesture while holding the digital pen 201 communicatively connected to the electronic device 101 while the digital pen 201 communicatively connected to the electronic device 101 is in one state. could be the situation.

According to various embodiments of the present document, the electronic device 101 may determine the type of the in-air gesture input from the digital pen 201 as 'sketch' and process the in-air gesture.

According to various embodiments of the present disclosure, the electronic device 101 may display an image of a muscle, blood vessel, or bone structure of the human body on the AR screen 800 . The second user 801 may input an in-air gesture to the first user 801 wearing the electronic device 101 with the digital pen 201 to visually explain the structure of the human body.

According to various embodiments of the present document, the second user 802 may input 810 a sketch corresponding to a description of a specific structure with the digital pen 201 . The electronic device 101 may process the sketch input 810 through the sketch and doodle recognition module 330 and display the processed sketch input 820 . For example, the electronic device 101 checks the position and movement trajectory of the sketch input 810 of the second user 802 , and applies preset display settings (eg, color, pen thickness) to the AR screen 800 . can be displayed in According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

Referring to FIG. 9 , an AR screen 900 may be displayed on the electronic device 101 . In the AR screen 900, information 910 on objects placed in the living room and a display for selecting objects (eg, flowerpot, picture frame, stand, etc.) are superimposed on the image of the living room of the house. can be

According to various embodiments of the present disclosure, the electronic device 101 checks the type of the in-air gesture input input from the digital pen 201 communicatively connected to the electronic device 101 and applies the in-air gesture to the AR screen 900 . ) can be confirmed as an input for selecting the displayed object. For example, when the movement trajectory of the in-air gesture corresponds to the shape of a specific object among the objects displayed on the AR screen 900 , the electronic device 101 may identify the corresponding object as an input for selecting the object. The electronic device 101 may display a movement trajectory (eg, 921 , 922 or 923 ) of the in-air gesture input corresponding to the shape of a specific object and corresponding in-air gesture input.

According to various embodiments of the present disclosure, the electronic device 101 may check a selection input for a specific object (eg, a flowerpot). For example, the electronic device 101 may determine that the selection input has been input according to whether the button 201a of the digital pen 201 is pressed.

According to various embodiments of the present disclosure, when a selection input for a specific object (eg, flowerpot) is input, the electronic device 101 may display information 910 on the corresponding object. For example, in the information 910 , information on a corresponding flowerpot among flowerpots placed in the living room (eg, 'jade or money tree', 'hedge cactus', or 'century plant') may be highlighted. According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

According to various embodiments of the present disclosure, as the electronic device 101 classifies and processes the in-air gesture input from the digital pen 201 , the user can conveniently manipulate the object displayed on the AR screen 900 . there is.

Referring to FIG. 10 , the AR screen 1000 may be displayed on the electronic device 101 . On the AR screen 1000 , the vacuum cleaner device 1001 communicatively connected to the electronic device 101 may be photographed and displayed.

According to various embodiments of the present disclosure, the cleaner device 1001 may operate according to a control command from the electronic device 101 , and the electronic device 101 may input an in-air gesture using the digital pen 201 to The cleaner apparatus 1001 may be controlled.

According to various embodiments of the present disclosure, the user may use the digital pen 201 to input an input 1010 corresponding to a command to the cleaner apparatus 1001 as an in-air gesture. For example, the user may input an arrow indicating that the target of the command is the cleaner device 1001 and text corresponding to 'Clean' to instruct the cleaner device 1001 to perform a cleaning function. According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

According to various embodiments of the present disclosure, the electronic device 101 checks the 'cleaning function' of the cleaner device 1001 with a command corresponding to the text recognized from the input 1010 , and the cleaner device 1001 cleans It can be controlled to perform a function.

According to various embodiments of the present disclosure, the user may draw and input the virtual wall 1020 with the digital pen 201 as an additional command for the cleaner device 1001 . The electronic device 101 determines that the virtual wall 1020 is the cleaning area of the cleaner apparatus 1001 , and the cleaner apparatus 1001 cleans the area corresponding to the virtual wall 1020 . You can control it not to.

Referring to FIG. 11 , an AR screen 1100 may be displayed on the electronic device 101 . For example, on the AR screen 1100, a 3D modeled costume graphic may be displayed on a stage image.

According to various embodiments of the present disclosure, a user may input an in-air gesture of sketching a clothing design using the digital pen 201 . The electronic device 101 determines that the type of the in-air gesture input through the in-air gesture classification module 310 is 'sketch', and controls the sketch and doodle recognition module 330 to process the in-air gesture. can According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

According to various embodiments of this document, the sketch and doodle recognition module 330 may process an in-air gesture classified as 'sketch'. For example, the sketch and doodle recognition module 330 may 3D model the input in-air gesture to control the user's sketch (eg, 1110 or 1120 ) to be three-dimensionally displayed.

According to various embodiments of the present disclosure, as the type of the in-air gesture is classified as 'sketch' in the electronic device 101, the sketch and doodle recognition module 330 controls to process the sketch, so that the digital pen 201 ) can be used for easy 3D modeling and can be used for prototyping.

Referring to FIG. 12 , a smart TV 1201 that is communicatively connected to the electronic device 101 to serve as a desktop PC is provided, and the smart TV 1201 is provided using a digital pen 201 communicatively connected to the electronic device 101 ( 1201) can be controlled. For example, the electronic device 101 may be connected to the smart TV 1201 based on DeX ^TM .

According to various embodiments of the present disclosure, the AR screen 1201 including the image 1210 captured in real time through the electronic device 101 may be displayed on the smart TV 1201 . For example, the AR screen 1201 may include items for selecting a type of in-air gesture input and items for selecting an in-air gesture input mode. Items for selecting the type of the in-air gesture input include a doodle ('doodle') 1211 , a sketch ('sketch') 1212 , a shape ('shape') 1213 , or a text ('text'). ) 1214 may include items for selection. The items for selecting the in-air gesture input mode include an item selection mode 1221 for selecting an item displayed on the AR screen 1201 or a gesture display mode 1222 for displaying the motion trajectory of the in-air gesture. It may include items for

According to various embodiments of the present disclosure, the user may point to the sketch 1212 item with the digital pen 201 while the item selection mode 1221 is selected. The electronic device 101 may identify the type of the in-air gesture input as 'sketch' when the sketch 1212 item is pointed to and maintained for a predetermined time.

According to various embodiments of the present disclosure, the electronic device 101 checks the in-air gesture input while the button 201a is pressed as a valid gesture, and determines the movement trajectory of the checked in-air gesture on the AR screen 1201 ) can be displayed on the

According to various embodiments of the present disclosure, when the electronic device 101 checks an input of an in-air gesture to control various electronic devices that are communicatively connected to the electronic device 101, the electronic device 101 checks and confirms the type of the in-air gesture. Depending on the type of in-air gesture, it can be processed.

Referring to FIG. 13 , the electronic device 101 may check an input of the in-air gesture 1310 through the digital pen 201 connected to communication. The in-air gesture 1310 input by the user may include a symbol 1211 corresponding to 'ⓜ->ⓔ' and a mathematical formula 1312 corresponding to 'E=mc ² '.

According to various embodiments of the present disclosure, the electronic device 101 may process the text 1211 and the mathematical formula 1312 according to the type of the in-air gesture. For example, the electronic device 101 classifies the symbol 1211 as a 'figure' and controls the figure and diagram recognition module 340 to process it, and classifies the mathematical formula 1312 as a 'formula' to a formula recognition module It can be controlled to process at 350 .

According to various embodiments of the present disclosure, the electronic device 101 may display the processing result 1320 of the input in-air gesture 1310 on the screen 1300 . For example, the figure and diagram recognition module 340 may identify a figure, shape, or sign corresponding to the sign 1211 , and output it as a character 1221 that can be processed by the electronic device 101 . The formula recognition module 350 may identify characters or mathematical operators constituting the mathematical formula 1212 , and output it in the form of a formula 1222 that can be processed by the electronic device 101 . According to various embodiments, the user may input a gesture while pressing the button 201a of the pen 201 for the user to input a desired gesture, and the electronic device 101 may display a gesture as described above with reference to FIG. 5 . The gesture input that the actual user wants to input may be confirmed as a valid gesture by filtering the movement gesture based on the pen up/down state in the gesture input input during the corresponding period.

According to various embodiments of the present disclosure, the electronic device 101 classifies and processes input in-air gestures in various forms according to the type of each gesture, and thus performs an operation on the corresponding gesture according to the user's intention. can make it

Referring to FIG. 14 , in operation 1410 , the electronic device may receive an in-air gesture and button selection information from a digital pen connected through a communication module of the electronic device. The button selection information may include selection and release timings of buttons of the digital pen.

The method of processing an in-air gesture in an electronic device according to various embodiments of the present document includes an operation of connecting with the digital pen through a short-range communication connection, and selecting the in-air gesture and the button from the digital pen through the short-distance communication The method may further include receiving information.

In operation 1420, the electronic device may determine a gesture type based on the received in-air gesture and button selection information. For example, the type of the gesture input may include handwriting, sketch, figure, diagram, or formula.

In operation 1430, the electronic device may recognize the in-air gesture according to the determined gesture type. For example, the recognized in-air gesture may be output through various output devices (eg, a display module) of the electronic device.

In the method for processing an in-air gesture in an electronic device according to various embodiments of the present disclosure, the operation of recognizing a gesture input in response to the selection and release timings in the received in-air gesture and the type of the gesture input are In the case of handwriting, the method may further include outputting text data based on a gesture corresponding to the selection and release timings.

According to various embodiments of the present disclosure, a method of processing an in-air gesture in an electronic device includes an operation of receiving sensor data sensed through an acceleration sensor or a gyro sensor of the digital pen from the digital pen, and the received sensor data The method may further include an operation of recognizing the in-air gesture based on

In the method of processing an in-air gesture in an electronic device according to various embodiments of the present disclosure, the received in-air gesture includes at least two gestures input according to at least two types of gestures, and the at least two gestures The in-air gesture processing method in an electronic device, further comprising an operation of processing each of the gestures based on a corresponding gesture type and an operation of displaying the at least two gestures processed according to the gesture type.

15 is a flowchart illustrating operations of the digital pen and the electronic device when the digital pen is removed from the electronic device according to various embodiments of the present disclosure;

Referring to FIG. 15 , in operation 1521 , the digital pen 201 may be removed from the storage space of the electronic device 101 . For example, the user may take out the digital pen 201 from the storage space of the electronic device 101 .

In operation 1523 , the electronic device 101 may detect removal of the digital pen 201 . For example, the electronic device 101 may detect the detachment of the digital pen 201 based on non-reception of a signal from the digital pen 201 , but there is no limitation on a detachment detection method.

In operation 1525 , the digital pen 201 may detect removal of the digital pen 201 . For example, the digital pen 201 may detect the detachment of the digital pen 201 based on non-reception of a signal from the electronic device 101 , but there is no limitation on a detachment detection method. When detachment is detected, the digital pen 201 may exchange parameters (eg, a connection interval and/or a slave latency) with the electronic device 101 .

According to various embodiments, in operation 1527 , the digital pen 201 may activate the acceleration sensor based on the detachment detection.

The digital pen 201 may sense acceleration information of the digital pen 201 in operation 1529 through the activated acceleration sensor. Although not shown, the digital pen 201 may transmit sensed acceleration information to the electronic device 101 . In various embodiments, the electronic device 101 may perform an operation based on the received acceleration information. In various embodiments, the digital pen 201 may be set to activate an acceleration sensor, and a gyro sensor that consumes relatively high power may be set to remain in an inactive state.

According to various embodiments, the digital pen 201 may confirm an input of a button (eg, the button 337 ) in operation 1531 . The button may be implemented in any one of a physical key, a touch key, a motion key, a pressure key, and a key less method, and there is no limitation in the implementation form.

When the button input is confirmed, the digital pen 201 may activate the gyro sensor in operation 1533 .

The digital pen 201 may sense rotation information through the activated gyro sensor in operation 1535 .

In operation 1537 , the digital pen 201 may transmit information based on the sensing result. For example, the digital pen 201 may transmit sensing information acquired through the acceleration sensor and the gyro sensor to the electronic device 101 . Alternatively, the digital pen 201 may check the coordinates (eg, two-dimensional coordinates or three-dimensional coordinates) of the digital pen 201 based on the sensing information acquired through the acceleration sensor and the gyro sensor, and the confirmed The coordinates may be transmitted to the electronic device 101 . Alternatively, the digital pen 201 may check displacement information of the coordinates (eg, two-dimensional coordinates or three-dimensional coordinates) of the digital pen 201 based on the sensing information obtained through the acceleration sensor and the gyro sensor, , the confirmed displacement information may be transmitted to the electronic device 101 .

In operation 1539, the electronic device 101 may perform an operation based on the received information. When sensing information is received, the electronic device 101 may check location information of the digital pen 201 based on the sensing information, check a gesture corresponding to the location information, and perform an operation corresponding to the gesture. there is. When receiving the location information of the digital pen 201 , the electronic device 101 may identify a gesture corresponding to the location information and perform an operation corresponding to the gesture. For example, the digital pen 201 may transmit information to the electronic device 101 until the input of the pen button is released. The electronic device 101 may identify the gesture based on the position information of the digital pen 201 that is confirmed until the release of the button input is detected. When the release of the button input is detected, the digital pen 201 may deactivate the gyro sensor again. However, in various embodiments, the digital pen 201 may activate both the gyro sensor and the acceleration sensor from the time of detection of detachment. In this case, the position information of the digital pen 201 before the button input may be used to correct the gesture direction, and gesture recognition accuracy may be improved. For example, the electronic device 101 may check the initial posture information of the digital pen 201 and recognize the gesture using a displacement based on this.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C" each may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the element from other elements in question, and may refer to elements in other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively". When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, a processor (eg, processor 120 ) of a device (eg, electronic device 101 ) may call at least one command among one or more commands stored from a storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not include a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly between smartphones (eg: smartphones) and online. In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

Claims

In an electronic device,

communication module;

at least one processor;

display module; and

memory; including;

The memory, when executed, causes the processor to:

receiving an in-air gesture and button selection information from a digital pen connected through the communication module;

determining a gesture type based on the received in-air gesture and button selection information;

Recognizing the in-air gesture according to the determined gesture type,

and storing instructions for controlling the display module to output the recognized in-air gesture.
According to claim 1,

The type of the gesture input includes handwriting, sketch, figure, diagram, or formula.
According to claim 1,

The button selection information includes selection and release timings of buttons of the digital pen,

The instructions allow the processor to

and to recognize a gesture input in response to the selection and release timings from the received in-air gesture.
4. The method of claim 3,

The instructions allow the processor to

and outputting text data based on a gesture corresponding to the selection and release time points when the type of the gesture input is the handwriting.
According to claim 1,

The instructions allow the processor to

receiving, from the digital pen, sensor data sensed through an acceleration sensor or a gyro sensor of the digital pen;

and to recognize the in-air gesture based on the received sensor data.
According to claim 1,

The received in-air gesture includes at least two gestures input according to at least two types of gestures,

The instructions allow the processor to

processing each of the at least two gestures based on a corresponding gesture type;

An electronic device configured to display the at least two gestures processed according to the gesture type.
According to claim 1,

The instructions allow the processor to

controlling the communication module to be connected to the digital pen through a short-distance communication connection;

and controlling to receive the in-air gesture and the button selection information from the digital pen through the short-distance communication.
According to claim 1,

The instructions allow the processor to

An electronic device that determines the classification of the gesture by using a machine learning technique.
According to claim 1,

The instructions allow the processor to

extracting two-dimensional characteristic information and three-dimensional characteristic information from the received in-air gesture;

An electronic device configured to determine the type of gesture based on the extracted 2D characteristic information and 3D characteristic information.
A method of processing an in-air gesture in an electronic device, the method comprising:

receiving in-air gesture and button selection information from a digital pen connected through a communication module of the electronic device;

determining a gesture type based on the received in-air gesture and button selection information;

recognizing the in-air gesture according to the determined gesture type; and

and outputting the recognized in-air gesture.
11. The method of claim 10,

The type of the gesture input includes handwriting, sketch, figure, diagram, or formula, in-air gesture processing method in an electronic device.
11. The method of claim 10,

The button selection information includes selection and release timings of buttons of the digital pen,

and recognizing an input gesture in response to the selection and release timings from the received in-air gesture.
12. The method of claim 11,

and outputting text data based on a gesture corresponding to the selection and release time points when the type of the gesture input is the handwriting.
11. The method of claim 10,

receiving, from the digital pen, sensor data sensed through an acceleration sensor or a gyro sensor of the digital pen; and

Recognizing the in-air gesture based on the received sensor data; further comprising: an in-air gesture processing method in an electronic device.
11. The method of claim 10,

The received in-air gesture includes at least two gestures input according to at least two types of gestures,

processing each of the at least two gestures based on a corresponding gesture type; and

and displaying the at least two gestures processed according to the gesture type.