KR20230030472A - Method for creating user defined gesture profile based on user's repetitive motion and recognizing gesture - Google Patents

Abstract

The present invention relates to an electronic device paired with a host device to transmit a control signal based on a motion of a user. The electronic device includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, a memory, and a first processor for processing data obtained from the acceleration sensor, the gyro sensor, and the geomagnetic sensor to obtain acceleration, angular velocity, and inclination values for at least one of 9 axes, and uses the acceleration, angular velocity, and inclination values for at least one of the 9 axes to create and recognize a gesture profile for a specific motion.

Description

Method for creating custom gesture profile and recognizing gesture based on user's repetitive motion

The present invention relates to a gesture recognition method.

Gesture or motion recognition technology is the growth of wearable devices and the content market based on virtual reality such as computer graphics technology, augmented reality (AR), virtual reality (VR), and mixed reality (MR). Thanks to development, etc., it is continuously evolving. However, since gesture recognition technology is mainly used to create virtual contents of games or computer graphics-based contents in movie production companies rather than to control devices, gesture recognition technology is being used very limitedly in everyday life.

For this reason, firstly, it is difficult to use in the field where errors in gesture recognition technology can cause fatal results, and secondly, for gesture recognition, an image of an object is obtained from a camera and based on image processing Gesture recognition methods are widely used, but since the image processing method is greatly influenced by the environment, there are problems in that recognition errors are frequent and the algorithm needs to be very sophisticated. Third, it is possible to increase the accuracy of gesture recognition by having the user attach a landmark or wear special equipment. This method is too cumbersome and inconvenient for the user, which is why gesture recognition is not used in daily life or daily work. .

In addition, although there are conventional techniques for obtaining motion data through a motion sensor and performing gesture recognition based thereon, most of them are techniques for performing gesture recognition by using only one or two types of data such as acceleration data and processing them. Recently, 9-axis sensors including accelerometer, gyro, and geomagnetic sensors have been widely and cheaply distributed, but by mounting them on wearable devices and recognizing gestures based on the acquired sensor data or creating a gesture profile specialized for a specific user, the gesture recognition rate is increased. There is no technology for

A method for generating a gesture profile specialized for a user through acquisition of motion data in a 3D space and recognizing a specific gesture based on the generated gesture profile may be provided.

The technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems can be inferred from the following embodiments.

An electronic device that is paired with a host device and transmits a control signal based on a user's motion includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, a memory, and a first axis in a left and right direction, forward and backward, based on acceleration data received from the acceleration sensor. Acceleration values for each of the second axis in the direction and the third axis in the up and down direction are acquired, and the yaw axis, pitch axis, and roll axis are obtained based on the gyro data received from the gyro sensor. Each angular velocity value is obtained, and the yaw axis and pitch of the electronic device are obtained based on the acceleration data, gyro data, and geomagnetic data obtained from the acceleration sensor, the gyro sensor, and the geomagnetic sensor, respectively. ) axis, a first processor for obtaining inclination values between each of the roll axis directions and a horizontal plane, obtaining the acceleration value, the angular velocity value, and the inclination value received from the first processor in real time, wherein the If the inclination value for the pitch axis and the inclination value for the roll axis are within the standard, it is determined that the motion is ready, and if the angular velocity with respect to the pitch axis is greater than or equal to the reference value, it is determined that the motion has started, but the inclination value of the motion with respect to the pitch axis A second processor that handles exceptions outside of this standard range may be included.

Under the gesture recording mode, when it is determined that the motion is complete, the second processor calculates a reference acceleration value obtained by cumulatively summing linear acceleration values from the start of the motion to the end of the motion and the maximum angular velocity value for the roll axis from the start to the end of the motion. A reference rotation amount obtained by dividing the maximum angular velocity value on the pitch axis and a reference interference amount obtained by dividing the maximum angular velocity value on the yaw axis from the start of the motion to the end by the maximum angular velocity value on the pitch axis can be obtained and stored in the memory.

When the second processor determines that the motion is complete under the gesture recognition mode, a value obtained by accumulative sum of linear acceleration values from the start of the motion to the end is similar to the reference acceleration value within a reference range, and , The rotation amount, which is a value obtained by dividing the maximum angular velocity for the roll axis from the start of the motion to the end by the maximum angular velocity for the pitch axis, is similar to the reference rotation amount within a reference range, and the maximum angular velocity for the yaw axis from the start of the motion to the end If the interference amount, which is a value obtained by dividing the value by the maximum angular velocity value with respect to the pitch axis, is similar to the reference interference amount within a reference range, the motion of the user may be matched with the motion.

A method for generating a gesture profile specialized for a user through motion data acquisition in a 3D space and recognizing a specific gesture of the user based on the generated gesture profile may be provided.

The method improves the accuracy of gesture recognition and enables easy and precise control of various contents on smartphones, TVs, computers, tablet PCs, holograms, and head mount displays (HMDs).

1A illustrates an electronic device (closed state) for controlling a host device, according to an exemplary embodiment.
1B illustrates an electronic device (open state) for controlling a host device, according to an exemplary embodiment.
2A shows a ring-shaped device in an electronic device, according to an embodiment.
2B illustrates controlling content using an electronic device, according to an embodiment.
3 shows a system including an electronic device, according to one embodiment.
4A illustrates mouse operations performed by an electronic device under a mouse mode, according to an embodiment.
4B shows that the front part of the electronic device is divided into three touch areas, according to an embodiment.
4C shows mouse motions recognized using three regions, according to an exemplary embodiment.
5 illustrates a decision model for determining motion information corresponding to motion information of a user, according to an exemplary embodiment.
6 illustrates a joystick utilizing an electronic device, according to an embodiment.
7 illustrates motions for controlling a host device with an electronic device under a gesture mode, according to an embodiment.
8 is a flowchart of a method of controlling content using an electronic device, according to an embodiment.
9 shows a flow diagram of a method for obtaining distance data, according to one embodiment.
10 shows a block diagram of an electronic device, according to an embodiment.
11 shows nine axes on a three-dimensional space, according to one embodiment.
12 illustrates a state in which a user creates a gesture profile by repeating a specific motion according to a guide according to an embodiment.
13 illustrates a flow diagram of a method for generating a gesture profile of a left or right move action, according to one embodiment.
Fig. 14 is a flowchart of a method for recognizing whether a current motion of a user matches a gesture profile of a left or right move operation, according to an embodiment.

In the following, several embodiments will be described clearly and in detail with reference to the accompanying drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily practice the present invention. will be.

Also, the term “unit” or “module” used in the specification may mean a hardware component or circuit such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC).

Hereinafter, “content” may include media itself or objects reproduced on the media, such as games, music, movies, images, animations, characters, items, and objects, but is not limited thereto. “Content” may include an operating system or software running on a host device. Software may include word processors, document programs such as PowerPoint, image processing programs for performing specialized tasks, CAD programs, and games. “Content” may include virtual content generated in virtual reality such as AR/VR/MR. "Content" may include an object reproduced on a 2D screen or a 3D stereoscopic object expressed on a 3D space such as a hologram. "Content" may be created, executed or reproduced by a host device. When the “content” is virtual content (eg, hologram) expressed in a 3D space, the physical location of the host device and the “content” may be different.

Hereinafter, “motion” is a meaningful movement taken by a user to control content, and may be captured, extracted, recognized, analyzed, or determined from the user's movement.

Hereinafter, a “control signal” is a signal including information about a motion itself or a motion type, and an electronic device generates a “control signal” and a host device operates content based on the “control signal” received from the electronic device. can be made or controlled. For example, the “control signal” may be in the form of a bit string, and each of the motions may be represented by a different bit string.

1A illustrates an electronic device (closed state) for controlling a host device, according to an exemplary embodiment. 1B illustrates an electronic device (open state) for controlling a host device, according to an exemplary embodiment.

The user may control the host device by holding the electronic device 1000 in his hand or by touching or moving his hand while holding the electronic device 1000 in his hand. The host device may include various types of electronic devices. For example, the host device may be a game machine, a smart phone, a tablet PC (Personal Computer), a TV, a desktop PC, a notebook PC, a mobile medical device, a camera, or a wearable device (eg, electronic glasses, electronic clothes, electronic bracelets) , electronic necklace, electronic accessory, electronic tattoo, or smart watch), but is not limited thereto. For example, the host device may include a Head Mounted Display (HMD) for displaying virtual content and a game device (eg, a console device) for executing or playing a virtual reality game or virtual reality content. The host device may include a computer for displaying presentation material.

Referring to FIGS. 1A and 1B , the electronic device 1000 may include a ring-shaped device 1200 wearable on a user's finger and a cradle device 1400 for accommodating the ring-shaped device 1200 . A user may place the ring-shaped device 1200 in the cradle device 1400 and close the lid.

When the ring-shaped device 1200 is housed in the cradle device 1400 and the lid of the cradle device 1400 is closed, the front part (head part) of the ring-shaped device 1200 is exposed to the front of the cradle device 1400, The user may generate a motion signal by moving the cradle device 1400 while holding it in his/her hand, or may perform a touch operation by touching the exposed front portion of the ring-shaped device 1200 . According to an embodiment, a touch sensing module may be located on the front portion of the ring-shaped device 1200 .

The user can take out the ring-shaped device 1200 by opening the lid of the cradle device 1400 . A user may control the host device by a touch operation or motion while wearing the ring-shaped device 1200 on a finger. The cradle device 1400 may be manufactured in a shape that is easy for a person to hold, and the center of gravity of the cradle device 1400 may be designed to be low by placing a center of gravity weight below the cradle device 1400 . The cradle device 1400 may include a charging terminal for charging the ring-shaped device 1200 and a power supply.

The ring-shaped device 1200 may include a motion sensor for obtaining user motion information and a touch sensor for obtaining user touch information. The ring-shaped device 1200 may generate a control signal based on the acquired motion information and touch information and output the generated control signal to the host device. The host device may control content based on a control signal received from the ring-type device 1200 .

Figure 2a shows a ring-shaped device, according to an embodiment, and Figure 2b shows a user controlling content while wearing a ring-shaped device, according to an embodiment.

Referring to FIGS. 2A and 2B , the ring-shaped device 1200 may be a small wearable device that can be attached to, connected to, or worn with a human body or an object. The ring-shaped device 1200 is comfortable to wear, and the user can intuitively operate the functions of the ring-shaped device 1200 without additional learning. Furthermore, the ring-shaped device 1200 can be used as a general-purpose device like a general-purpose mouse by using motion information and touch information.

The ring-shaped device 1200 may include a connection unit 1220 to be worn on a user's finger 2300 and a main module 1240 to obtain motion information and touch information using a sensor. The connection unit 1220 may be made of a material such as silicon or metal. The main module 1240 may obtain user touch information and motion information and output a control signal corresponding to the obtained information. The main module 1240 may include components of the electronic device 3000 to be described later and a case containing the components. According to an embodiment, the main module 1240 may be separated from the connection unit 1220, and the user inserts, attaches, or embeds only the main module 1240 into various types of objects (sticks, dice, pens, etc.) Various types of objects can be used to control the host device.

The main module 1240 acquires touch information and motion information (eg, angular velocity, acceleration, speed, distance, angle, direction, and location (3D space coordinates) information for the movement of the main module 1240) and obtains them. By processing and processing, a control signal for controlling the content 2500 may be output.

Although the ring-type device 1200 is shown to be worn on the user's finger 2300 in FIG. 2B, the ring-type device 1200 may be connected to or attached to other types of objects.

For example, the main module 1240 may be embedded in a dice and the contents 2500 may be controlled based on the movement of the dice. Alternatively, the main module 1240 may be attached to the wand and the content 2500 may be controlled based on the movement of the wand. Alternatively, the main module 1240 may be built into the pen and the contents 2500 on the smart phone may be controlled based on the movement of the pen. Hereinafter, an object may mean a body part (eg, a finger) of a person, an object that a person can wear or hold, or an electronic device 3000 itself to be described later.

In addition, although the content 2500 is shown as a hologram object in a 3D space in FIG. 2B, the content 2500 may include any type of content or software (Microsoft's MS Office, games, etc.) played on the host device. can

3 shows a system including an electronic device, according to one embodiment.

Referring to FIG. 3 , the system 100 may include an electronic device 3000 and a host device (or target device). The electronic device 3000 may be connected to the host device through a wireless communication method. For example, the electronic device 3000 may be paired with a host device using a Bluetooth method. The electronic device 3000 includes the cradle device 1400 in which the ring-shaped device 1200 of FIG. 1A is stored, the ring-shaped device 1200 separated from the cradle device 1400, or the main module of the ring-shaped device 1200 of FIG. 2A ( 1240).

A user may control various contents of the host device using the electronic device 3000 . According to an embodiment, a user may control contents of a host device based on a motion of the electronic device 3000 itself or an object connected to the electronic device 3000 and/or a user touch motion input to the electronic device 3000. there is. For example, the user may control various contents of the host device by wearing the electronic device 3000 on his/her finger and moving the electronic device 3000 or by touching the electronic device 3000 with the finger.

Referring to FIG. 3 , an electronic device 3000 may include a motion sensing module 3200, a touch sensing module 3300, a communication channel 3400, and a control signal output unit 3600.

The electronic device 3000 may operate in a mouse mode or a gesture mode. Under the mouse mode, the electronic device 3000 can operate like a general-purpose mouse, and under the gesture mode, the electronic device 3000 can operate as a motion recognition device.

Under the mouse mode, the electronic device 3000 determines a mouse motion based on at least one of a touch motion detected through the touch sensing module 3300 and motion information detected through the motion sensing module 3200, and indicates the mouse motion. Mouse signals can be output.

For example, a mouse click is performed by touching one surface of the electronic device 3000 once, and a mouse double click is performed by touching one surface of the electronic device 3000 twice within a reference time. The mouse move is determined from motion information (eg, second motion information described below) of the electronic device 3000, and the mouse scroll (Scroll Up/Down) is a continuous change of touch (eg, , Scroll up is a movement of writing one side of the electronic device 3000 with a finger from left to right, scroll down is a movement of writing one side of the electronic device 3000 with a finger from right to left), and mouse drag is a touch time ( For example, long touch) and motion information of the electronic device 3000. According to an embodiment, if the touch time is less than 200 ms, it may be determined as a short touch, and if the touch time is greater than 500 ms, it may be determined as a long touch. One side of the electronic device 3000 on which a touch is performed may be the touch sensing module 3300, which may be the front portion described with reference to FIG. 2A.

Under the gesture mode, the electronic device 3000 may obtain motion information of the electronic device 3000 using a sensor and determine a motion corresponding to the motion of the electronic device 3000 based on the acquired motion information. The electronic device 3000 may output a motion signal representing the determined motion to the host device. According to an embodiment, the motion information may include characteristics (eg, at least one of angular velocity, acceleration, velocity, distance, angle, direction, and position) of the object's movement.

Hereinafter, operations and functions of components of the electronic device 3000 will be described.

Referring to FIG. 3 , the touch sensing module 3300 may detect a user's touch. For example, when the user touches the front portion of the ring-type device 1200 with a thumb while the user wears the ring-type device 1200 on the index finger, the touch sensing module 3300 may detect the touch operation. there is. A touch operation detected by the touch sensing module 3300 may be transferred to the control signal output unit 3600 through the communication channel 3400 .

A touch motion detected by the touch sensing module 3300 may be used to determine a mouse motion under the above-described mouse mode. Alternatively, the touch operation detected by the touch sensing module 3300 may be used to determine a motion corresponding to the motion of an object under the gesture mode. Also, a touch operation detected by the touch sensing module 3300 may be used to switch between a mouse mode and a gesture mode.

Switching between the mouse mode and the gesture mode may be performed by a touch operation. For example, when a user's short touch, short touch, and long touch are continuously detected through the touch sensing module 3300, the gesture mode may be switched to the mouse mode or the mouse mode to the gesture mode. Alternatively, when the central portion of the front portion of the ring-shaped device 1200 is touched through the touch sensing module 3300 for a reference period of time or longer, the gesture mode may be switched to the mouse mode or the mouse mode to the gesture mode.

The motion sensing module 3200 may obtain first motion information of the electronic device 3000 . The first motion information can be used in both mouse mode and gesture mode. The first motion information may include at least one of acceleration data obtained through an acceleration sensor and angular velocity data obtained through a gyro sensor.

The motion sensing module 3200 may include an accelerometer 3220, a gyroscope 3240, a magnetometer 3260, and a sensor fusion unit 3280. The gyro sensor 3240 according to an embodiment is a sensor for measuring an angular velocity. The acceleration sensor 3220 according to an embodiment is a sensor for measuring acceleration and dynamic forces such as vibration and shock. The geomagnetic sensor 3260 according to an embodiment is a sensor for measuring Earth's magnetism and detecting its magnitude.

An error occurs in the value measured by the gyro sensor 3240 due to the influence of temperature, and a phenomenon in which the final value drifts may occur as the error is accumulated during the integration process. Therefore, it is necessary to compensate the error of the gyro sensor 3240 by using the temperature sensor together.

From the point of view of a long time in a stationary state, the tilt angle calculated by the acceleration sensor 3220 indicates a correct value, but the gyro sensor 3240 may indicate an incorrect value due to cumulative drift over time. Conversely, in terms of a short moving time, the gyro sensor 3240 indicates the correct angular velocity, but the acceleration sensor 3220 may derive a calculated value different from the inclination angle. In addition, inclination measurement is impossible when the subject moves in a straight direction from a stationary state.

Accordingly, a filter such as a Kalman filter or a compensation and fusion algorithm may be applied in order to compensate for and compensate for each disadvantage by using both the acceleration sensor 3220 and the gyro sensor 3240 . However, despite these compensation and convergence operations, when only the gyro sensor 3240 and the acceleration sensor 3220 are used, the error rate increases when calculating coordinates in a 3D space. Inappropriate for use as an interface. In addition, when only the acceleration sensor 3220 and the gyro sensor 3240 are used, it is difficult to determine the absolute position of a moving subject because a relative azimuth angle is used instead of an absolute azimuth angle.

Therefore, the motion sensing module 3200 further includes a geomagnetic sensor 3260, so that the change in absolute azimuth measured by the geomagnetic sensor 3260 is recorded together with the data measured by the acceleration sensor 3220 and the gyro sensor 3240. By calculating, data with a low error rate can be generated. The motion sensing module 3200 includes the geomagnetic sensor 3260, so that the cumulative drift generated by the gyro sensor 3240 can be more perfectly compensated. It can play a role in supplementing and compensating for each other's shortcomings by resolving the momentary magnetic field bouncing phenomenon (suddenly large change in magnetic field) caused by

According to an embodiment, the motion sensing module 3200 may include a 9-axis sensor capable of accurately acquiring location data in a 3D space. The 9-axis sensor is a sensor composed of 3 axes of acceleration, 2 axes of gyro, 3 axes of geomagnetism, and 1 axis of temperature.

Referring to FIG. 3 , the motion sensing module 3200 is a sensor fusion unit for performing a sensor fusion operation in which outputs of sensors 3220, 3240, and 3260 are compensated and fused to generate optimized position data. (3280). The sensor fusion unit 3280 removes noise, compensates for, and fuses the data obtained from the acceleration sensor 3220, the gyro sensor 3240, and the geomagnetic sensor 3260, and optimizes the first motion information. can create If the row data obtained by the sensors 3220, 3240, and 3260 are used as they are, accurate position data cannot be obtained, so position estimation through a filter optimizes position data. can For example, the sensor fusion operation may be performed based on a filter such as a Kalman filter or a data compensation and fusion algorithm.

The first motion information obtained through the motion sensing module 3200 may be transferred to the control signal output unit 3600 through the communication channel 3400 . According to an embodiment, the communication channel 3400 may be an internal bus within the electronic device 3000 for transferring first motion information to the processor 3620 . The motion sensing module 3200 and the control signal output unit 3600 may exchange data with each other based on the bus format of the communication channel 3400 . For example, the bus format may include one or more of various interface protocols such as Universal Serial Bus (USB), Serial Peripheral Interface (SPI), and Inter-Integrated Circuit (I2C).

The control signal output unit 3600 may output a control signal for controlling the host device. The control signal may include a motion signal and a mouse signal. The control signal output unit 3600 may obtain second motion information through calculation of the first motion information. The control signal output unit 3600 may determine a motion corresponding to the motion of the electronic device 3000 based on the second motion information under the gesture mode and output a motion signal representing the determined motion. The control signal output unit 3600 may determine a mouse operation based on at least one of touch information and second motion information obtained from the touch sensing module 3300 and output a mouse signal indicating the mouse operation under the mouse mode. . The control signal may be an interrupt signal for controlling contents of the host device. For example, the control signal may include a bit string representing a specific mouse signal or a specific motion signal.

The control signal output unit 3600 may generate second motion information by calculating first motion information received through the communication channel 3400 . The second motion information may include at least one of angle data, distance data, speed data, and direction data of the electronic device 3000 . The second motion information of the electronic device 3000 can be utilized in both the mouse mode and the gesture mode. For example, under the mouse mode, the second motion information may be used to determine a mouse move operation of the electronic device 3000 . Under the gesture mode, the second motion information may be used to determine various motion signals output from the electronic device 3000 .

According to an embodiment, the control signal output unit 3600 may include a processor 3620 and a communication unit 3640.

The processor 3620 may generate second motion information by calculating first motion information received from the motion sensing module 3200 through the communication channel 3400 . The second motion information may include at least one of angle data, distance data, speed data, and direction data for motion. The processor 3620 may obtain the second motion information by performing an operation on the first motion information every reference time (eg, 5 ms). The reference time may be 30 ms or less, but is not limited thereto.

The angle data may include angle data for each of the x-axis direction, the y-axis direction, and the z-axis direction. According to an embodiment, the processor 3620 may obtain angular data by performing an integration operation on angular velocity data.

The speed data may include speed data for each of the x-axis direction, the y-axis direction, and the z-axis direction. The distance data may include distance data for each of the x-axis direction, the y-axis direction, and the z-axis direction. According to an embodiment, the processor 3620 may obtain speed data and distance data by performing an integration operation on acceleration data. The processor 3620 may obtain linear acceleration data by removing the gravitational acceleration component from the acceleration data. The processor 3620 may obtain velocity data by performing an integration operation on linear acceleration data, and obtain distance data by performing an integration operation on the velocity data again.

The direction data relates to the instantaneous movement direction of the object, and may include whether or not there is an increase or decrease in the x-axis direction, whether or not there is an increase or decrease in the y-axis direction, and whether or not there is an increase or decrease in the z-axis direction. According to an embodiment, the processor 3620 may include direction data based on comparison between current distance data and previous distance data. For example, the x-axis value of the current distance data is +50, the y-axis value is +10, and the z-axis value is -5, and the x-axis value of the previous distance data is +60 and the y-axis value is +60. If the value of 15 or the z-axis direction is -10, the processor 3620 may determine the current movement direction as an increase in the x-axis direction, an increase in the y-axis direction, and a decrease in the z-axis direction.

Under the mouse mode, the processor 3620 may determine a corresponding mouse motion based on touch information and second motion information obtained from the touch sensing module 3300 . 4A illustrates mouse operations performed by the electronic device 3000 under a mouse mode, according to an embodiment. The mouse operation may include mouse click, zoom in/out (or scroll up/down), mouse movement, and mouse drag. A mouse click may include a single click, a double click, and a long click. Mouse Move can move the mouse pointer of the host device.

Referring to FIG. 4B, for a mouse operation, the surface of the touch sensing module 3300 of the electronic device 3000 (the front of the main module 1240 in FIG. 2A) has a touch area R1 located on the left and a touch area R1 located in the center. It can be divided into a touch area R2 and a right touch area R3. Referring to FIG. 4C , when a user touches only the touch area R1 or touches both the touch area R1 and the touch area R2, the corresponding operation may be determined as a left click. When the user touches only the touch area R3 or touches the touch area R3 and the touch area R2 together, the corresponding operation may be determined as a right click. When the user touches only the touch area R2 or touches the touch area R2, the touch area R1, and the touch area R3 together, the corresponding operation may be determined as a mode change between mouse mode and gesture mode. there is. When the user sequentially and continuously touches the touch area R1 , the touch area R2 , and the touch area R3 , the corresponding operation may be determined as scrolling up. When the user sequentially and continuously touches the touch area R3, the touch area R2, and the touch area R3, the corresponding operation may be determined as scrolling down.

Under the mouse mode, the processor 3620 distinguishes between when the user is using the keyboard and when the user is using the mouse, and when it is determined that the user is using the keyboard, the processor 3620 may not output a mouse signal.

Under the gesture mode, the processor 3620 may determine a motion corresponding to the motion of the electronic device 3000 based on the second motion information. For example, the processor 3620 may determine one motion corresponding to the motion of the electronic device 3000 from among predefined motions based on the second motion information. The processor 3620 may generate a motion signal representing the determined motion and transmit it to the host device through the communication unit 3640. If the distance between the electronic device 3000 and the host device is greater than the reference distance, the processor 3620 determines that the motion of the electronic device 3000 does not correspond to any of the predefined motions or is meaningless. In this case, an exception can be made.

Predefined motions may include move, tap, grab, scroll, swipe, gesture, rotation, and the like. The move motion is an operation of moving the electronic device 3000 in an arbitrary direction, and may be used for an operation of moving virtual content or turning a page. For example, the move motion may include movement in three axis directions (x, y, and z axes). A tap motion is a motion of tapping something, and may be used to select or click virtual content. The user may double-click the virtual content by making a tap motion twice consecutively within a reference time. A tap motion is a motion separate from a click motion under a mouse mode.) A grab motion is a motion in which two distant objects come into contact with each other, and may be used to grab virtual content. A gesture may mean a movement for expressing text, a symbol, or a shape (eg, '?' or 'X').

Pre-defined motions can be added by user-defined motions. A user-defined motion is a motion defined by a user, not a manufacturer of the electronic device 3000, and the user can add a specific motion input by the user as a user-defined motion. For example, the user may repeatedly take a specific motion while holding or wearing the electronic device 3000 and match it with a specific function or motion. In the electronic device 3000, motion information repeatedly taken by a user and functions or motions corresponding to the motion information may be stored in correspondence with each other. For example, a user who wears the electronic device 3000 on his or her finger may make 10 stabbing motions, designate them as stabbing motions, and save the motion. Then, when the user wears the electronic device 3000 and performs a stabbing motion, the processor 3620 may transmit a motion signal representing the stabbing motion to the host device through the communication unit 3640 .

Motion information of the electronic device 3000 may be matched with a specific motion based on machine learning. That is, when specific motion information is input to a decision model learned through machine learning, the decision model may output a motion signal corresponding to the input specific motion information.

According to an embodiment, the electronic device 3000 may use an independent decision model for each user. This is because generated motion information is different for each user even if the users perform the same motion. For example, assuming that an operation of drawing a circle by turning an arm corresponds to a specific motion signal output from the electronic device 3000 or a specific function of the host device, all motion information generated while performing the above operation for each user This is because it is not the same and may have a unique pattern for each user.

Referring to FIG. 5, the first decision model DEC#1, the second decision model DEC#2, and the third decision model DEC#3 are motion information of the first user and motion information of the second user, respectively. , can be used to output a motion signal corresponding to the motion information of the third user. If the user currently using the electronic device 3000 is the first user, the control signal output unit 3600 or the processor 3620 converts the acquired motion information of the first user (eg, the above-described second motion information) to the first user. Input to the first decision model DEC#1 and a corresponding motion signal may be determined. If the user currently using the electronic device 3000 is the third user, the control signal output unit 3600 or the processor 3620 converts the obtained third user's motion information (eg, the aforementioned second motion information) to the third user. Input to the third decision model (DEC#3) and corresponding motion can be determined.

A decision model for determining the motion signal may be created based on machine learning. For example, the first decision model DEC#1 is generated by performing machine learning that repeatedly applies the first user's motion information (eg, the above-described second motion information) and a specific motion as inputs and outputs, respectively. It can be. The first decision model (DEC#1) receives the second motion information of the first user generated from the first user's motion of drawing a circle by rotating his arm 10 or more times, and makes the input motion information correspond to a specific motion signal. can be learned For example, the second decision model DEC#2 is generated by performing machine learning that repeatedly applies the second user's motion information (eg, the above-described second motion information) and a specific motion as inputs and outputs, respectively. It can be. The second decision model (DEC#2) receives the second user's second motion information generated from the second user's motion of drawing a circle by rotating his arm 10 or more times, and makes the input motion information correspond to a specific motion signal. can be learned

Machine learning techniques include Support Vector Machine (SVM), Random forest, Support Vector Machine (SVM), Naive Bayes, Adaptive Boosting (AdaBoost), Random Forest ( Random Forest), gradient boosting, K-means clustering, artificial neural network, and the like.

The machine learning-based decision model used to determine the motion signal may be stored in a memory (not shown) in the electronic device 3000 or stored in the host device. Also, learning to generate a decision model may be performed in the electronic device 3000 or in the host device. According to an embodiment, learning to generate a decision model may be performed in a host device, and the generated decision model may be stored in a memory (not shown) in the electronic device 3000 . Alternatively, learning of the decision model may be performed in the electronic device 3000 and the decision model may be stored in a memory (not shown) of the electronic device 3000 .

7 is a left move motion, a right move motion, an up move motion, and a down move motion for controlling a host device with the electronic device 3000 under a gesture mode, according to an embodiment. It represents Down Move motion, rotation (Clockwise Rotation, Counter Clockwise Rotation) motion, and Forward/Back Move motion. The left move motion and the right move motion may be determined from the movement in the +/- direction of the x-axis. The upper move motion and the lower move motion may be determined from the movement in the z-axis +/- direction. The forward move motion and the back move motion may be determined from motion in the +/- direction of the y-axis.

In this embodiment, the user may turn the page of a word document being executed in the host device by making a left move or a right move motion in a 3D space while wearing the electronic device 3000 . Under the gesture mode, the motion for the electronic device 3000 to control the host device is not limited to the above-described embodiment. For example, motions supported by the electronic device 3000 may further include tap, grab, scroll, and swipe. In addition, motions (including user-defined motions) under the gesture mode may be added and used by the user through the above-described machine learning-based decision model.

Referring back to FIG. 3 , the processor 3620 may generate a control signal representing a motion signal or a mouse signal. For example, when the user's motion is determined to be a left moving motion, the processor 3620 may generate a first bit string representing a left moving motion as a control signal. When the user's movement is determined to be the clockwise rotation motion, the processor 3620 may generate a second bit string representing the clockwise rotation motion as a control signal. Alternatively, in the case of using a protocol agreed upon between the electronic device 2000 and the host device, a number assigned to each of the motions may be generated as a control signal. When the user's movement is determined as a mouse movement, the processor 3620 may generate a third bit string representing the mouse movement as a control signal.

The processor 3620 may include one processor core (Single Core) or may include a plurality of processor cores (Multi-Core). For example, the processor 3620 may include a multi-core such as a dual-core, quad-core, or hexa-core. In addition, the processor 3620 may further include an internal or external cache memory.

The communication unit 3640 may transmit a control signal to the host device through a wireless communication interface. The communication unit 3640 may include a wireless local area network (WLAN) such as Wi-Fi (Wireless Fidelity), a wireless personal area network (WPAN) such as Bluetooth, and wireless USB (Wireless Universal Serial Bus), Zigbee, NFC (Near Field Communication), RFID (Radio-frequency identification), or 3G (3rd Generation), 4G (4th Generation), LTE (Long Term Evolution), etc. It may include a modem communication interface and the like. The Bluetooth interface may support Bluetooth Low Energy (BLE).

The electronic device 3000 may include a memory (not shown) necessary for an operation performed in the electronic device 3000 . For example, the electronic device 3000 may include a memory (not shown) required to perform a sensor fusion operation in the sensor fusion unit 3280 . Also, the electronic device 3000 may include a memory (not shown) used to store predefined motions and/or user defined motions or required for an operation performed by the processor 3620 . A memory (not shown) may store a decision model generated based on machine learning to determine a motion signal corresponding to a user movement. The memory (not shown) includes a volatile memory device such as dynamic random access memory (DRAM) and static random access memory (SRAM), a flash memory device, and a solid state drive (Solid State Drive; SSD) and the like.

The electronic device 3000 may include a battery (not shown) for supplying power necessary for operations performed by the electronic device 3000 . The battery (not shown) may include, but is not limited to, a lithium ion battery or a lithium polymer battery. For example, a battery (not shown) may be included in the control signal output unit 3600, and some of the power output from the battery (not shown) may be bypassed to the motion sensing module 3200.

The electronic device 3000 may include a charging terminal for charging a battery (not shown). The electronic device 3000 may include a USB type charging terminal. Current flowing through the charging terminal may be used to charge the battery. According to an embodiment, a charging terminal may exist in the ring-shaped device 1200 of FIGS. 1A and 1B, and a charging terminal may exist in the cradle device 1400. For example, charging of the main module 1240 may be performed by having a charging terminal in the main module 1240 and receiving the ring-shaped device 1200 in the cradle device 1400 .

Hereinafter, a method of controlling content using an electronic device will be described with reference to FIGS. 8 to 14 . The method described with reference to FIGS. 8 to 14 may be performed in at least one of the electronic device 3000 of FIG. 3 and a host device. Therefore, even if omitted below, the description of the electronic device 3000 or the host device of FIG. 3 may be applied to FIGS. 8 to 14 . 8 to 14 may also be applied to the electronic device 3000 of FIG. 3 or the host device.

8 is a flowchart of a method of controlling content using an electronic device, according to an embodiment.

In step S200, the electronic device may acquire first motion information of the object based on the sensing module. An object may mean an electronic device itself. The sensing module may include an acceleration sensor, a gyro sensor, and a geomagnetic sensor. The first motion information may include acceleration data and angular velocity data for the motion of the object. For example, the first motion information may be data obtained by optimizing acceleration data obtained through an acceleration sensor and angular velocity data obtained through a gyro sensor by a sensor fusion unit.

In step S400, the electronic device may generate second motion information by calculating the first motion information obtained in step S200. The second motion information may include at least one of angle data, speed data, distance data, and direction data. The electronic device may calculate and obtain second motion information in real time while the object is moving. For example, the electronic device may obtain the second motion information by performing an operation on the first motion information every reference time (eg, 5 ms). The reference time may be 30 ms or less, but is not limited thereto. As the electronic device is worn on the middle joint of the index finger, the joint between the first joint and the middle joint of the index finger can be used as an axis to determine the moving angle and speed of the middle joint of the index finger. Also, since the electronic device is worn on the tip joint of the index finger, the angle and speed at which the tip joint of the index finger moves can be determined with the joint between the first joint and the middle joint of the index finger as an axis.

In step S500, the electronic device may determine whether the current mode is a mouse mode or a gesture mode. If the current mode is the mouse mode (Yes), the electronic device may acquire touch information in step S520 and determine a mouse signal based on at least one of the second motion information and touch information in step S540.

If the current mode of the electronic device is the gesture mode (No), in step S600, the electronic device may determine a motion signal corresponding to the motion of the object based on the acquired second motion information. According to an embodiment, the electronic device may determine a motion signal corresponding to the motion of the object. According to an embodiment, the electronic device may determine the motion based on the speed, angle, distance, etc. of the index finger. Motions may include, but are not limited to, move, tap, grab, scroll, swipe, gesture, and rotation motion. The motions may include gesture profiles (LEFT/RIGHT, FORWARD/BACKWARD, CIRCLE (CLOCKWISE/COUNTERCLOCKWISE)) that are created to suit the user's characteristics. In electronic devices, motions of objects are predefined by manufacturers or user-defined. If none of the motions are applicable or if the motion is determined to be meaningless, an exception may be processed without generating a motion signal.

In step S800, the electronic device may transmit a control signal representing the determined motion signal or mouse signal to the host device through a wireless communication interface. The control signal may be an interrupt signal for controlling the host device. According to an embodiment, the electronic device may determine whether the location of the object is within a standard distance from the location where content is reproduced, and transmit a control signal to the host device only when the location of the object is within the standard distance as a result of the determination. This is because when the user is far away from the content, it is difficult to view the user's movement as a movement for controlling the content.

In step S900, the host device may control content based on the received control signal. For example, when the received control signal is a move motion, the host device may move a baseball in a game in a direction, speed, and distance proportional to the motion of the object. The host device may select an item in the game when the received control signal is a tap motion. The host device may rotate the disc in the game when the received control signal is a rotation motion. When the received control signal is a move motion, the host device may zoom-in or zoom-out the content according to a distance between the object and the content. When the received control signal is a page turning motion (eg, LEFT MOVE, RIGHT MOVE), the host device may turn pages of the currently executing word document or presentation document. When the received control signal is a mouse click, the host device may perform a click operation at the current mouse position. The host device may perform a scroll-up operation when the received control signal is a mouse scroll-up.

9 is a flowchart of a method for obtaining, by an electronic device, distance data about motion of an object, according to an embodiment.

In step S420, the electronic device may generate linear acceleration data by removing the gravitational acceleration component from the acceleration data. Acceleration data for the motion of the object may be obtained by removing the influence of the gravitational acceleration from the acceleration data.

In step S440, the electronic device may obtain speed data by performing an integral operation on the linear acceleration data.

In step S460, the electronic device may obtain distance data by performing an integration operation on the speed data.

11 is a flowchart of a method for determining the movement of an object as a mouse move in a mouse mode or a move motion in a gesture mode, and acquiring movement distance data, according to an embodiment. FIG. 12 may represent sub-steps of step S540 or S600 of FIG. 10 .

In step S612, the electronic device may determine the angle and speed of the initial movement of the object. The electronic device may acquire angle data and velocity data for an initial movement (eg, within a reference time after the movement starts) after the movement of the object starts.

In step S614, the electronic device may determine whether the angle data and speed data obtained in step S612 satisfy a reference condition. For example, when the speed is equal to or greater than the reference value and the range of change in angle is within 20 degrees, the electronic device may determine the motion of the object as a motion in a straight direction. If the angle data and the velocity data do not satisfy the reference conditions (No), the electronic device may determine whether the motion of the object corresponds to another motion or may process an exception if it is determined that the motion of the object does not correspond to any motion (S618).

If the angle data and the velocity data satisfy the reference condition (Yes), the electronic device may determine the movement as a move motion or a mouse move and obtain distance data for the movement of the object (S616). For example, the electronic device may determine a point in time when the electronic device moves at a predetermined speed or more in a desired direction as a starting point for the movement. Distance data for the movement of the object may be determined by the method described above with reference to FIG. 11 .

According to an embodiment, the electronic device may obtain location data (eg, 3D space coordinates of an object) and direction data in addition to distance data.

The electronic device may obtain current location data of an object based on data of a distance moved by the object when the location of the object at an arbitrary time or a location in an arbitrary space is used as a reference point. The electronic device may determine a moving distance for each movement unit of the object and store the determined distance data in a memory. The electronic device may determine location data of the current object by reading distance data about motions from a memory and summing the read distance data.

For example, when the position data of an object at a previous point in time is (0, 0, 0) and the move motion occurs three times in succession, the first distance data of the first move motion is (10, -20, 30), the second distance data of the second move motion is (-10, -30, -10), and the third distance data of the third move motion is (20, 100, 100), the current object The location data of can be determined as (20, 50, 120). For example, if the first move motion occurs once, the object is simply moved as a person moves, and another move motion occurs again, the first distance data of the first move motion is (5, 30, 20), when the second distance data for simple movement is (500, 500, 0) and the second distance data for second move motion is (10, 30, 30), the position data of the current object is set to (515). , 560, 50).

The electronic device may obtain direction data in which the object is moving based on comparison between current distance data and previous distance data. Previous distance data may refer to distance data obtained immediately before. For example, if distance data is calculated every 5 ms, whether to increase or decrease in the x-axis direction, whether to increase or decrease in the y-axis direction, and the z-axis based on the distance data at time point t and the distance data obtained at time point t-5 ms You can decide whether to increase or decrease in the direction. For example, the x-axis value of the current distance data is +50, the y-axis value is +10, and the z-axis value is -5, and the x-axis value of the previous distance data is +60 and the y-axis value is +60. If the value of 15 or the z-axis direction is -10, the electronic device 3000 may determine the current movement direction as an increase in the x-axis direction, an increase in the y-axis direction, and a decrease in the z-axis direction.

10 shows a block diagram of an electronic device, according to an embodiment.

The electronic device MD of FIG. 10 may be the electronic device described above with reference to FIGS. 1A to 9 . For example, the electronic device may be a ring-shaped device. Referring to FIG. 12, the electronic device MD includes an acceleration sensor ACC, a gyro sensor GYR, a geomagnetic sensor MAG, a first processor P1, a second processor P2, and a memory M. can include

The accelerometer (ACC), gyro sensor (GYR), and geomagnetic sensor (MAG) are the same as the accelerometer (3220), gyroscope (3240), and geomagnetic sensor (Magnetometer, 3260) described above with reference to FIG. Since they are the same, detailed descriptions are omitted.

10 and 11 together, the first processor P1 may obtain acceleration values for each of the three axes (Left/Right, Forward/Back, Up/Down) based on data received from the acceleration sensor ACC. there is. The first processor P1 may obtain angular velocity values for each of three axes (Yaw/Pitch/Roll) based on data received from the gyro sensor GYR. The first processor (P1) may acquire the inclination value of the horizontal reference between the 3-axis (Yaw/Pitch/Roll module and the ground) based on the acceleration sensor (ACC), the gyro sensor (GYR), and the geomagnetic sensor (MAG). .

The second processor P2 may generate a gesture profile for the user based on the values received from the first processor P1 or determine whether a motion actually taken by the user matches one of the previously generated gesture profiles. . Predefined conditions for determining whether a gesture is a specific gesture may be referred to as a gesture profile by checking whether a motion actually taken by the user to control the host device meets the predefined conditions. For example, the gesture profile may include 8 gesture profiles (LEFT/RIGHT/UP/DOWN MOVE, FORWARD/BACKWARD MOVE, CIRCLE (CLOCKWISE/COUNTERCLOCKWISE)).

According to an embodiment, the electronic device instructs the user to repeatedly perform specific motions LEFT/RIGHT/UP/DOWN MOVE, FORWARD/BACKWARD MOVE, CIRCLE (CLOCKWISE/COUNTERCLOCKWISE) (eg, the motions of FIG. 7). The guide and electronic device may store a gesture profile based on data obtained from repeated motions of the user. According to an embodiment, a gesture profile may be generated for each of six motions including LEFT/RIGHT, FORWARD/BACKWARD, and CIRCLE (CLOCKWISE/COUNTERCLOCKWISE). 12 shows that a gesture profile is created by displaying a left move motion in the host device and following it by the user, according to an embodiment.

An electronic device that is paired with a host device and transmits a control signal based on a user's motion includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, a memory, and a first axis in a left and right direction, forward and backward, based on acceleration data received from the acceleration sensor. Acceleration values for each of the second axis in the direction and the third axis in the up and down direction are acquired, and the yaw axis, pitch axis, and roll axis are obtained based on the gyro data received from the gyro sensor. Each angular velocity value is obtained, and the yaw axis and pitch of the electronic device are obtained based on the acceleration data, gyro data, and geomagnetic data obtained from the acceleration sensor, the gyro sensor, and the geomagnetic sensor, respectively. ) axis, a first processor for obtaining inclination values between each of the roll axis directions and a horizontal plane, obtaining the acceleration value, the angular velocity value, and the inclination value received from the first processor in real time, wherein the If the inclination value for the pitch axis and the inclination value for the roll axis are within the standard, it is determined that the motion is ready, and if the angular velocity with respect to the pitch axis is greater than or equal to the reference value, it is determined that the motion has started, but the inclination value of the motion with respect to the pitch axis A second processor that handles exceptions outside of this standard range may be included.

Under the gesture recording mode, when it is determined that the motion is complete, the second processor calculates a reference acceleration value obtained by cumulatively summing linear acceleration values from the start of the motion to the end of the motion and the maximum angular velocity value for the roll axis from the start to the end of the motion. A reference rotation amount obtained by dividing the maximum angular velocity value on the pitch axis and a reference interference amount obtained by dividing the maximum angular velocity value on the yaw axis from the start of the motion to the end by the maximum angular velocity value on the pitch axis can be obtained and stored in the memory.

When the second processor determines that the motion is complete under the gesture recognition mode, a value obtained by accumulative sum of linear acceleration values from the start of the motion to the end is similar to the reference acceleration value within a reference range, and , The rotation amount, which is a value obtained by dividing the maximum angular velocity for the roll axis from the start of the motion to the end by the maximum angular velocity for the pitch axis, is similar to the reference rotation amount within a reference range, and the maximum angular velocity for the yaw axis from the start of the motion to the end If the interference amount, which is a value obtained by dividing the value by the maximum angular velocity value with respect to the pitch axis, is similar to the reference interference amount within a reference range, the motion of the user may be matched with the motion.

13 illustrates a flow diagram of a method for generating a gesture profile of a left or right move action, according to one embodiment.

In step S200, the electronic device may determine whether motion is ready. For example, if the slope with respect to the pitch axis and the slope with respect to the roll axis are within a reference range, it may be determined that the user is ready to make a motion. Otherwise, it returns to standby mode.

In a next step, the electronic device may determine whether motion has started. For example, if the angular velocity with respect to the pitch axis is greater than or equal to the reference value, it may be determined that the motion has started. Otherwise, it returns to standby mode.

In step S300, the electronic device may determine a gradient condition. For example, if the inclination of the pitch axis and the inclination of the roll axis are out of the standard range, it returns to the standby mode.

In step S400, the electronic device may determine a motion completion point. For example, it may be determined that the motion is completed when the current angular velocity value is less than or equal to the reference rate than the maximum angular velocity value after the motion starts.

In step S500, the electronic device may calculate and obtain a reference rotation amount and a reference interference amount, and store them in a memory.

Fig. 14 is a flow diagram of a method for recognizing, ie, whether a current motion of a user matches a gesture profile of a left or right move operation, according to an embodiment.

The method of FIG. 14 is mostly the same as that of FIG. 13, but the difference is that in the last step, the reference rotation amount and reference interference amount are calculated and compared with the previously stored reference values, rather than the reference rotation amount and reference interference amount calculated and stored in memory. there is.

Meanwhile, the above-described content control method can be implemented as computer readable codes on a computer readable recording medium. Computer-readable recording media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium is distributed in computer systems connected through a network, so that processor-readable codes can be stored and executed in a distributed manner.

The above descriptions are intended to provide exemplary configurations and operations for implementing the present invention. The technical idea of the present invention will include not only the above-described embodiments, but also implementations that can be obtained by simply changing or modifying the above embodiments. In addition, the technical idea of the present invention will also include implementations that can be achieved by easily changing or modifying the embodiments described above in the future.

Claims (2)

An electronic device paired with a host device to transmit a control signal based on a motion of a user, comprising:
accelerometer;
gyro sensor;
geomagnetic sensor;
Memory;
Based on the acceleration data received from the acceleration sensor, acceleration values for each of the first axis in the left and right direction, the second axis in the forward and backward direction, and the third axis in the up and down direction are obtained, and the gyro data received from the gyro sensor is obtained. Based on this, angular velocity values for each of the yaw axis, pitch axis, and roll axis are obtained, and acceleration data obtained from the acceleration sensor, the gyro sensor, and the geomagnetic sensor, respectively, gyro data, and a first processor for obtaining inclination values between a horizontal plane and each of a yaw axis, a pitch axis, and a roll axis direction of the electronic device based on geomagnetic data;
The acceleration value, the angular velocity value, and the inclination value received from the first processor are acquired in real time, and if the inclination value with respect to the pitch axis and the inclination value with respect to the roll axis are within a standard, it is determined that the motion is ready; A second processor that determines that the motion has started when the angular velocity with respect to the pitch axis is greater than or equal to a reference value, and handles an exception if the gradient value with respect to the pitch axis of the motion is out of the reference range,
Under the gesture recording mode, when it is determined that the motion is complete, the second processor calculates a reference acceleration value obtained by cumulatively summing linear acceleration values from the start of the motion to the end of the motion and the maximum angular velocity value for the roll axis from the start to the end of the motion. A reference rotation amount obtained by dividing the maximum angular velocity with respect to the pitch axis and a reference interference amount obtained by dividing the maximum angular velocity with respect to the yaw axis from the start of the motion to the end by the maximum angular velocity with respect to the pitch axis, and storing them in the memory. An electronic device containing two processors. According to claim 1,
When the second processor determines that the motion is complete under the gesture recognition mode, a value obtained by accumulative sum of linear acceleration values from the start of the motion to the end is similar to the reference acceleration value within a reference range, and , The rotation amount, which is a value obtained by dividing the maximum angular velocity for the roll axis from the start of the motion to the end by the maximum angular velocity for the pitch axis, is similar to the reference rotation amount within a reference range, and the maximum angular velocity for the yaw axis from the start of the motion to the end The electronic device matching the motion of the user with the motion when the interference amount, which is a value obtained by dividing the value by the maximum angular velocity value on the pitch axis, is similar to the reference interference amount within a reference range.

Images