KR101609553B1 - Apparatus and method for 3d motion recognition information input, and recording medium storing program for executing the same - Google Patents

Abstract

The present invention relates to an apparatus and method for inputting three-dimensional motion recognition information and a recording medium on which a program for implementing the same is stored. More specifically, the present invention relates to a three- Dimensional motion recognition information input device and method capable of remotely controlling the three-dimensional motion recognition information input device, and a recording medium storing a program for implementing the same.

Description

TECHNICAL FIELD [0001] The present invention relates to a three-dimensional motion recognition information input apparatus and method, and a recording medium storing a program for implementing the same. BACKGROUND OF THE INVENTION [0002]

The present invention relates to an apparatus and method for inputting three-dimensional motion recognition information and a recording medium on which a program for implementing the same is stored. More specifically, the present invention relates to a three- Dimensional motion recognition information input device and method capable of remotely controlling the three-dimensional motion recognition information input device, and a recording medium storing a program for implementing the same.

Various types of home appliances, office equipment, or portable smart devices have been developed and various types of input devices for controlling and inputting each device have been used as the spread of wearable devices has increased.

In the conventional PC environment, a mouse for inputting characters and a mouse for supporting a GUI environment are common. In a lecture or presentation environment, a wireless mouse is used to remotely control the PC or a presentation remote control is used.

In a conventional TV, a button-type remote control, in which one button is a function, is often used.

In addition, as smart devices and wearable devices become common, input interfaces different from those of existing devices are used.

Touch devices are common because smart devices or wearable devices are often portably emphasized and use small displays. A user can control the device through a touch using the body, and a technique of touching using a separate pen-shaped device.

In case of smart TV, mouse type remote control is used to support various functions of smart TV instead of conventional button type remote control. The mouse type remote control includes a technique of moving a mouse cursor by incorporating an acceleration sensor, or a mode of mounting a touch panel on a remote control.

In case of the remote controller with the built-in acceleration sensor, the user extracts the moving distance of the remote controller through the acceleration data, moves the mouse cursor, and controls the device by pressing a separate click button.

In addition, in the case of attaching a touch panel to a remote control, intuitive control such as a smart phone and a tablet PC is possible, but power consumption for the touch panel is large, which causes a problem of consuming a large amount of battery.

As described above, there is a problem that various input devices are required to use each device, and various input devices must be separately managed.

Korean Patent No. 10-0509913 discloses a multi-format input device and method.

Korea Registered Patent [10-0509913] (Registered Date: Aug. 17, 2005)

SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above-mentioned problems occurring in the prior art, and it is an object of the present invention to provide a mobile communication terminal, which is connected to a PC, smart device, wearable device, A portable / three-dimensional motion recognition information input device and method capable of intuitive device control and input by integrating functions such as a mouse, a keyboard, and a remote controller by using the axis information set by the user regardless of movement And a program for implementing the program.

Also, a wearable / portable three-dimensional motion recognition information input device and method capable of controlling various devices in a mouse mode, an operation mode, a handwriting mode, and a blank mode by designing a gesture capable of extracting sensor data in consideration of characteristics of a sensor, And a recording medium on which a program for implementing the method is stored.

According to an aspect of the present invention, there is provided an apparatus for inputting three-dimensional motion recognition information, comprising: an input device including an inertial sensor on a hand or a finger; The recognition information input device includes a sensor unit 110 composed of an IMU (Inertial Measurement Unit) sensor module including a 3-axis acceleration sensor, a 3-axis gyro sensor, and a 3-axis geomagnetic sensor, A motion input unit (100) including a wireless communication unit (120) for communication, a power supply unit (130) for supplying power to the sensor unit (110) and the wireless communication unit (120); And an integrated management unit 200 for analyzing a signal received from the wireless communication unit 120 through a wireless network and controlling an external device. The integrated management unit 200 receives information sensed from the sensor unit 110 A mouse mode, a blank mode, a handwriting mode, and transmits a control command defined in the mode to an external device.

In addition, the integrated management unit 200 includes a virtual plane management unit 210 for generating a virtual plane by calibration when connected to an external device, and moving the virtual plane according to the movement of the motion input unit 100; An operation recognition unit 220 for determining a valid motion based on predetermined sensitivity information and preset operation information when motion is detected from the motion input unit 100; And a command processing unit for checking whether there is an instruction corresponding to an effective motion when the motion recognition unit 220 determines that the motion is valid and controlling the external apparatus according to a command corresponding to a valid motion 230). ≪ / RTI >

The 3D motion recognition information input method according to an embodiment of the present invention includes a motion input unit 100 including a sensor unit 110, a wireless communication unit 120 and a power supply unit 130, a virtual plane management unit 210, A three-dimensional motion recognition information input method that includes a recognition unit 220 and an instruction processing unit 230 and includes an integrated management unit 200 and recognizes motion in a space by carrying a motion input unit 100 to a hand or a finger The integrated management unit 200 adjusts the sensitivity (the ratio of the motion of the motion input unit 100 and the motion of the external device) to an external device when the external device is connected or when a predetermined callee command is input (S10) of creating a virtual plane, and setting x, y, and z axes of the virtual plane; An effective motion determination step (S20) of determining an effective motion based on the sensitivity information set in the calibration step (S10) by the integrated management unit (200); A mode selection command determination step (S30) of determining, by the integrated management unit (200), whether motion is a motion corresponding to a predetermined mode selection command if it is determined that the motion is valid in the valid motion determination step (S20); A mouse mode, a blank mode, a handwriting mode, and a handwriting mode according to the predetermined mode selection command, when the integrated management unit 200 determines that the motion corresponds to the mode selection command in the mode selection command determination step S30 (S40) of controlling an external device in accordance with a corresponding command by comparing the mode information with preset operation information when it is determined that the selected mode is not the motion corresponding to the mode selection command .

Further, the 3D motion recognition information input method starts in a gesture mode when the 3D motion recognition information inputting method is interlocked with an external device.

In the mode selection command determination step S30, an operation of rotating at a predetermined angle about an axis perpendicular to the virtual plane with respect to the center of the motion input unit 100 and then returning a predetermined angle or more in the opposite direction is performed by the integrated management unit 200), it is recognized as a mode selection command.

In the command execution step S40, when the motion of the motion input unit 100 is analyzed in the gesture mode, if an operation on the virtual plane, an operation perpendicular to the virtual plane, and an operation combining the two are detected, Wherein the gesture command is a command for confirming whether the gesture command corresponds to the stored gesture command and controlling the external device according to the confirmed gesture command. .

The command execution step S40 analyzes the motion of the motion input unit 100 when the mouse mode is selected, generates a click or touch event when recognizing the motion perpendicular to the virtual plane, and recognizes the motion on the virtual plane The controller controls the movement of the mouse cursor displayed on the external device screen.

In the command execution step (S40), only the mode selection command is executed in the blank mode.

Also, in the command execution step S40, when the handwriting mode is selected, the corresponding character or handwriting is input by recognizing the typing or handwriting in the three-dimensional space.

In addition, the 3D motion recognition information input method is characterized in that the virtual plane is moved according to the movement of the motion input unit 100.

According to an embodiment of the present invention, there is provided a computer-readable recording medium storing a program for implementing the 3D motion recognition information input method.

According to an embodiment of the present invention, there is provided an apparatus and method for inputting three-dimensional motion recognition information, and a recording medium on which a program for implementing the same is stored. The motion input unit includes an inertial sensor It is possible to determine any one of the gesture mode, the mouse mode, the blank mode, and the handwriting mode according to the spatial operation, and to control the external device according to the control command defined in the mode, Intuitive input is possible.

More specifically,

The present invention can be applied to a smart device and a personal computer through motion of a sensor attached to a user's hand or carried by a hand in a three-dimensional space in place of a conventional input device such as a mouse, a keyboard, a presentation remote controller, Remote control and information input can be performed.

In addition, unlike an input device such as a mouse used in a conventional two-dimensional space, a user can intuitively control a smart device and a PC through movement of a hand in a three-dimensional space, Such motion information is transmitted to a device through a wireless network, and thus remote control and input of information can be performed.

In addition, the present invention has an effect of replacing the function of the presentation remote controller such as slide turning or slide end by interpreting the sensor movement information generated according to the movement of the user's hand in the presentation presentation and the lecture environment.

In addition, by using a wirelessly connected wearable / portable information input system instead of a touch-based interface used as a main input method in a smart device, it is possible to perform the same function as a touch input of a smart device, So that it is possible to control the apparatus at the same time.

In addition, since different user interfaces can be integrally used for each device such as a remote controller, a touch pad, and a touch screen in addition to the conventional keyboard and mouse, the user can reduce the learning burden and can be used as an integrated interface between the devices.

In addition, since the virtual plane is automatically changed according to the posture of the user, there is an effect that the user can comfortably control the external device without adhering to a specific posture.

In addition, it can be applied as a remote input device of a system using a touch interface such as a computer, a smart phone, a tablet, a smart TV, a smart home appliance, etc. It can be commercialized as a next generation input device capable of replacing a mouse or a keyboard, It can be applied to a pointing device that utilizes the recognition of the operation of a device having a Bluetooth function because it can be interlocked with the device on which the device is mounted so that the wearable device becomes more interested in the device and the input environment is changed, As an input device, an existing mouse and keyboard can be implemented as a small sensor device, and the technology can be applied.

1 is a conceptual diagram of a three-dimensional motion recognition information input device according to an embodiment of the present invention;
2 is a conceptual diagram showing the integrated management unit of FIG. 1 in more detail;
3 is a flowchart of a method of inputting three-dimensional motion recognition information according to an embodiment of the present invention.
Fig. 4 is a diagram for explaining that a change of the user attitude is not recognized as an instruction; Fig.
FIG. 5 is an exemplary view illustrating an operation algorithm of a gesture mode of a three-dimensional motion recognition information input method according to an embodiment of the present invention; FIG.
6 is a diagram illustrating an example of a gesture judgment algorithm of a method for inputting three-dimensional motion recognition information according to an embodiment of the present invention.
FIG. 7 is an exemplary diagram illustrating an information input method algorithm according to a mode of a three-dimensional motion recognition information input method according to an embodiment of the present invention; FIG.
FIG. 8 is an exemplary diagram illustrating a kind of a command in a three-dimensional motion recognition information input method according to an embodiment of the present invention; FIG.

Hereinafter, the present invention will be described in more detail with reference to the drawings. The following drawings are provided by way of example so that those skilled in the art can fully understand the spirit of the present invention. Therefore, the present invention is not limited to the following drawings, but may be embodied in other forms. In addition, like reference numerals designate like elements throughout the specification. It is to be noted that the same elements among the drawings are denoted by the same reference numerals whenever possible. Further, it is to be understood that, unless otherwise defined, technical terms and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Descriptions of known functions and configurations that may be unnecessarily blurred are omitted.

FIG. 1 is a conceptual diagram of a three-dimensional motion recognition information input apparatus according to an embodiment of the present invention, FIG. 2 is a conceptual diagram illustrating the integrated management unit of FIG. 1 in more detail, Dimensional motion recognition information input method according to an embodiment of the present invention, FIG. 4 is a diagram for explaining that a change of a user's posture is not recognized as an instruction, 6 is a diagram illustrating an example of a gesture judgment algorithm of the method for inputting 3D motion recognition information according to an embodiment of the present invention. Dimensional motion recognition information input method according to an embodiment of the present invention. FIG. 8 is a diagram illustrating a method of inputting a three-dimensional motion recognition information according to an embodiment of the present invention. Gourd is an exemplary view.

1, a three-dimensional motion recognition information input device according to an embodiment of the present invention includes a three-dimensional motion recognition information recognition unit 3 for recognizing motion in a space by carrying an input device including an inertial sensor on a hand or a finger, The input device includes a motion input unit 100 and an integrated management unit 200. The integrated management unit 200 controls the gesture mode, the mouse mode, the blank mode, and the motion mode based on the information sensed by the sensor unit 110, A mode selected in the handwriting mode is determined, and a control command defined in the mode is transmitted to the external device.

The motion input unit 100 includes a sensor unit 110, a wireless communication unit 120, and a power source unit 130.

The sensor unit 110 includes an IMU (Inertial Measurement Unit) sensor module including a 3-axis acceleration sensor, a 3-axis gyro sensor, and a 3-axis geomagnetic sensor.

Next, each sensor will be described in more detail.

The acceleration sensor senses the acceleration caused by the movement of the user's finger and transmits it to the integrated management unit through the wireless network. The integrated management unit integrates the acceleration data to obtain the velocity, and integrates this value again to obtain distance information of the sensor.

The gyro sensor senses the angular velocity that is rotated about an axis generated by the movement of the user's finger and transmits the detected angular velocity to the integrated management unit through the wireless network. The integrated management unit integrates the angular velocity data and obtains rotation angle information about the three axes of the sensor.

The geomagnetic sensor can measure the earth's magnetic field.

The IMU (Inertial Measurement Unit) sensor module can measure the direction of the sensor using acceleration, gyro and geomagnetic sensor values. That is, it can output direction information, which consists of yaw, roll and pitch have. yaw is the angle of rotation of the sensor from the direction indicated by the compass with respect to the axis parallel to the direction of gravity, pitch is the angle of rotation about the y axis of the sensor with respect to the plane parallel to the ground, And the angle of rotation about the x-axis.

The wireless communication unit 120 performs wireless network communication with an external device.

That is, the wireless communication unit 120 transmits information sensed by the sensor unit 110 to an external device using wireless communication.

Here, the external device may be a device such as a computer, a tablet, a notebook, a smart device, or the like using a keyboard, a mouse, a touch panel, or a touch screen as input means.

In other words, the three-dimensional motion recognition information input device according to an embodiment of the present invention remotely controls the smart device by attaching it to a user's finger or carrying it in a hand.

And supplies power to the sensor unit 110 and the wireless communication unit 120.

The integrated management unit 200 analyzes the signal received from the wireless communication unit 120 through the wireless network and controls the external device. At this time, the integrated management unit 200 may also include a wireless communication module capable of wireless communication.

That is, the integrated management unit analyzes the finger movement of the user through the sensor data transmitted through the wireless network, determines the corresponding mode among the mouse mode, the gesture mode, the blank mode, and the handwriting mode, To the external device.

2, the integrated management unit 200 of the three-dimensional motion recognition information input device according to an embodiment of the present invention includes a virtual plane management unit 210, an operation recognition unit 220, and a command processing unit 230, . ≪ / RTI >

The virtual plane management unit 210 creates a virtual plane by calibration when connected to an external device, and moves the virtual plane according to the movement of the motion input unit 100.

The virtual plane management unit 210 manages information about virtual planes and axes that the user is writing. The virtual plane management unit 210 stores contiguous recent data obtained from the recent acceleration sensor and the gyro sensor, and periodically updates the axis information of the virtual plane using the latest data. The generated axis information is periodically updated when sensor movement occurs in each mode and is initialized through calibration when it is linked with the smart device as a wireless network for the first time.

When motion is detected from the motion input unit 100, the motion recognition unit 220 determines valid motion based on predetermined sensitivity information and predetermined motion information.

That is, the information received through the motion input unit 100 is transmitted to the motion recognition unit 220 to perform motion recognition.

In addition, the axis for gesture recognition may be based on an axis stored in the virtual plane management unit 210.

For example, the predetermined operation information may be an operation of returning to a starting point after performing an operation at a certain starting point, and may be interpreted as a completion of an operation when the apparatus returns to the starting point . This is natural in the operation other than the command, and it is for filtering out the operation for commanding.

The preset operation information may be an operation on a virtual plane or an operation perpendicular to a virtual plane, or a combination of the two operations.

Further, if the movement of the sensor caused by the movement of the user's finger or hand is curved, if it is analyzed whether it is moving clockwise or counterclockwise on the virtual plane, more various motions can be used as a control command. When the sensor moves in the counterclockwise direction on the xz plane, it is determined that the sensor moves in the (-) direction of the x axis. When the sensor moves in the clockwise direction on the xz plane, the sensor moves in the (+) direction of the x axis It can be judged. Also, when the movement of the curved sensor in the counterclockwise direction on the yz plane occurs, it is determined that the sensor has moved in the positive direction of the y axis, and when the movement in the clockwise direction on the yz plane occurs, -) direction.

In order to determine one gesture, the motion recognition unit 220 may store a direction when a motion that exceeds the effective acceleration in one direction occurs. This makes it possible to store in seven types including directions representing six directions (front, back, left, right, top, bottom) and circles including the (-) and (+) directions of each axis.

For example, when the direction of the sensor movement is changed by more than 40 degrees, the motion recognition unit 220 may display different strokes and store the direction and the start coordinates for each stroke. That is, when a motion parallel to an axis different from the current direction of movement is detected, it can be determined as a new stroke. You can save the number of strokes and add one each as the stroke increases. In addition, when the circular motion is detected on the virtual plane, the same control as that of pressing the home button of the smart device can be performed, and if the circular motion is consecutively detected, it can be interpreted as enlargement or reduction. The clockwise direction can be expanded and the counterclockwise direction can be reduced.

If it is determined by the motion recognition unit 220 that the motion is valid motion, the command processing unit 230 checks whether there is an instruction corresponding to the valid motion, and if there is an instruction corresponding to the valid motion, .

That is, the three-dimensional motion recognition information input device according to an embodiment of the present invention is a wearable / portable information input system using motion recognition on a three-dimensional space, and the integrated management unit 200 controls the movement input unit 100, And the integrated management unit 200 can control an external device according to the analyzed result.

3, the 3D motion recognition information input method according to an embodiment of the present invention includes a motion input unit 100 including a sensor unit 110, a wireless communication unit 120, and a power source unit 130, And an integrated management unit 200 including a plane management unit 210, an operation recognition unit 220 and an instruction processing unit 230. The integrated management unit 200 receives a motion input unit 100, Dimensional motion recognition information input method includes a calibration step (S10), an effective motion determination step (S20), a mode selection command determination step (S30), and an instruction execution step (S40).

In the calibration step S10, when the integrated management unit 200 is connected to an external device or when a predetermined callee command is input, the sensitivity information (the ratio of the motion of the motion input unit 100 to the motion of the external device) Adjust to the instrument, create a virtual plane, and set the x, y, and z axes of the virtual plane.

The three-dimensional motion recognition information input device according to an embodiment of the present invention performs calibration when interworking with an external device. Calibration tracks the movement of the sensor by setting the x, y, and z axes of the virtual plane using data about the axis of the ground for the space the sensor is to move. The axis thus set is used to recognize a virtual writing plane arbitrarily assumed by the user in the three-dimensional space. Also, when the motion of the sensor is measured with respect to the set axis, it may be subjected to a sensitivity setting step for determining whether it is a valid motion (motion, gesture). This is because the size of a single gesture is different depending on the user. It is desirable to set the sensitivity to distinguish the gesture from the non-gesture depending on the movement of the sensor. It acts as a mouse sensitivity control on the computer. For example, in the case of a gesture parallel to the virtual plane, the sensitivity can be set so that the gesture recognizes only a movement of 1/3 or more of the screen size set in the calibration.

It is also possible to set the sensitivity to the default value without having to calibrate.

The user can perform the calibration by inputting motion such as 'L' shape at the initial stage of interlocking with the external device. At this time, the plane in which 'L' is drawn can be recognized as a virtual writing plane (virtual plane) of the user. In addition, the least square method can be used to recognize the plane on which 'L' is drawn. In addition, one virtual handwriting plane can be recognized by using the least square method using the three-dimensional space coordinates obtained by integrating the acceleration data. In addition, if the starting axis of the letter "L" is set to the x axis, a vector perpendicular to the x axis among the vectors on the recognized virtual writing plane can be set to the y axis, and an axis perpendicular to the virtual writing plane is set to the z axis .

In this case, the 3D motion recognition information input method according to an embodiment of the present invention may be started in a gesture mode when the 3D motion recognition information is interlocked with an external device. This is to reduce crosstalk because it does not know what mode to start when connecting to an external device, and moreover, to easily control an external device with an intuitive operation. The gesture mode is started when the external device is connected to the external device. However, the present invention is not limited thereto, and it is also possible to start the mouse mode, the blank mode, or the handwriting mode.

In the valid motion determination step S20, the integrated management unit 200 determines valid motion based on the sensitivity information set in the calibration step S10.

Determining valid motion is to filter out noises and to filter out minor movements.

In the mode selection command determination step S30, if the integrated management unit 200 determines that the motion is valid motion in the valid motion determination step S20, it determines whether the motion corresponds to a predetermined mode selection command.

At this time, in the mode selection command determination step S30, an operation of rotating at a predetermined angle about an axis perpendicular to the virtual plane with respect to the center of the motion input unit 100, and returning a predetermined angle or more in the opposite direction, 200), it is recognized as a mode selection command.

For example, if an operation of rotating 180 degrees around an axis perpendicular to the virtual plane with respect to the center of the sensor and then returning to the original position is detected, the motion recognition unit can recognize the mode change. This can be detected by changing the values of pitch, roll, and yaw only when the acceleration value of the sensor is not changed, unlike the case of drawing a circle.

This is because it is the operation of operating a rotary switch, which is intuitively related to mode selection. Although an example of the mode selection command has been described above, the present invention is not limited thereto, and it is also possible to designate another operation as a mode selection command.

That is, in the method for inputting three-dimensional motion recognition information according to an embodiment of the present invention, a user can select a mode, such as a mouse mode, a handwriting mode, a gesture mode, or a blank mode, Can be changed.

In the command execution step S40, when the integrated management unit 200 determines that the motion corresponding to the mode selection command is determined in the mode selection command determination step S30, the gesture mode, the mouse mode The blank mode, and the handwriting mode is determined. When it is determined that the motion is not the motion corresponding to the mode selection command, the external device is controlled according to the corresponding command by comparing with the preset operation information.

Next, each mode will be described in more detail.

In the command execution step S40, when the gesture mode is selected, the operation of the motion input unit 100 is analyzed. When an operation on the virtual plane, an operation perpendicular to the virtual plane, and an operation combining the two are detected, And the external device is controlled according to the confirmed gesture command. The gesture command stored in advance may be a gesture command for performing an operation at a starting point and returning a predetermined distance or more in the direction of the starting point again . ≪ / RTI >

For example, when an operation of returning more than 2/3 of the moving distance again with respect to the moving distance with respect to an axis is detected, it can be set as a valid gesture. In the case of a gesture using motion information of two or more axes among the x, y, and z axes, it is preferable that the motion is recognized as a gesture only when an operation of returning to all the axes proceeding is detected. If motion is detected only for partial axes, it can only be interpreted as a gesture for this axis. If motion is returned only for a partial axis, it is desirable to interpret it as an effective gesture only after returning in the forward direction.

Gesture mode is a mode that can be used to control the status of the device such as screen on / off, volume control, or to control the screen output range of the device such as scrolling, slide flipping, zooming, etc., It is possible to control the device with the corresponding control information (control command) in comparison with the set operation information information.

The virtual plane for gesture judgment can be held parallel or perpendicular to the ground. It is preferable not to update the virtual plane for the operation judged as a gesture with respect to the virtual plane used previously. For a non-gesture operation, the virtual plane can be changed by comparing the posture of the sensor before the operation and the posture of the sensor after the operation. At this time, it is preferable not to use the roll value (rotational angle with respect to the axis indicated by the finger) in determining the posture of the sensor. That is, if the x-axis of the existing virtual plane is parallel to the ground, it is desirable to keep it.

  For example, assuming that the virtual plane is perpendicular to the ground and in front of the user and the user has rotated 90 degrees to the left, the movement of the sensor is not recognized as a gesture because it does not show reciprocity. This situation is shown in Fig. The virtual plane is expressed by expressing the x and y axes of the virtual plane as vectors, and the blue arrows represent the movement path. Since the yaw is changed by 90 degrees, the vector constituting the virtual plane is rotated by the rotated angle. Here, the y axis holds (0,0,1) and the x axis rotates from (1,0,0) to (0,1,0). The matrix used for the rotation equation is the same as the rotation matrix used previously.

Here, it is desirable to not update the virtual plane every time the operation is performed because the user frequently uses the direction unchanged, and it is inefficient to update the virtual plane every operation. In the case of the gesture of turning the screen to the left and right, the user's palm may not always be directed toward the floor because the user does not have any inconvenience when the palm is facing to the user's side. In this case, it is preferable not to create a virtual plane only on the axis of the sensor because the axis can be changed only in the same virtual plane.

An operation algorithm of the gesture mode may be an example as shown in Fig.

When creating a virtual plane, the left and right direction can be determined according to the finger posture. For example, if the direction of the z-axis matches the direction of the gravitational acceleration, the + x axis direction can be judged as the user's right.

In the case of a right-handed gesture, the gesture judgment algorithm is as shown in FIG. 6 for the finger-posture virtual plane for the left and right screen flick gesture in the gesture mode.

In order to avoid judging fine movement as a gesture, it is possible to recognize as a gesture when a moving distance exceeds a predetermined ratio of the screen size when matching to the screen of an external device. Then, it is judged in which axis direction (ex) + x, -x ... relative to the virtual plane, and if the movement in the opposite direction is started within a certain time (for example, (movement on the -x axis occurs), it is recognized as one gesture.

The command execution step S40 analyzes a motion of the motion input unit 100 in a mouse mode, generates a click or a touch event when recognizing an action perpendicular to a virtual plane, and recognizes an action on a virtual plane And controls the movement of the mouse cursor displayed on the external device screen.

The mouse mode can control the movement of the mouse cursor displayed on the external device screen, and can generate a touch event when the user's touch operation is recognized by analyzing the finger operation.

For example, when the movement of the sensor in the z-axis direction is removed, the latest three-axis motion data amount is set to be orthogonal to the plane having the smallest error periodically obtained for the movement, You can control the movement of the mouse as much as proportionally.

In addition, information on the virtual writing plane (virtual plane) is periodically updated according to the motion of the sensor, and this information is transmitted to the virtual plane managing unit 210 and can be managed by the virtual plane managing unit 210.

The mouse mode can be interpreted as a touch event occurring when the sensor (motion input unit 100) proceeds in the direction of entering the virtual plane in the z-axis direction and then proceeds in the direction coming out of the virtual plane again . A double-click or a double-touch can be interpreted as a double-click when the action corresponding to the click / touch is performed twice consecutively. It can be interpreted that the touch is released when the movement on the virtual plane does not return to the place after performing the operation to enter the z-axis of the virtual plane, and then interprets it as a drag operation and then returns to the original position.

The mouse mode is a movement (touch movement, etc.) that reciprocates in the direction of the finger point (when the sensor is placed on the finger, the direction of the y-axis of the sensor) . That is, it is preferable that the virtual plane is periodically obtained by using the least square method on the data excluding the y-axis acceleration data of the sensor. The determination of the direction of the x-axis of the virtual plane (left and right of the user) can be based on the axis of the sensor, and it is possible to express the movement of the mouse by orthogonalizing the data on the obtained virtual plane.

The movement of the mouse cursor can be controlled using the amount of angular change measured by the IMU sensor. In other words, when a user attaches a sensor to a hand or a finger to move the sensor vertically or horizontally, the amount of change of the angle with respect to each axis of the sensor is measured and compared with the sensitivity, Can be controlled.

For example, if the y axis of the sensor points to the front of the user, the left and right movement of the user can be detected as yaw, and the up and down movement can be detected as the change of the roll value. The yaw and roll values are stored when the device and the sensor start interlocking, and the direction corresponding to this value is recognized as the front of the user. Accumulating the changes of angles occurring thereafter based on this, You can control the cursor.

Even when the angular change amount is measured in the same manner, the movement distance of the sensor is different.

In order to reflect this, the weight of the moving distance is expressed by multiplying the measured distance by the movement distance obtained by integrating the acceleration, and the movement of the mouse cursor can be controlled differently according to the moving distance even when the movement is the same. Also, the point where the sign of the angle change data changes is determined as the direction change point, and the velocity data obtained by integrating the acceleration at this point is made to be zero. By using this, the cumulative error for the acceleration integral value is reduced, The error can be reduced.

In the command execution step S40, in the case of the blank mode, only the mode selection command is executed.

Blank mode is a mode that does not recognize gestures, handwriting, mouse cursor movements, etc. while the screen is on, and motion for other motions is not interpreted except mode switching.

That is, when a motion (change in the gyro sensor value) corresponding to a mode selection command necessary for mode switching is detected, only the mode selection command is executed.

The blank mode can create a new virtual plane in line with the axis of the sensor after the mode conversion operation. It is common that the palm is faced downward after the mode conversion operation, and it is preferable to consider the direction of the finger to be the front of the user and create the virtual plane. The principle of virtual plane generation is as follows.

When the movement starts in the stopped state, the direction in which the finger is pointing can be regarded as the front of the user, and a virtual plane can be created. The axis of the virtual plane to be created is made up of an axis perpendicular to the plane of the plane. For example, when creating a new virtual plane, the y-axis of the virtual plane matches the direction of the gravitational acceleration. The left and right (+ x, -x) directions of the virtual plane can be determined by the axis (y-axis of the sensor) and the axis excluding the axis corresponding to the gravitational acceleration direction with respect to the axis of the sensor. For example, if the direction of gravitational acceleration matches the z-axis of the sensor, the x-axis of the sensor determines the left and right of the virtual plane.

In the command execution step S40, when the handwriting mode is selected, the corresponding character or handwriting is input by recognizing the typing or handwriting in the three-dimensional space.

The handwriting mode can be switched to the handwriting mode through a mode selection command by the user clicking on the input window through the keyboard or the touch keyboard or the like using the mouse cursor on the external device screen.

The handwriting mode recognizes the handwriting in the user's three-dimensional space and recognizes the corresponding character. This can be done through a handwriting recognition unit.

The handwriting mode is Korean, English lower case, English upper case and '!', '?', '^', '.' You can enter special characters such as, and space, and line breaks can also be input.

For example, in the handwriting mode, characters and characters can be distinguished by recognizing that the user has stopped writing for a certain period of time after writing one character. That is, when the acceleration data is not measured for a predetermined time or more after writing one character, it can be interpreted that input for one character is completed.

In the handwriting mode, the virtual plane management unit can update the virtual plane information for each character. When the input for one character is completed, the virtual plane can be recognized through the least square method. The determination of the direction of the x-axis of the virtual plane (left and right of the user) can be based on the axis of the sensor, and the character can be discriminated against the orthographic data on the virtual plane.

The handwriting mode can recognize a character by orthogonalizing the handwriting data of the user recognized on the virtual plane. Spaces and line breaks are the same as those used in gesture mode.

In the handwriting mode, the user can exit the input window through the mode selection command when handwriting is completed.

Next, one embodiment of a method of tracking movement to each sensor will be described in more detail.

When the yaw, roll, and pitch are all 0, the axis of the sensor is recognized as the axis with respect to the ground. And this is an axis that does not change regardless of the movement of the sensor.

The acceleration data measured by the acceleration sensor is a value including gravitational acceleration, and the system has a signal preprocessing step of removing noise and gravitational acceleration. The equation for obtaining acceleration data by removing gravitational acceleration is as follows.

X-axis acceleration = (based on sensor with gravity acceleration) x-axis acceleration-gravity acceleration * sin (pitch)

Y axis Acceleration = y Axis Acceleration - Gravity Acceleration (based on sensor with gravity acceleration) sin (roll)

Z axis Acceleration = (based on the sensor with gravity acceleration) z axis acceleration - gravity acceleration * cos (pitch) * cos (roll)

Tracking the movement of the sensor by measuring the acceleration on the axis of the sensor reference requires an axis that does not change with the movement of the sensor because the axis continuously changes as the sensor moves. Therefore, the axis of the sensor is used at positions where yaw, roll, and pitch are both zero. This axis is an axis that does not change regardless of the motion of the sensor. Rotate the axis of the sensor so that yaw, roll, and pitch are zero.

(Acceleration) = cos (-yaw) * cos (-pitch) * x-axis acceleration + (cos (-yaw) * sin (-pitch) (-yaw) * cos (-roll) * y-axis acceleration (based on the sensor) + (cos (-yaw) * sin (-pitch) )) * Z-axis acceleration (based on sensor)

Y axis acceleration = sin (-yaw) * cos (pitch) * x axis acceleration + sin (-yaw) * sin (-pitch) * sin (-roll) + cos (-yaw) * cos (-roll) * y axis acceleration + sin (-yaw * sin (-pitch) * cos (-roll) )) * Z-axis acceleration (based on sensor)

(Relative to the ground) z-axis acceleration = -sin (-pitch) * x-axis acceleration + cos (-pitch) * sin (-roll) * y- ) * cos (-roll) * z-axis acceleration

The axis for the virtual plane is expressed in vector format with respect to the data obtained based on the axis with respect to the ground.

[Example]

The apparatus and method for inputting three-dimensional motion recognition information according to an embodiment of the present invention can move a sensor in a three-dimensional space of a motion input unit 100 in a predetermined operation (a mode selection command, a control command defined in a corresponding mode) So that the external device can be controlled. The mode selection command or the control command defined in the corresponding mode can be classified into a two-dimensional motion expressing the movement of the sensor on the set virtual plane and an operation perpendicular to the plane. In general, users are familiar with two-dimensional movement, such as typing with a mouse or tablet PC, so that most gestures can be performed on virtual planes and commands such as clicks can be defined as vertical movements, The gesture can be made to correspond to the command.

7 shows an algorithm for inputting information according to modes.

In order to recognize the mode selection command or the control command defined in the corresponding mode, it is first necessary to determine whether the motion data of the sensor is generated by a user performing a gesture or a non-gesture movement caused by movement of the user or arm or hand . It is possible to determine whether the motion is valid through the sensitivity information set at the beginning of the device interlock in the valid motion recognition unit (not shown) included in the command processing unit 230. [ For this purpose, it is possible to set a reference value of the acceleration which is determined as the effective motion of the valid motion recognition unit (not shown). When the acceleration is greater than the valid motion reference value among the absolute values of the acceleration values of the three axes when the acceleration data is generated by moving the sensor, the user recognizes that the control command specified in the mode selection command or the corresponding mode is taken and performs gesture recognition .

If it is recognized as an effective gesture in the valid motion determination step S20, the sensor acceleration is integrated to check whether the extracted sensor movement data matches a movement direction of the prescribed gesture. The kind of the predefined gesture can be defined as shown in FIG. 8 indicates movement in the xy plane, a thick arrow indicates movement in a direction perpendicular to the xy plane, and arrows in the circle indicate motions rotating about the center of the sensor (the acceleration sensor data is 0 And the gyro sensor value is present), a circle with one arrow indicates a rotational movement in the xy plane, and a circle with two arrows indicates a continuous rotational movement. This is to define an operation in which the distinction between the gesture and the non-gesture is easily possible and the positions of the beginning and end of the gesture coincide with each other in order to solve the problem of gesture separation occurring when inputting successive gestures. In addition, since the acceleration pattern generated upon returning is easier to detect than when it proceeds in one direction, the false alarm probability of the non-gesture operation can be reduced.

The correspondence between the motion (gesture) and the instruction through the motion recognition can be specified differently according to the user, but can be designated and used in a main command used in a smart device, a wearable device, or a PC. Major commands are Left (Previous slide), Right (Next slide), Up (Volume up), Down (Volume down), Click, Double click, Zoom, Home button, Previous button, Menu button, to be.

In the case of an action corresponding to a click, it can be implemented through movement in the z-axis similar to the operation of clicking the mouse with the sensor attached to the finger. Double clicking can be achieved by performing a click gesture sequentially. The operation that is opposite to the click operation, that is, the operation of moving the sensor toward the user's body first and then moving the sensor backward, can be used in correspondence with the 'previous' button of the smart device. This is a valid gesture even in mouse movement mode. In addition, the volume up / down function, which is relatively less used to distinguish between up and down scroll commands and volume up / down, can be used by adding z-axis movement. The home button is recognized when the sensor moves in a direction toward the virtual plane, and when the circle is drawn, and the enlargement or reduction can be set to recognize when drawing a circle clockwise or counterclockwise on the virtual plane.

The 3D motion recognition information input method according to an embodiment of the present invention is characterized in that a virtual plane is moved according to movement of the motion input unit 100. [

That is, when the posture of the user changes and the center axis of the motion input unit 100 changes, the virtual plane can be updated accordingly.

Although the 3D motion recognition information input method according to the embodiment of the present invention has been described above, the 3D motion recognition information input method described above can be implemented by a tangible program of instructions for implementing the 3D motion recognition information, It will be understood by those skilled in the art that the present invention may be embodied in a recording medium capable of being recorded. In other words, it can be implemented in the form of a program command that can be executed through various computer means, and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the computer software. Examples of the computer-readable medium include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs and DVDs, and optical disks such as floppy disks. Magneto-optical media and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, USB memory, and the like. The computer-readable recording medium may be a transmission medium such as a light or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

100: motion input unit 110:
120: wireless communication unit 130:
200: Integrated management unit 210: Virtual plane management unit
220: operation recognition unit 230: command processing unit
S10: Calibration step
S20: valid motion determination step
S30: Mode selection command judgment step
S40: Command execution phase

Claims (11)

A three-dimensional motion recognition information input device for recognizing movement in space by carrying an input device including an inertial sensor on a hand or a finger,
A sensor unit 110 composed of an IMU (Inertial Measurement Unit) sensor module including a 3-axis acceleration sensor, a 3-axis gyro sensor, and a 3-axis geomagnetic sensor, a wireless communication unit 120 A motion input unit 100 including a sensor unit 110 and a power supply unit 130 for supplying power to the wireless communication unit 120; And
An integrated management unit 200 for analyzing a signal received from the wireless communication unit 120 through a wireless network and controlling an external device;
, ≪ / RTI &
The integrated management unit 200
A virtual plane management unit 210 for creating a virtual plane by calibration when connected to an external device and moving the virtual plane according to the movement of the motion input unit 100;
An operation recognition unit 220 for determining a valid motion based on predetermined sensitivity information and preset operation information when motion is detected from the motion input unit 100; And
A command processing unit 230 for controlling the external device according to the command if there is an instruction corresponding to the valid motion, if the motion recognition unit 220 determines that the motion is valid; );
Lt; / RTI >
The integrated management unit 200 determines any one of a gesture mode, a mouse mode, a blank mode, and a handwriting mode based on the information sensed by the sensor unit 110, To an external device,
In the gesture mode,
When the operation of the motion input unit 100 is analyzed and an operation on the virtual plane, an operation perpendicular to the virtual plane, and an operation combining the two are detected, it is checked whether the operation corresponds to a pre-stored gesture command, The gesture command is a type of operation in which the gesture command is returned from the start point and then returned a predetermined distance or more within a predetermined time from the start point to the start point, It is determined that the operation is not a gesture, and the virtual plane is changed by comparing the posture of the sensor before the operation and the posture of the sensor after the operation,
In the blank mode, which is a mode for not recognizing gestures, handwriting, and movement of the mouse cursor while the screen is on,
The mode switching is performed only when a change in the gyro sensor value, which is a motion corresponding to the mode selection command necessary for mode switching, is detected, and other motions other than the mode switching are not interpreted. Dimensional motion recognition information input device.
delete An integrated management unit including a motion input unit 100 including a sensor unit 110, a wireless communication unit 120 and a power supply unit 130, a virtual plane management unit 210, an operation recognition unit 220, and a command processing unit 230 200) for recognizing movement in a space by carrying a motion input unit (100) on a hand or a finger, the method comprising:
The integrated management unit 200 adjusts the sensitivity (the ratio of the motion of the motion input unit 100 and the motion of the external device) information to an external device when the external device is connected or when a predetermined calle command is input, A calibration step (S10) of creating a virtual plane, and setting x, y, and z axes of the virtual plane;
An effective motion determination step (S20) of determining an effective motion based on the sensitivity information set in the calibration step (S10) by the integrated management unit (200); And
A mode selection command determination step (S30) of determining, by the integrated management unit (200), whether motion is a motion corresponding to a predetermined mode selection command if it is determined that the motion is valid in the valid motion determination step (S20); And
If it is determined by the integrated management unit 200 that the motion corresponding to the mode selection command is determined in the mode selection command determination step S30, the gesture mode, the mouse mode, the blank mode, the handwriting mode An instruction execution step (S40) of controlling an external device in accordance with a corresponding command by comparing the selected operation mode with predetermined operation information if it is determined that the selected mode is not the motion corresponding to the mode selection command;
/ RTI >
The command execution step (S40)
In the gesture mode,
When the operation of the motion input unit 100 is analyzed and an operation on the virtual plane, an operation perpendicular to the virtual plane, and an operation combining the two are detected, it is checked whether the operation corresponds to a pre-stored gesture command, The gesture command is a type of operation in which the gesture command is returned from a start point and then returned a predetermined distance or more within a predetermined time in the direction of the starting point, It is determined that the operation is not a gesture, and the virtual plane is changed by comparing the posture of the sensor before the operation and the posture of the sensor after the operation,
In the blank mode, which is a mode for not recognizing gestures, handwriting, and movement of the mouse cursor while the screen is on,
The mode switching is performed only when a change in the gyro sensor value, which is a motion corresponding to the mode selection command necessary for mode switching, is detected, and other motions other than the mode switching are not interpreted. Dimensional motion recognition information.
The method of claim 3,
The three-dimensional motion recognition information input method
Wherein the gesture mode is started when the external device is interlocked with the external device.
The method of claim 3,
In the mode selection command determination step S30,
When the integrated management unit 200 detects an operation in which the rotation is rotated at a predetermined angle about an axis perpendicular to the virtual plane with respect to the center of the motion input unit 100 and then returns by a predetermined angle or more in the opposite direction, Dimensional motion recognition information.
delete The method of claim 3,
The command execution step (S40)
In the mouse mode,
The motion of the mouse cursor displayed on the external device screen is controlled by analyzing the motion of the motion input unit 100 to generate a click or touch event when recognizing the motion perpendicular to the virtual plane and recognizing the motion on the virtual plane Dimensional motion recognition information input method.
delete The method of claim 3,
The command execution step (S40)
In the handwriting mode,
And recognizing the typing or handwriting in the three-dimensional space and inputting corresponding letters or handwriting.
delete A computer-readable recording medium storing a program for implementing the three-dimensional motion recognition information input method according to any one of claims 3 to 5, 7, and 9.

Images