CN111580686B - Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit - Google Patents

Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit Download PDF

Info

Publication number
CN111580686B
CN111580686B CN202010389693.3A CN202010389693A CN111580686B CN 111580686 B CN111580686 B CN 111580686B CN 202010389693 A CN202010389693 A CN 202010389693A CN 111580686 B CN111580686 B CN 111580686B
Authority
CN
China
Prior art keywords
sequence
pen
gesture
dimensional
ave
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010389693.3A
Other languages
Chinese (zh)
Other versions
CN111580686A (en
Inventor
陈建
王程
孙晓颖
燕学智
刘思宇
于帆
林琳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jilin University
Original Assignee
Jilin University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jilin University filed Critical Jilin University
Priority to CN202010389693.3A priority Critical patent/CN111580686B/en
Publication of CN111580686A publication Critical patent/CN111580686A/en
Application granted granted Critical
Publication of CN111580686B publication Critical patent/CN111580686B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The invention provides a three-dimensional stroke recognition method integrating ultrasonic positioning and an inertia measurement unit, and belongs to three-dimensional stroke recognition methods. The method for recognizing the three-dimensional pen gesture comprises the following steps of defining the three-dimensional pen gesture by using four kinds of attribute information including track form, motion direction, pen body posture and space position, wherein the recognition method specifically comprises the following steps: a track form identification method based on normalized center distance coding; a motion direction identification method based on pen body speed estimation; a pen body posture identification method based on the posture angle sequence variance and a space position identification method based on the distance sum of the inner point of the subspace and the center of the interface. The invention can better meet the characteristics of large interaction space, six degrees of freedom and various pen holding postures of three-dimensional gesture interaction, and can be applied to the fields of human-computer interaction, virtual reality, augmented reality, somatosensory games and the like.

Description

Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit
Technical Field
The invention belongs to a three-dimensional handwriting recognition method. The method firstly uses four kinds of attribute information of track form, motion direction, pen body posture and space position to define the three-dimensional pen gesture. On this basis, the identification method specifically comprises the following steps: a track form identification method based on normalized center distance coding; a motion direction identification method based on pen body speed estimation; a pen body posture identification method based on the posture angle sequence variance and a space position identification method based on the distance sum of the inner point of the subspace and the center of the interface. The method provided by the invention is better suitable for the characteristics of large interaction space, six degrees of freedom, various pen holding postures and the like of three-dimensional gesture interaction, and can be applied to the technical fields of human-computer interaction, virtual reality, somatosensory games and the like.
Background
Human-computer interaction is the subject of studying the interaction between a user and a system. With the development of computer technology and the emergence of new interaction requirements, the current traditional two-dimensional interaction mode cannot meet the requirements of users due to the reasons of small interaction space, low interaction freedom, lack of interaction mode and the like, and the three-dimensional interaction becomes a research hotspot in the field of human-computer interaction. The three-dimensional pen type interaction reduces the learning cost of a user by means of a paper pen metaphor so as to realize efficient and natural human-computer interaction, and is a novel three-dimensional interaction mode. The three-dimensional gesture interaction is an interaction mode that a user holds an electronic pen to complete a specific action in the pen interaction process, and has the characteristics of large interaction space, six degrees of freedom and various pen holding gestures, and the characteristics are not fully researched by the existing gesture recognition scheme. Therefore, the method for defining and identifying the three-dimensional pen posture by fusing the multi-channel information is researched, new functions and applications are provided for three-dimensional pen type interaction, and the method has important research significance and application value.
There are several patents and academic research on handwriting recognition methods.
Chinese patent 'a three-dimensional handwriting recognition method based on ultrasonic positioning' (patent number 201611157261.X) discloses a three-dimensional handwriting recognition method based on ultrasonic positioning, and belongs to the field of human-computer interaction. Acquiring a three-dimensional coordinate of a track drawn in space by a user when inputting a three-dimensional gesture through an ultrasonic positioning device; calculating projection coordinates of three-dimensional coordinates of the space passing through the track on XoY, ZoX and YoZ planes, and performing smoothing processing to remove tiny deviation caused by hand shake of a user or environmental noise; and matching the three-dimensional handwriting projection coordinates after smoothing with projection coordinates in a system three-dimensional handwriting base, executing a corresponding instruction if matching is successful, automatically performing self-adaptive updating of the system handwriting base if matching is failed, and adding three-dimensional handwriting habituated by a user into the system to adapt to input habits of different users. The Chinese patent 'a three-dimensional handwriting recognition method based on direction chain codes' (patent number 201710402132.0) discloses a three-dimensional handwriting recognition method based on direction chain codes, which belongs to the field of human-computer interaction, and is characterized in that the direction chain codes are adopted to describe the motion track of a pen by collecting the space position information of the pen in a pen input channel, so as to establish a three-dimensional handwriting model, and the interaction intention of a user is recognized in real time by matching with the model set in a system. The method can identify some gesture operations of the user in the three-dimensional space in real time, has certain universality and expansibility, supports three-dimensional book space identification, enhances the naturalness of a human-computer interaction platform, adopts a coding mode of the direction chain code, has simple and convenient algorithm, small calculated amount and high real-time performance, and can carry out accurate identification through denoising and integrating the direction chain code. Although the two methods both realize the gesture recognition function based on the ultrasonic positioning data, only the data of the sensor channel of the ultrasonic positioning coordinates is used, the recognized gesture content is only defined by the attribute of the track form, the factors of other aspects such as the motion direction, the gesture of the pen body, the space position and the like are not considered, and the user requirements can not be met in practical application.
Jeen-Shing Wang researchers in Taiwan of China integrate a three-axis accelerometer, a control module and a wireless communication module on an electronic pen. The user presses the key and simultaneously starts the gesture operation, and the gesture is finished and the key is loosened. And in the period of time, the real-time data of the accelerometer is sent to the upper computer by the wireless communication module and recorded, and the acquisition of the handwriting information is completed. On the basis, the statistical characteristics such as mean, variance, standard deviation and the like are extracted from the triaxial acceleration data. And substituting the characteristics into a neural network model for training and recognition. Excessive constraints are added to the gesture actions in related researches in current gesture recognition researches based on an inertial measurement unit, for example: the experimenter is required to keep the posture of the pen body fixed when completing the gesture action, the pen point points to a specific plane, the sizes of tracks formed by the gestures are consistent as much as possible, and the like. The purpose of these constraints is to reduce the difficulty of feature design and improve recognition rates. However, in practical applications, these constraints may reduce the applicability of the handwriting recognition method.
Disclosure of Invention
The invention provides a three-dimensional gesture recognition method integrating ultrasonic positioning and an inertia measurement unit, and aims to solve the problem that the traditional method cannot recognize three-dimensional gestures with various attribute information.
The technical scheme adopted by the invention is that the method comprises the following steps:
defining a three-dimensional gesture based on four attribute information of a track form, a motion direction, a pen body posture and a space position;
(1) defining track form attribute information: the track form of the three-dimensional gesture defines the form of a space track formed by the movement of a pen point in the process of completing gesture actions of a user, the attribute is a basic attribute of all three-dimensional gestures, and typical track forms comprise line segments, broken lines and closed geometric figures;
(2) defining motion direction attribute information: the movement direction of the pen body is the speed direction of the pen body relative to an external reference system in the action process of the pen gesture, and the definition of the three-dimensional pen gesture is perfectly supplemented through a movement direction attribute information department;
(3) defining pen body posture attribute information: the posture of the pen body of the three-dimensional pen gesture defines the posture change among different pen gestures and the posture change condition of the electronic pen in the pen gesture pointing process;
(4) defining spatial location attribute information: the spatial position of the three-dimensional gesture defines spatial region information of the current gesture action;
and (II) sequentially identifying each attribute information of the three-dimensional gesture according to the priority of the track form, the motion direction, the gesture of the pen body and the spatial position, and finishing gesture identification.
In the second step of the invention, for each attribute information, a proper data channel and an identification algorithm are selected, and the specific steps are as follows:
(1) according to an ultrasonic three-dimensional positioning coordinate sequence, hereinafter referred to as an ultrasonic coordinate, coding the ultrasonic coordinate sequence by using a normalized center distance coding method, then matching the similarity between a sample sequence and a template sequence by using sequence matching methods such as a dynamic time warping algorithm, and the like, and determining the track form attribute information of the sample according to the similarity between the sample and different templates;
(2) acquiring three-axis acceleration information, estimating motion velocity components of the pen body in three axes of X, Y and Z in the process of the pen gesture on the basis, and further determining the motion direction attribute information of the pen gesture;
(3) calculating the variance of each attitude angle sequence based on the attitude angle sequence in the stroke process, and setting a proper threshold value to identify the attitude attribute information of the pen body;
(4) identifying the spatial position attribute information of the occurrence of the gesture action: and identifying the attribute information of the space region where the gesture action occurs by comparing the distances from each point on the two sides of the interface to the center of the interface.
In the second step of the invention, (1) is as follows:
1) taking an ultrasonic coordinate sequence c [ m ]]＝c1,c2,...,ci,...,cMM is more than or equal to 1 and less than or equal to M, wherein ci＝(xi,yi,zi) The ultrasonic coordinates of the spatial position of the pen point in any non-repeated sampling are obtained, and M is the sequence length;
2) using c [ m ]]Calculating the center of the trajectory cc＝(xc,yc,zc) And three-dimensional coordinates of a track central point are as follows:
Figure BDA0002485308970000031
3) calculating track coordinate sequence c [ m ] in sequence]Inner group of ultrasonic coordinates ci＝(xi,yi,zi)TTo the central point a of the trackc＝(xc,yc,zc)TEuclidean distance of (a):
Figure BDA0002485308970000032
4) obtaining a track center distance coding sequence:
d[m]＝d1,d2,...,dM
5) calculating the total length L of the track from the starting point c1(x1,y1,z1) Calculating the linear distance between two adjacent points in the sequence in sequence and accumulating, wherein the accumulated distance can be approximately considered as the total track length:
Figure BDA0002485308970000033
6) the full length L of the track is utilized to carry out normalization processing on the coding sequence d [ m ] of the central distance of the track, and a coding sequence s [ m ] of the central distance of the normalized track is obtained:
Figure BDA0002485308970000041
7) and matching the sample sequence with the template sequence by using a dynamic time warping algorithm, and identifying the sample track form through the relative size of the shortest warping distance.
In step two of the invention, (2) the Y-axis and Z-axis processing steps are the same as those of the X-axis, taking the X-axis acceleration sequence as an example, the steps are as follows:
1) setting the X-axis acceleration sequence corresponding to the writing gesture process as ax [ n ]]＝ax1,ax2,...,axNWherein axiThe acceleration of X axis generated by the movement of the electronic pen at any sampling point, N is the sequence length, and ax is acquired when the known sampling frequency is fixed at 60HziAnd axi+1The time interval between two data points is
Figure BDA0002485308970000042
2) The triaxial speeds are all 0 at the starting moment of the pen gesture movement, then at t1The moment can be approximated as the instantaneous velocity component of the electronic pen motion on the X-axis:
vx1＝ax1·Δt
3) the approximation is considered from t-0 to t-t1In the time period, the movement speeds of the electronic pen on the X axis are vx1For the next time t2Then, there are:
vx2＝ax2·Δt+vx1
4) approximation is considered to be when t is t1To t ═ t2In the time period, the movement speeds of the electronic pen on the X axis are vx2By analogy, the speed component sequence of the electronic pen in the X-axis direction during the gesture motion can be obtained as follows:
vx[n]＝vx1,vx2,...,vxN
5) calculating the sequence mean value to obtain the average speed vx of the electronic pen in the X-axis directionave：
Figure BDA0002485308970000043
6) Similarly, the average speed of the Y-axis and the Z-axis generated by the motion of the electronic pen can be obtained and respectively recorded as vyave,vzave；
7) The axial direction of the pen body movement is determined by comparing the modes of the three-axis average speed, and the judgment relation is as follows:
if max (| vx)ave|,|vyave|,|vzave|)＝|vxaveIf yes, the electronic pen moves along the X axial direction;
if max (| vx)ave|,|vyave|,|vzave|)＝|vyaveIf yes, the electronic pen moves along the Y axis;
if max (| vx)ave|,|vyave|,|vzave|)＝|vzaveAnd | the electronic pen moves along the Z-axis.
8) The positive direction and the negative direction of the movement of the pen body are determined through the axial speed value, and the judgment relationship is that the electronic pen moves along the X axial direction as an example:
if vxaveIf the motion direction of the electronic pen is more than or equal to 0, the motion direction of the electronic pen is the positive direction of the X axis;
if vxaveIf the motion direction is less than 0, the motion direction of the electronic pen is the negative direction of the X axis.
The step two (3) of the invention comprises the following steps:
1) setting a pitch angle (pitch) theta sequence, a yaw angle (yaw) psi sequence and a roll angle (roll) sequence of the posture of the pen body in the pen gesture process as follows:
θ[n]＝θ1,θ2,..,θN，ψ[n]＝ψ1,ψ2,...,ψN，φ[n]＝φ1,φ2,...,φN
2) respectively calculating the variance of the attitude angle sequence to obtain Vθ，Vψ，Vφ；
3) Average value of substitution number to obtain Vave：
Figure BDA0002485308970000051
4) Setting a threshold value VthThe pen body translation or rotation identification method is used for identifying translation or rotation of the pen body, and the judgment relation is as follows:
if Vave≥VthThen the gesture process is considered.
In the second step (4) of the present invention, taking the spatial position as the left side and the right side of the interface as an example, the specific steps for distinguishing the spatial position attribute information are as follows:
1) taking a coordinate sequence c [ m ] of a sample to be identified]＝c1,c2,...,ci,...,cMJudging whether the point is positioned on the left side or the right side of the interface for each point in sequence, and when the coordinate point just falls on the interface, defaulting that the point is positioned on the left side of the interface;
2) definition MlNumber of points falling on the left side of the interface, MrNumber of points falling to the right of the interface, Ml+MrGet new sequence e [ M ═ Ml]And f [ m ]r]Wherein e [ m ]l]Are each c [ m ]]All points falling internally on the left side of the interface; according to a sequence of samples arranged in sequence, fmr]Is c [ m ]]All points falling on the right side of the interface are arranged in sequence according to sampling;
3) calculate e [ m ]l]And f [ m ]r]All points in both sequences to the center of the interface O (x)o,yo,zo) Is recorded as sum of distances ofl,sumr：
Figure BDA0002485308970000052
Figure BDA0002485308970000053
4) According to sumlAnd sumrThe size relationship of (2) determines the spatial position of the gesture action, and the determination relationship is as follows:
if suml≤sumrIf the gesture action is detected, the gesture action is considered to occur on the right side of the interface;
if suml＞sumrThen the gesture action is considered to occur on the left side of the interface.
Based on multi-channel information of ultrasonic coordinates, three-axis acceleration data and pen body posture data, a three-dimensional pen posture defining method which takes a track form as a basic attribute and takes a motion direction, a pen body posture and a space position as supplementary attributes is provided, and the characteristics of large interaction space, six degrees of freedom and various pen holding postures of the three-dimensional pen posture are fully considered, so that the three-dimensional pen posture interaction requirement can be better met. The three-dimensional handwriting recognition method provided by the invention gradually judges the attribute information of the three-dimensional handwriting according to the priority, selects the corresponding data channel according to the currently judged handwriting attribute, and designs the recognition method by combining the data characteristics to finish handwriting recognition.
The invention has the advantages that: a three-dimensional gesture definition method based on track morphology, motion direction, pen body posture and space position is provided, and relevant information of the four attributes is described by introducing multi-channel data of an ultrasonic positioning and inertia measurement unit. The three-dimensional gesture defined by the method can better meet the characteristics of large interaction space, six degrees of freedom and various pen holding postures of three-dimensional gesture interaction. The three-dimensional gesture recognition method provided by the invention sequentially recognizes the track form, the motion direction, the pen body posture and the spatial position information according to the priority, and selects a proper data mode and a proper recognition method according to the current recognition content. Compared with a typical machine learning method, the method provided by the invention has the advantages of no need of a large number of training samples, low computation amount, good real-time performance, strong interpretability and the like. Compared with the existing gesture recognition scheme, the three-dimensional gesture recognition method provided by the invention has fewer constraints on gesture actions (common constraints such as track proportion, action orientation and the like), and has better applicability.
Drawings
FIG. 1 is a diagram of the trace of a line segment in a three-dimensional gesture according to the present invention;
FIG. 2 is a diagram of a circular trajectory in a three-dimensional gesture according to the present invention;
FIG. 3 is a diagram of the V-shaped trajectory in the three-dimensional gesture according to the present invention;
FIG. 4 is a three-dimensional gesture motion pattern of the present invention;
FIG. 5 is a diagram of the translational posture of the pen body in the three-dimensional gesture according to the present invention;
FIG. 6 is a diagram illustrating the rotational posture of the pen body in the three-dimensional gesture according to the present invention;
FIG. 7 is a three-dimensional position diagram of the pen posture space in the present invention.
Detailed Description
Comprises the following steps:
defining a three-dimensional gesture based on four attribute information of a track form, a motion direction, a pen body posture and a space position;
(1) the track form of the three-dimensional gesture defines the form of a space track formed by the movement of a pen point in the process of completing gesture actions of a user. In the specific implementation process of the invention, three track forms are defined as a line segment track, a circular track and a V-shaped track respectively, as shown in figure 1. And in the aspect of interactive intention, the track form is used as the basic attribute of the three-dimensional gesture and expresses the basic interactive intention of the gesture. For example, in a typical three-dimensional interactive teaching scene, a line segment trajectory gesture may represent character movement and view angle control, a circular trajectory gesture may represent a related operation on the current interactive content, and a V-shaped trajectory gesture may represent a reset on the current interactive content or scene.
(2) The motion direction of the three-dimensional gesture defines the motion direction of the pen body of the electronic pen relative to an external reference system in the gesture motion process. In the practice of the present invention, six motion directions are defined as + X, -X, + Y, -Y, + Z, -Z, respectively, and the reference system is shown in FIG. 2, wherein the + Z direction is perpendicular to the paper surface. In the aspect of interaction intents, the interaction intents related to the multi-expression direction of the track direction attributes and the visual angle control can be regarded as supplements to the basic interaction intents expressed by the track form attributes. For example, line segment trajectory-like gestures express movement and perspective control within a three-dimensional interactive scene. On the basis, the movement direction of the pen body further expresses the specific visual angle moving direction.
(3) The posture of the pen body of the three-dimensional pen gesture defines the posture change among different pen gestures and the posture change condition of the electronic pen in the pen gesture pointing process. In the specific implementation process of the invention, two pen body postures are defined as pen body translation and pen body rotation respectively, as shown in fig. 3. In the aspect of interaction intention, the posture attribute of the pen body can supplement and enrich the basic interaction intention expressed by the track form attribute. For example, a circular track type gesture expression is used for operating the current interactive object, on the basis, the pen body translation can represent that the current object is rotated, and the pen body rotation gesture can represent that the current object is subjected to mirror image overturning operation.
(4) The spatial position of the three-dimensional gesture defines spatial region information of the current gesture action. In the implementation process of the invention, the selected reference object is an electronic whiteboard, and two spatial positions are defined as the left area and the right area of the whiteboard respectively, as shown in fig. 4. And at the interaction intention level, the interaction intention expressed by the spatial position attribute is related to the selection of the interaction object in each subspace. For example, the circular track type gesture of the translation of the pen body represents the rotation operation of the object in the interactive space, and the subspace information of the gesture motion represents the specifically selected interactive object.
Sequentially identifying each attribute information of the three-dimensional gesture according to the track form, the motion direction, the gesture of the pen body and the priority of the space position, and selecting a proper data channel and an identification method according to the current identification content to finish gesture identification;
the identification method relates to a sequence matching template, a template generation mode is described by taking a circular track in fig. 1 as an example, and corresponding template sequences can be obtained by adopting the same method for a line segment track and a V-shaped track, and the method comprises the following specific steps:
1) setting an arbitrary circular track, wherein for the sake of convenience of calculation, the circular track is not assumed to be located on the XoY plane, the center of the circle is (0,0), and the radius is r;
2) setting template sequence length LiTaking L on the circumferenceiDividing points equally;
3) meterCalculating the normalized central distance from each equant point to the center of the circle,
Figure BDA0002485308970000071
4) obtaining a template sequence T of circular trajectories1[Li]＝d1,d2,...,dLi
(1) According to the ultrasonic coordinate sequence, a track form is identified by using a normalized center distance coding method and a dynamic time warping method:
1) taking an ultrasonic coordinate sequence c [ m ]]＝c1,c2,...,ci,...,cMM is more than or equal to 1 and less than or equal to M, wherein ci＝(xi,yi,zi) The ultrasonic coordinates of the spatial position of the pen point in any non-repeated sampling are obtained, and M is the sequence length.
2) Using c [ m ]]Calculating the center of the trajectory cc＝(xc,yc,zc) And three-dimensional coordinates of a track central point are as follows:
Figure BDA0002485308970000081
3) calculating track coordinate sequence c [ m ] in sequence]Inner group of ultrasonic coordinates ci＝(xi,yi,zi)TTo the central point a of the trackc＝(xc,yc,zc)TEuclidean distance of (a):
Figure BDA0002485308970000082
4) obtaining a track center distance coding sequence:
d[m]＝d1,d2,...,dM
5) calculating the total length L of the track from the starting point c1(x1,y1,z1) Calculating the linear distance between two adjacent points in the sequence in sequence and accumulating, wherein the accumulated distance can be approximately considered as the total track length:
Figure BDA0002485308970000083
6) the full length L of the track is utilized to carry out normalization processing on the coding sequence d [ m ] of the central distance of the track, and a coding sequence s [ m ] of the central distance of the normalized track is obtained:
Figure BDA0002485308970000084
7) matching the sample sequence with the template sequence by using a dynamic time warping algorithm, and identifying the sample track form through the relative size of the shortest warping distance;
(2) acquiring three-axis acceleration information, identifying the motion direction of the pen body by using a pen body motion speed estimation method by taking an X-axis acceleration sequence as an example, and processing steps of a Y axis and a Z axis are the same as those of the X axis, wherein the steps are as follows:
1) setting the X-axis acceleration sequence corresponding to the writing gesture process as ax [ n ]]＝ax1,ax2,...,axNWherein axiThe acceleration of the X axis generated by the motion of the electronic pen at any sampling point, and N is the length of the sequence. Ax is collected knowing that the sampling frequency is fixed at 60HziAnd axi+1The time interval between two data points is
Figure BDA0002485308970000085
2) The triaxial speeds are all 0 at the starting moment of the pen gesture movement, then at t1The moment can be approximated as the instantaneous velocity component of the electronic pen motion on the X-axis:
vx1＝ax1·Δt
3) the approximation is considered from t-0 to t-t1In the time period, the movement speeds of the electronic pen on the X axis are vx1For the next time t2Then, there are:
vx2＝ax2·Δt+vx1
4) approximation is considered to be when t is t1To t ═ t2The moving speed of the electronic pen in the X axis in the time periodAre all vx2By analogy, the speed component sequence of the electronic pen in the X-axis direction during the gesture motion can be obtained as follows:
vx[n]＝vx1,vx2,...,vxN
5) calculating the sequence mean value to obtain the average speed vx of the electronic pen in the X-axis directionave：
Figure BDA0002485308970000091
6) Similarly, the average speed of the Y-axis and the Z-axis generated by the motion of the electronic pen can be obtained and respectively recorded as vyave,vzave。
7) The axial direction of the pen body movement is determined by comparing the modes of the three-axis average speed, and the judgment relation is as follows:
if max (| vx)ave|,|vyave|,|vzave|)＝|vxaveIf yes, the electronic pen moves along the X axial direction;
if max (| vx)ave|,|vyave|,|vzave|)＝|vyaveIf yes, the electronic pen moves along the Y axis;
if max (| vx)ave|,|vyave|,|vzave|)＝|vzaveAnd | the electronic pen moves along the Z-axis.
8) The positive direction and the negative direction of the movement of the pen body are determined through the axial speed value, and the judgment relationship is that the electronic pen moves along the X axial direction as an example:
if vxaveIf the motion direction of the electronic pen is more than or equal to 0, the motion direction of the electronic pen is the positive direction of the X axis;
if vxaveIf the motion direction is less than 0, the motion direction of the electronic pen is the negative direction of the X axis.
(3) Based on the posture angle sequence in the stroke process, calculating the variance of each posture angle sequence, and setting a proper threshold value to identify the posture of the pen body, the method specifically comprises the following steps:
1) setting a pitch angle (pitch) theta sequence, a yaw angle (yaw) psi sequence and a roll angle (roll) sequence of the posture of the pen body in the pen gesture process as follows:
θ[n]＝θ1,θ2,..,θN，ψ[n]＝ψ1,ψ2,...,ψN，φ[n]＝φ1,φ2,...,φN
2) respectively calculating the variance of the attitude angle sequence to obtain Vθ，Vψ，Vφ；
3) Average value of substitution number to obtain Vave：
Figure BDA0002485308970000092
4) Setting a threshold value VthThe pen body translation or rotation identification method is used for identifying translation or rotation of the pen body, and the judgment relation is as follows:
if Vave≥VthThe pen body is considered to rotate in the process of the gesture;
if Vave＜VthAnd the pen body is considered to be in translation in the writing process.
(4) Identifying the spatial position of the occurrence of the gesture action: and identifying the space region where the gesture action occurs by comparing the distances from each point on the two sides of the interface to the center of the interface. Taking the spatial position divided into the left side and the right side of the interface as an example, the specific steps for distinguishing the spatial position are as follows:
1) taking a coordinate sequence c [ m ] of a sample to be identified]＝c1,c2,...,ci,...,cMWherein M is the sequence length. Sequentially judging whether the point is positioned on the left side or the right side of the interface for each point, and when the coordinate point just falls on the interface, defaulting that the point is positioned on the left side of the interface;
2) definition MlNumber of points falling on the left side of the interface, MrNumber of points falling to the right of the interface, Ml+MrM. Obtaining a new sequence e [ m ]l]And f [ m ]r]Wherein e [ m ]l]Are each c [ m ]]All points falling internally on the left side of the interface; according to a sequence of samples arranged in sequence, fmr]Is c [ m ]]All points falling on the right side of the interface are arranged in sequence according to sampling;
3) calculate e [ m ]l]And f [ m ]r]All points in both sequences to the center of the interface O (x)o,yo,zo) Is recorded as sum of distances ofl,sumr：
Figure BDA0002485308970000101
Figure BDA0002485308970000102
4) According to sumlAnd sumrThe size relationship of (2) determines the spatial position of the gesture action, and the determination relationship is as follows:
if suml≤sumrIf the gesture action is detected, the gesture action is considered to occur on the right side of the interface;
if suml＞sumrThen the gesture action is considered to occur on the left side of the interface.

Claims (5)

1. A three-dimensional gesture recognition method integrating ultrasonic positioning and an inertia measurement unit is characterized by comprising the following steps of: comprises the following steps:
defining a three-dimensional gesture based on four attribute information of a track form, a motion direction, a pen body posture and a space position;
(1) defining track form attribute information: the track form of the three-dimensional gesture defines the form of a space track formed by the movement of a pen point in the process of completing gesture actions of a user, the attribute is a basic attribute of all three-dimensional gestures, and typical track forms comprise line segments, broken lines and closed geometric figures;
(2) defining motion direction attribute information: the movement direction of the pen body is the speed direction of the pen body relative to an external reference system in the action process of the pen gesture, and the definition of the three-dimensional pen gesture is perfectly supplemented through a movement direction attribute information department;
(3) defining pen body posture attribute information: the posture of the pen body of the three-dimensional pen gesture defines the posture change among different pen gestures and the posture change condition of the electronic pen in the pen gesture pointing process;
(4) defining spatial location attribute information: the spatial position of the three-dimensional gesture defines spatial region information of the current gesture action;
sequentially identifying each attribute information of the three-dimensional pen gesture according to the priority of the track form, the motion direction, the pen body posture and the space position to finish pen gesture identification, and selecting a proper data channel and an identification algorithm for each attribute information, wherein the specific steps are as follows:
(1) according to an ultrasonic three-dimensional positioning coordinate sequence, hereinafter referred to as an ultrasonic coordinate, coding the ultrasonic coordinate sequence by using a normalized center distance coding method, then matching the similarity between a sample sequence and a template sequence by using sequence matching methods such as a dynamic time warping algorithm, and the like, and determining the track form attribute information of the sample according to the similarity between the sample and different templates;
(2) acquiring three-axis acceleration information, estimating motion velocity components of the pen body in three axes of X, Y and Z in the process of the pen gesture on the basis, and further determining the motion direction attribute information of the pen gesture;
(3) calculating the variance of each attitude angle sequence based on the attitude angle sequence in the stroke process, and setting a proper threshold value to identify the attitude attribute information of the pen body;
(4) identifying the spatial position attribute information of the occurrence of the gesture action: and identifying the attribute information of the space region where the gesture action occurs by comparing the distances from each point on the two sides of the interface to the center of the interface.
2. The three-dimensional gesture recognition method of the fusion ultrasound localization and inertial measurement unit of claim 1, characterized in that: the step (1) of the second step is as follows:
1) taking an ultrasonic coordinate sequence c [ m ]]＝c1,c2,...,ci,...,cMM is more than or equal to 1 and less than or equal to M, wherein ci＝(xi,yi,zi) The ultrasonic coordinates of the spatial position of the pen point in any non-repeated sampling are obtained, and M is the sequence length;
2) using c [ m ]]Calculating the center of the trajectory cc＝(xc,yc,zc) And three-dimensional coordinates of a track central point are as follows:
Figure FDA0002995618630000021
3) calculating track coordinate sequence c [ m ] in sequence]Inner group of ultrasonic coordinates ci＝(xi,yi,zi)TTo the central point a of the trackc＝(xc,yc,zc)TEuclidean distance of (a):
Figure FDA0002995618630000022
4) obtaining a track center distance coding sequence:
d[m]＝d1,d2,...,dM
5) calculating the total length L of the track from the starting point c1(x1,y1,z1) Calculating the linear distance between two adjacent points in the sequence in sequence and accumulating, wherein the accumulated distance can be approximately considered as the total track length:
Figure FDA0002995618630000023
6) the full length L of the track is utilized to carry out normalization processing on the coding sequence d [ m ] of the central distance of the track, and a coding sequence s [ m ] of the central distance of the normalized track is obtained:
Figure FDA0002995618630000024
7) and matching the sample sequence with the template sequence by using a dynamic time warping algorithm, and identifying the sample track form through the relative size of the shortest warping distance.
3. The three-dimensional gesture recognition method of the fusion ultrasound localization and inertial measurement unit of claim 1, characterized in that: the Y-axis and Z-axis processing steps in the step (2) are the same as those in the X-axis, and the step is as follows by taking an X-axis acceleration sequence as an example:
1) setting the X-axis acceleration sequence corresponding to the writing gesture process as ax [ n ]]＝ax1,ax2,...,axNWherein axiThe acceleration of X axis generated by the movement of the electronic pen at any sampling point, N is the sequence length, and ax is acquired when the known sampling frequency is fixed at 60HziAnd axi+1The time interval between two data points is
Figure FDA0002995618630000025
2) The triaxial speeds are all 0 at the starting moment of the pen gesture movement, then at t1The moment can be approximated as the instantaneous velocity component of the electronic pen motion on the X-axis:
vx1＝ax1·△t
3) the approximation is considered from t-0 to t-t1In the time period, the movement speeds of the electronic pen on the X axis are vx1For the next time t2Then, there are:
vx2＝ax2·△t+vx1
4) approximation is considered to be when t is t1To t ═ t2In the time period, the movement speeds of the electronic pen on the X axis are vx2By analogy, the speed component sequence of the electronic pen in the X-axis direction during the gesture motion can be obtained as follows:
vx[n]＝vx1,vx2,...,vxN
5) calculating the sequence mean value to obtain the average speed vx of the electronic pen in the X-axis directionave：
Figure FDA0002995618630000031
6) Similarly, the average speed of the Y-axis and the Z-axis generated by the motion of the electronic pen can be obtained and respectively recorded as vyave,vzave；
7) The axial direction of the pen body movement is determined by comparing the modes of the three-axis average speed, and the judgment relation is as follows:
if max (| vx)ave|,|vyave|,|vzave|)＝|vxaveIf yes, the electronic pen moves along the X axial direction;
if max (| vx)ave|,|vyave|,|vzave|)＝|vyaveIf yes, the electronic pen moves along the Y axis;
if max (| vx)ave|,|vyave|,|vzave|)＝|vzaveIf yes, the electronic pen moves along the Z axial direction;
8) the positive direction and the negative direction of the movement of the pen body are determined through the axial speed value, and the judgment relationship is that the electronic pen moves along the X axial direction as an example:
if vxaveIf the motion direction of the electronic pen is more than or equal to 0, the motion direction of the electronic pen is the positive direction of the X axis;
if vxaveIf the motion direction is less than 0, the motion direction of the electronic pen is the negative direction of the X axis.
4. The three-dimensional gesture recognition method of the fusion ultrasound localization and inertial measurement unit of claim 1, characterized in that: the step (3) of the second step comprises the following specific steps:
1) setting a pitch angle (pitch) theta sequence, a yaw angle (yaw) psi sequence and a roll angle (roll) sequence of the posture of the pen body in the pen gesture process as follows:
θ[n]＝θ1,θ2,..,θN，ψ[n]＝ψ1,ψ2,...,ψN，φ[n]＝φ1,φ2,...,φN
2) respectively calculating the variance of the attitude angle sequence to obtain Vθ，Vψ，Vφ；
3) Average value of substitution number to obtain Vave：
Figure FDA0002995618630000032
4) Setting a threshold value VthThe pen body translation or rotation identification method is used for identifying translation or rotation of the pen body, and the judgment relation is as follows:
if Vave≥VthThen the gesture process is considered.
5. The three-dimensional gesture recognition method of the fusion ultrasound localization and inertial measurement unit of claim 1, characterized in that: taking the spatial position divided into the left side and the right side of the interface as an example in the step (4), the specific steps of distinguishing the spatial position attribute information are as follows:
1) taking a coordinate sequence c [ m ] of a sample to be identified]＝c1,c2,...,ci,...,cMJudging whether the point is positioned on the left side or the right side of the interface for each point in sequence, and when the coordinate point just falls on the interface, defaulting that the point is positioned on the left side of the interface;
2) definition MlNumber of points falling on the left side of the interface, MrNumber of points falling to the right of the interface, Ml+MrGet new sequence e [ M ═ Ml]And f [ m ]r]Wherein e [ m ]l]Are each c [ m ]]All points falling internally on the left side of the interface; according to a sequence of samples arranged in sequence, fmr]Is c [ m ]]All points falling on the right side of the interface are arranged in sequence according to sampling;
3) calculate e [ m ]l]And f [ m ]r]All points in both sequences to the center of the interface O (x)o,yo,zo) Is recorded as sum of distances ofl,sumr：
Figure FDA0002995618630000041
Figure FDA0002995618630000042
4) According to sumlAnd sumrThe size relationship of (2) determines the spatial position of the gesture action, and the determination relationship is as follows:
if suml≤sumrIf the gesture action is detected, the gesture action is considered to occur on the right side of the interface;
if suml>sumrThen, the gesture is considered as the strokeThe action takes place to the left of the interface.
CN202010389693.3A 2020-05-10 2020-05-10 Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit Active CN111580686B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010389693.3A CN111580686B (en) 2020-05-10 2020-05-10 Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010389693.3A CN111580686B (en) 2020-05-10 2020-05-10 Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit

Publications (2)

Publication Number Publication Date
CN111580686A CN111580686A (en) 2020-08-25
CN111580686B true CN111580686B (en) 2021-07-20

Family

ID=72113502

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010389693.3A Active CN111580686B (en) 2020-05-10 2020-05-10 Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit

Country Status (1)

Country Link
CN (1) CN111580686B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107102750A (en) * 2017-04-23 2017-08-29 吉林大学 The system of selection of target in a kind of virtual three-dimensional space based on pen type interactive system
CN107102749A (en) * 2017-04-23 2017-08-29 吉林大学 A kind of three-dimensional pen type localization method based on ultrasonic wave and inertial sensor
CN107122109A (en) * 2017-05-31 2017-09-01 吉林大学 A kind of multi-channel adaptive operating method towards three-dimensional pen-based interaction platform
CN110647282A (en) * 2019-09-18 2020-01-03 中北大学 Handwritten track information acquisition method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107102750A (en) * 2017-04-23 2017-08-29 吉林大学 The system of selection of target in a kind of virtual three-dimensional space based on pen type interactive system
CN107102749A (en) * 2017-04-23 2017-08-29 吉林大学 A kind of three-dimensional pen type localization method based on ultrasonic wave and inertial sensor
CN107122109A (en) * 2017-05-31 2017-09-01 吉林大学 A kind of multi-channel adaptive operating method towards three-dimensional pen-based interaction platform
CN110647282A (en) * 2019-09-18 2020-01-03 中北大学 Handwritten track information acquisition method

Also Published As

Publication number Publication date
CN111580686A (en) 2020-08-25

Similar Documents

Publication Publication Date Title
US7340077B2 (en) Gesture recognition system using depth perceptive sensors
US10242255B2 (en) Gesture recognition system using depth perceptive sensors
US9959463B2 (en) Gesture recognition system using depth perceptive sensors
JP6941198B2 (en) Gesture recognition methods, gesture recognition devices, electronic devices, and storage media
Wu et al. Hand modeling, analysis and recognition
US20070124703A1 (en) Command input method using motion recognition device
Feng et al. Real-time fingertip tracking and detection using Kinect depth sensor for a new writing-in-the air system
CN110232308B (en) Robot-following gesture track recognition method based on hand speed and track distribution
CN109993073A (en) A kind of complicated dynamic gesture identification method based on Leap Motion
Jeen-Shing et al. Online handwriting recognition using an accelerometer-based pen device
CN104850773A (en) User identity authentication method for intelligent mobile terminal
WO2012163124A1 (en) Spatial motion-based input method and terminal
Francis et al. Significance of hand gesture recognition systems in vehicular automation-a survey
Abualola et al. Flexible gesture recognition using wearable inertial sensors
CN107292295B (en) Gesture segmentation method and device
Patil et al. Handwriting recognition in free space using WIMU‐based hand motion analysis
CN107346207B (en) Dynamic gesture segmentation recognition method based on hidden Markov model
Li et al. A novel hand gesture recognition based on high-level features
Alam et al. Trajectory-based air-writing character recognition using convolutional neural network
Liu et al. Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition
Sooai et al. Comparison of recognition accuracy on dynamic hand gesture using feature selection
Iyer et al. Generalized hand gesture recognition for wearable devices in IoT: Application and implementation challenges
CN111580686B (en) Three-dimensional stroke recognition method integrating ultrasonic positioning and inertial measurement unit
Yang et al. A novel hand gesture input device based on inertial sensing technique
Chen et al. A fusion recognition method based on multifeature hidden markov model for dynamic hand gesture

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant