CN112486331A - IMU-based three-dimensional space handwriting input method and device - Google Patents
IMU-based three-dimensional space handwriting input method and device Download PDFInfo
- Publication number
- CN112486331A CN112486331A CN202011506838.XA CN202011506838A CN112486331A CN 112486331 A CN112486331 A CN 112486331A CN 202011506838 A CN202011506838 A CN 202011506838A CN 112486331 A CN112486331 A CN 112486331A
- Authority
- CN
- China
- Prior art keywords
- imu
- information
- user
- dimensional
- skeleton
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
- G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The application provides a three-dimensional space handwriting input method and device based on an IMU, and relates to the technical field of artificial intelligence, wherein the method comprises the following steps: acquiring a plurality of pieces of inertia information acquired by an Inertial Measurement Unit (IMU) sensor at each position of a user; processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user; and solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information. Therefore, a user can draw in the air by hands and watch the drawn three-dimensional track in real time to realize handwriting input, and the user requirements are met.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a three-dimensional space handwriting input method and apparatus based on an IMU (Inertial Measurement Unit).
Background
Since devices for displaying handwriting, such as screens, papers, blackboards, and the like, are two-dimensional, there are various two-dimensional space handwriting input modes at present, and with the development of Virtual Reality, Augmented Reality (VR, Augmented Reality) technology, and computer graphics, the display of objects in a three-dimensional space becomes increasingly simple and real. At present, most of handwriting input devices write on a plane and do not have direct three-dimensional drawing capability.
In the related art, methods for recording a three-dimensional handwriting track include using various high-precision distance sensors (ultrasonic distance sensors, laser sensors, etc.), or using a depth camera, a multi-camera system, etc., which require an external base station or camera and have high requirements on the use environment (such as lighting requirements, shielding requirements, placement of the camera or the base station, etc.). Deployment of such a system is time consuming, labor intensive, costly and power intensive.
Disclosure of Invention
The present application is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, a first objective of the present application is to provide an IMU-based three-dimensional space handwriting input method to achieve increased user experience, reduced cost of an input system, and enhanced usability and robustness of the system.
A second object of the present application is to provide an IMU-based three-dimensional space handwriting input device.
In order to achieve the above object, an embodiment of a first aspect of the present application provides an IMU-based three-dimensional space handwriting input method, including:
acquiring a plurality of pieces of inertia information acquired by an Inertial Measurement Unit (IMU) sensor at each position of a user;
processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user;
and solving the skeletal chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information.
According to the IMU-based three-dimensional space handwriting input method, a plurality of pieces of inertia information acquired by an IMU sensor on each position of a user are acquired; processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user; and solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information. Therefore, a user can draw in the air by hands and watch the drawn three-dimensional track in real time to realize handwriting input, and the user requirements are met.
In one embodiment of the present application, the inertial measurement unit IMU sensors are mounted on any wrist, any upper arm and waist of the user, respectively;
and receiving a plurality of pieces of inertia information acquired by the inertial measurement unit IMU sensor through Bluetooth or wireless technology.
In one embodiment of the application, the rotational offset between IMU measurements and true bone rotation is determined for initial use by the transformation relationship between the IMU device coordinate system, IMU global inertial coordinate system, bone model global coordinate system, and each bone own coordinate system acquired by the user at the determined pose.
In an embodiment of the present application, before the processing the plurality of pieces of inertial information by the deep neural network, the method further includes:
checking the plurality of pieces of inertia information through a preset algorithm, and down-sampling the plurality of pieces of inertia information to a preset frequency; wherein the preset frequency formula is: f. ofs＝max(30,min(fIMU1,fIMU2,fIMU3))。
In an embodiment of the present application, the processing the plurality of inertial information through a deep neural network to obtain the bone chain posture information of the user includes:
and calculating a rotation matrix and acceleration of the skeleton according to the rotation deviation amount and the plurality of pieces of inertia information, inputting the rotation matrix and the acceleration into a recurrent neural network, estimating rotation information of joints on an arm skeleton chain, and acquiring posture information of the skeleton chain.
In an embodiment of the application, the solving the skeletal chain posture information through forward dynamics, obtaining a hand position of the user, and forming a three-dimensional handwriting track according to the hand position and historical position information includes:
calculating hand joint coordinates according to the skeleton chain posture information and known skeleton length information;
and forming the three-dimensional handwriting track according to the hand joint coordinates and the historical position information.
In order to achieve the above object, a second embodiment of the present application provides an IMU-based three-dimensional space handwriting input apparatus, including:
the acquisition module is used for acquiring a plurality of pieces of inertia information acquired by the IMU sensors at all positions of a user;
the processing module is used for processing the plurality of inertial information through a deep neural network to acquire the skeletal chain posture information of the user;
and the generation module is used for solving the skeletal chain posture information through forward dynamics, acquiring the hand position of the user and forming a three-dimensional handwriting track according to the hand position and the historical position information.
According to the IMU-based three-dimensional space handwriting input device, a plurality of pieces of inertia information acquired by an IMU sensor on each position of a user are acquired; processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user; and solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information. Therefore, a user can draw in the air by hands and watch the drawn three-dimensional track in real time to realize handwriting input, and the user requirements are met.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic flowchart of a three-dimensional space handwriting input method based on an IMU according to an embodiment of the present application;
FIG. 2 is an exemplary diagram of an IMU wearing position according to an embodiment of the present application;
fig. 3 is an exemplary diagram of an IMU-based three-dimensional space handwriting input method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an IMU-based three-dimensional space handwriting input apparatus according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
The method and apparatus for inputting three-dimensional space handwriting based on IMU according to the embodiments of the present application are described below with reference to the accompanying drawings.
According to the handwriting input method in the three-dimensional space, a processor such as a computer receives input data and processes the input data, and then the writing track can be displayed through various three-dimensional graphic software and VR/AR technologies.
Specifically, the inertial measurement is performed as input data by using three micro IMU sensors, i.e., inertial measurement units, which are commonly built in smart wearable devices (such as smart bands, watches, belts, glasses, etc.) and personal mobile devices (such as smart phones, etc.), and along with the popularization of smart wearable devices, the practicability of the IMU-based system is also greatly increased. The input system has no requirement on the type of the IMU, and any device capable of acquiring inertial information can be used for system input. After a user wears the IMU on the wrist, the upper arm, the waist and the like, the computer receives inertia information of the three sensors through Bluetooth or a wireless network, estimates the posture of an arm skeleton chain of the user in real time by using a deep learning method, and calculates a hand movement track. The trajectory may be rendered to a screen by three-dimensional graphics software or presented directly in a virtual three-dimensional space by a VR device. Therefore, the invention enables the user to draw in the air by hands and watch the drawn three-dimensional track in real time, and has great use value.
Fig. 1 is a flowchart illustrating a three-dimensional space handwriting input method based on an IMU according to an embodiment of the present application.
As shown in fig. 1, the method for inputting handwriting in three-dimensional space based on IMU includes the following steps:
In the embodiment of the application, the inertial measurement unit IMU sensors are respectively arranged on any wrist, any big arm and waist of a user; and receiving a plurality of pieces of inertia information acquired by an Inertial Measurement Unit (IMU) sensor through Bluetooth or wireless technology.
Specifically, inertial information collected by three IMU sensors bound on the wrist, the upper arm and the waist is subjected to data inspection and synchronous processing.
And 102, processing the plurality of inertial information through a deep neural network to obtain the skeletal chain posture information of the user.
And 103, solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information.
In an embodiment of the application, the rotational deviation between the IMU measurement value and the true bone rotation is determined by the transformation relationship between the IMU device coordinate system, the IMU global inertial coordinate system, the bone model global coordinate system, and each bone own coordinate system acquired by the user in the determined pose at the time of initial use.
In the embodiment of the application, a plurality of pieces of inertia information are checked through a preset algorithm, and the plurality of pieces of inertia information are sampled to a preset frequency; wherein, the preset frequency formula is as follows: f. ofs＝max(30,min(fIMU1,fIMU2,fIMU3))。
In the embodiment of the application, a rotation matrix and acceleration of a framework are calculated according to a rotation deviation amount and a plurality of pieces of inertia information and input into a recurrent neural network, the rotation information of joints on an arm skeleton chain is estimated, and the posture information of the skeleton chain is obtained.
In the embodiment of the application, hand joint coordinates are calculated according to the skeleton chain posture information and the known skeleton length information; and forming the three-dimensional handwriting track according to the hand joint coordinates and historical position information.
Specifically, the current hand position is estimated and recorded in real time for each frame. The method uses inertial information (including acceleration and angle information) obtained through Bluetooth or wireless technology, considers the information of all frames of the current frame and the history together, and estimates a skeletal chain posture from a hand to a waist through a deep learning method, wherein the skeletal chain posture comprises rotation information of all joints. Then the system solves the position of the hand by a forward dynamics method, and the hand position estimated in real time in each frame forms an input moving track. The method combines a rendering engine and a VR system, and can directly render the generated track in a two-dimensional or three-dimensional space.
Specifically, IMU data are collected under a determined posture, a transformation relation among an IMU equipment coordinate system, an IMU global inertia coordinate system, a skeleton model global coordinate system and each skeleton own coordinate system is utilized to determine a rotation deviation value between an IMU measurement value and real skeleton rotation, each frame of IMU data is calibrated and normalized, a deep circulation neural network is utilized, the posture of an arm skeleton chain is estimated through collected inertia information, the position of each frame of hand joint is solved in real time by a forward dynamics method, and a complete three-dimensional handwriting track is formed by combining historical position information.
Specifically, as shown in fig. 2, the IMU device is worn in a manner that allows the IMU to be placed in any direction, as long as it is worn near a designated location. It is desirable to avoid as much as possible the effects of muscle strain on the rotation of the IMU during exercise, i.e. to place the IMU near the skin (e.g. wrist, back or side of arm, back of waist) where large strain does not occur during exercise. The three-dimensional handwriting track can be recovered through the inertia information measured by the three IMUs.
Specifically, the three-dimensional handwriting input method shown in fig. 3 includes:
1) decoding verification and synchronization are performed on the read IMU measurements. Because three IMU devices are connected with a computer through a wireless technology to send data, the original acceleration and angle data of the IMU devices generally need to be decoded and checked, and firstly, the inertial information is decoded according to the type of the used IMU. In addition, for some IMUs which can only acquire the original data of the acceleration, the angular velocity and the magnetic field values, the angular values need to be firstly solved through algorithms such as complementary filtering or Kalman filtering. And synchronizing the data passing the data verification with the data of different IMUs (if the models of the three IMUs are different), specifically, selecting the frequency of the IMU with the lowest sampling frequency as the frequency of an input system (but not lower than 30Hz), and downsampling the data measured by other IMUs to the frequency to complete data synchronization. If the lowest frequency IMU frequency is below 30Hz, it is advisable to modify its frequency or use other IMUs, or to interpolate its measurements up to 30 Hz. Wherein the frequency f of the input systemsThe mathematical definition of (a) is as follows:
fs＝max(30,min(fIMU1,fIMU2,fIMU3)) (1)
2) a calibration matrix is calculated. Specifically, initial use requires the user to acquire IMU data at a certain pose (defined as the state of natural arm drop) and determine the amount of rotational deviation (which can be expressed as a constant rotation matrix) between IMU measurements and true bone rotation using the transformation relationships between the IMU device coordinate system, the IMU global inertial coordinate system, the bone model global coordinate system, and the coordinate systems of each bone itself.
Defining a skeleton model global coordinate system FMIMU global inertial coordinate system FIIMU device coordinate system FS. Require the user to align the x, y, z axes of the IMU sensors of the wrist portion with the right, top, front of the user's body, i.e., FSAnd FMAlignment, at which time the rotation measured by the sensor is FITo FMOf (2) a transition matrix
BIRIM＝BM (2)
Here the base matrix of the coordinate system is denoted by B. Subsequently, during the calibration process, the user is enabled to maintain a determined natural arm drop attitude, i.e. a determined joint rotation
(where i is the joint number), reading the mean of rotation of the IMU measurements
Can calculate the measured rotational deviation caused by the IMU placing rotation and the included angle between the muscle and the skeleton
3) And calibrating and normalizing IMU data, and estimating the posture of the arm skeleton chain by using a deep circulation neural network. Calculating the rotation of the skeleton by using the calibration matrix calculated in the calibration process and the rotation information acquired in the first step
Wherein
And
for the rotation matrix and the acceleration measured by the ith IMU, the rotation and the acceleration of the skeleton under the model coordinate system can be obtained, the values are normalized and input into a recurrent neural network, and the rotation information of the joint on the arm skeleton chain (the whole skeleton chain from the wrist joint to the human root joint) is estimated. The network may be trained using a motion data set with IMU data.
4) And solving the position of the hand joint of each frame in real time by applying a forward dynamics method. The network of the third step outputs the rotation of all joints on the arm skeleton chain of the current frame, and by using the rotation and the known skeleton length information, the position of the hand (namely the terminal joint) can be solved according to the forward dynamics. Specifically, if the relative rotation matrix of the ith joint in the skeleton chain is R(i)Then the homogeneous coordinate of the ith joint in this attitude should be
Wherein
Three-dimensional coordinates of the i-th joint in the attitude θ, and an processor (i) represents all ancestor joints of the i-th joint, n is the total number of joints,
represents the ith bone vector under the standard posture, and satisfies
Where θ ═ 0 denotes the standard posture, parent (i) denotes the parent joint of joint i. Therefore, the hand joint coordinates of the current frame can be calculated, and a complete three-dimensional handwriting track can be formed by combining the historical position information.
Therefore, the method has great significance for acquiring the hand movement track in the three-dimensional space. The method of the invention firstly processes the raw measurement data of the sensors, carries out data decoding and inspection and data synchronization work among multiple sensors. The user is then asked to maintain the natural arm drop attitude for one second for the calibration matrix to be calculated. And then calibrating the inertial data received by each frame by using a transformation matrix, normalizing, estimating the posture of the arm skeleton by using a recurrent neural network, and finally calculating the three-dimensional position of the hand by using forward dynamics. The method can acquire the hand movement track in real time and can be directly added into a three-dimensional scene.
The method of the embodiment can be realized on a common PC, and the use environment including requirements of illumination, shielding and the like is not limited; the type of IMU is not limited, and only three IMUs are needed; the placement angle of the IMU is not limited, and the IMU only needs to be worn on a specified position and subjected to a short calibration process.
From this, the user can wear behind the three miniature IMU equipment and draw a picture in sky bare-handed, and the system can render user's hand orbit in real time in three-dimensional scene to show through VR equipment or screen, this system does not have any requirement to service environment (like illumination, sheltering from), and sensor data transmits through bluetooth or wireless network, has high user-friendly degree.
According to the IMU-based three-dimensional space handwriting input method, a plurality of pieces of inertia information acquired by an IMU sensor on each position of a user are acquired; processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user; and solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information. Therefore, a user can draw in the air by hands and watch the drawn three-dimensional track in real time to realize handwriting input, and the user requirements are met.
In order to implement the above embodiments, the present application further provides an IMU-based three-dimensional space handwriting input device.
Fig. 4 is a schematic structural diagram of an IMU-based three-dimensional space handwriting input apparatus according to an embodiment of the present application.
As shown in fig. 4, the IMU-based three-dimensional space handwriting input apparatus includes: an acquisition module 410, a processing module 420, and a generation module 430.
The obtaining module 410 is configured to obtain a plurality of pieces of inertial information collected by the inertial measurement unit IMU sensor at each position of the user.
The processing module 420 is configured to process the plurality of inertial information through a deep neural network to obtain the bone chain posture information of the user.
The generating module 430 is configured to solve the skeleton chain posture information through forward dynamics, acquire a hand position of the user, and form a three-dimensional handwriting track according to the hand position and historical position information.
In the embodiment of the application, the inertial measurement unit IMU sensors are respectively installed on any wrist, any big arm and waist of the user; the acquisition module is specifically configured to: and receiving a plurality of pieces of inertia information acquired by the inertial measurement unit IMU sensor through Bluetooth or wireless technology.
In an embodiment of the present application, the apparatus for inputting handwriting in three-dimensional space based on IMU further includes:
and the determining module is used for determining the rotation deviation amount between the IMU measurement value and the real bone rotation according to the transformation relation among the IMU equipment coordinate system, the IMU global inertia coordinate system, the bone model global coordinate system and each bone self coordinate system which are acquired by the user in the determined posture during initial use.
In an embodiment of the present application, the processing module is specifically configured to: and calculating a rotation matrix and acceleration of the skeleton according to the rotation deviation amount and the plurality of pieces of inertia information, inputting the rotation matrix and the acceleration into a recurrent neural network, estimating rotation information of joints on an arm skeleton chain, and acquiring posture information of the skeleton chain.
In this embodiment of the application, solving the skeleton chain posture information through forward dynamics, obtaining a hand position of the user, and forming a three-dimensional handwriting track according to the hand position and historical position information includes: calculating hand joint coordinates according to the skeleton chain posture information and known skeleton length information; and forming the three-dimensional handwriting track according to the hand joint coordinates and the historical position information.
According to the IMU-based three-dimensional space handwriting input device, a plurality of pieces of inertia information acquired by an IMU sensor on each position of a user are acquired; processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user; and solving the skeleton chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information. Therefore, a user can draw in the air by hands and watch the drawn three-dimensional track in real time to realize handwriting input, and the user requirements are met.
It should be noted that the foregoing explanation on the embodiment of the method for inputting handwriting in three-dimensional space based on an IMU is also applicable to the apparatus for inputting handwriting in three-dimensional space based on an IMU in this embodiment, and is not repeated here.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. An IMU-based three-dimensional space handwriting input method is characterized by comprising the following steps:
acquiring a plurality of pieces of inertia information acquired by an Inertial Measurement Unit (IMU) sensor at each position of a user;
processing the plurality of inertial information through a deep neural network to obtain skeletal chain posture information of the user;
and solving the skeletal chain posture information through forward dynamics to obtain the hand position of the user, and forming a three-dimensional handwriting track according to the hand position and the historical position information.
2. The IMU-based three-dimensional spatial handwriting input method of claim 1, wherein said inertial measurement unit IMU sensors are mounted on any one of said user's wrist, arm and waist, respectively;
and receiving a plurality of pieces of inertia information acquired by the inertial measurement unit IMU sensor through Bluetooth or wireless technology.
3. The IMU-based three-dimensional space handwriting input method of claim 1, further comprising:
and during initial use, the user determines the rotation deviation between the IMU measurement value and the real skeleton rotation according to the transformation relation among the IMU equipment coordinate system, the IMU global inertia coordinate system, the skeleton model global coordinate system and each skeleton own coordinate system acquired under the determined posture.
4. The IMU-based three-dimensional spatial handwriting input method of claim 1, further comprising, prior to said processing said plurality of inertial information by the deep neural network:
checking the plurality of pieces of inertia information through a preset algorithm, and down-sampling the plurality of pieces of inertia information to a preset frequency; wherein the preset frequency formula is: f. ofs＝max(30,min(fIMU1,fIMU2,fIMU3))。
5. The IMU-based three-dimensional spatial handwriting input method of claim 3, wherein said processing said plurality of inertial information through a deep neural network to obtain skeletal chain pose information of said user comprises:
and calculating a rotation matrix and acceleration of the skeleton according to the rotation deviation amount and the plurality of pieces of inertia information, inputting the rotation matrix and the acceleration into a recurrent neural network, estimating rotation information of joints on an arm skeleton chain, and acquiring posture information of the skeleton chain.
6. The IMU-based three-dimensional space handwriting input method of claim 1, wherein said solving said skeletal chain pose information by forward dynamics, obtaining hand position of said user, and forming a three-dimensional handwriting trajectory from said hand position and historical position information comprises:
calculating hand joint coordinates according to the skeleton chain posture information and known skeleton length information;
and forming the three-dimensional handwriting track according to the hand joint coordinates and the historical position information.
7. An IMU-based three-dimensional space handwriting input apparatus, comprising:
the acquisition module is used for acquiring a plurality of pieces of inertia information acquired by the IMU sensors at all positions of a user;
the processing module is used for processing the plurality of inertial information through a deep neural network to acquire the skeletal chain posture information of the user;
and the generation module is used for solving the skeletal chain posture information through forward dynamics, acquiring the hand position of the user and forming a three-dimensional handwriting track according to the hand position and the historical position information.
8. The IMU-based three-dimensional spatial handwriting input apparatus of claim 7, wherein said inertial measurement unit IMU sensors are mounted on any one of said user's wrist, arm and waist, respectively; the acquisition module is specifically configured to:
and receiving a plurality of pieces of inertia information acquired by the inertial measurement unit IMU sensor through Bluetooth or wireless technology.
9. The IMU-based three-dimensional spatial handwriting input apparatus of claim 6, further comprising:
and the determining module is used for determining the rotation deviation amount between the IMU measurement value and the real bone rotation according to the transformation relation among the IMU equipment coordinate system, the IMU global inertia coordinate system, the bone model global coordinate system and each bone self coordinate system which are acquired by the user in the determined posture during initial use.
10. The IMU-based three-dimensional spatial handwriting input apparatus of claim 9, wherein the processing module is specifically configured to:
and calculating a rotation matrix and acceleration of the skeleton according to the rotation deviation amount and the plurality of pieces of inertia information, inputting the rotation matrix and the acceleration into a recurrent neural network, estimating rotation information of joints on an arm skeleton chain, and acquiring posture information of the skeleton chain.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011506838.XA CN112486331A (en) | 2020-12-18 | 2020-12-18 | IMU-based three-dimensional space handwriting input method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011506838.XA CN112486331A (en) | 2020-12-18 | 2020-12-18 | IMU-based three-dimensional space handwriting input method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486331A true CN112486331A (en) | 2021-03-12 |
Family
ID=74914659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011506838.XA Pending CN112486331A (en) | 2020-12-18 | 2020-12-18 | IMU-based three-dimensional space handwriting input method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486331A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113406646A (en) * | 2021-06-18 | 2021-09-17 | 北京师范大学 | Method and equipment for three-dimensional positioning based on multi-direction ultrasonic ranging and IMU (inertial measurement Unit) |
| CN113749651A (en) * | 2021-10-18 | 2021-12-07 | 长春理工大学 | Pressure evaluation method and pressure evaluation system based on human body posture recognition |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101655743A (en) * | 2008-08-18 | 2010-02-24 | 财团法人工业技术研究院 | Instant action identifying method and inertia sensing and track reconstructing device thereof |
| CN108876815A (en) * | 2018-04-28 | 2018-11-23 | 深圳市瑞立视多媒体科技有限公司 | Bone computation method for attitude, personage's dummy model driving method and storage medium |
| CN109102572A (en) * | 2018-07-27 | 2018-12-28 | 国网江苏省电力有限公司常州供电分公司 | Power transformation emulates virtual hand bone ratio in VR system and estimates method |
| CN109741371A (en) * | 2019-01-07 | 2019-05-10 | 西安理工大学 | A kind of human action based on inertial sensor describes method |
| CN110300542A (en) * | 2016-07-25 | 2019-10-01 | 开创拉布斯公司 | Use the method and apparatus of wearable automated sensor prediction muscle skeleton location information |
| CN112057077A (en) * | 2019-06-11 | 2020-12-11 | 本田技研工业株式会社 | Information processing apparatus, information processing method, and storage medium |
-
2020
- 2020-12-18 CN CN202011506838.XA patent/CN112486331A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101655743A (en) * | 2008-08-18 | 2010-02-24 | 财团法人工业技术研究院 | Instant action identifying method and inertia sensing and track reconstructing device thereof |
| CN110300542A (en) * | 2016-07-25 | 2019-10-01 | 开创拉布斯公司 | Use the method and apparatus of wearable automated sensor prediction muscle skeleton location information |
| CN108876815A (en) * | 2018-04-28 | 2018-11-23 | 深圳市瑞立视多媒体科技有限公司 | Bone computation method for attitude, personage's dummy model driving method and storage medium |
| CN109102572A (en) * | 2018-07-27 | 2018-12-28 | 国网江苏省电力有限公司常州供电分公司 | Power transformation emulates virtual hand bone ratio in VR system and estimates method |
| CN109741371A (en) * | 2019-01-07 | 2019-05-10 | 西安理工大学 | A kind of human action based on inertial sensor describes method |
| CN112057077A (en) * | 2019-06-11 | 2020-12-11 | 本田技研工业株式会社 | Information processing apparatus, information processing method, and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113406646A (en) * | 2021-06-18 | 2021-09-17 | 北京师范大学 | Method and equipment for three-dimensional positioning based on multi-direction ultrasonic ranging and IMU (inertial measurement Unit) |
| CN113749651A (en) * | 2021-10-18 | 2021-12-07 | 长春理工大学 | Pressure evaluation method and pressure evaluation system based on human body posture recognition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11016297B2 (en) | Image generation apparatus and image generation method | |
| US10353482B2 (en) | Systems and methods for tracking motion and gesture of heads and eyes | |
| JP7046189B2 (en) | How to calibrate an augmented reality device | |
| EP1886281B1 (en) | Image processing method and image processing apparatus | |
| US10269139B2 (en) | Computer program, head-mounted display device, and calibration method | |
| CN108805917A (en) | Sterically defined method, medium, device and computing device | |
| EP3640889A1 (en) | In situ creation of planar natural feature targets | |
| KR20150093831A (en) | Direct interaction system for mixed reality environments | |
| US11662589B2 (en) | Geometry modeling of eyewear devices with flexible frames | |
| Hermanis et al. | Acceleration and magnetic sensor network for shape sensing | |
| CN109186596B (en) | IMU measurement data generation method, system, computer device and readable storage medium | |
| JP6852673B2 (en) | Sensor device, sensor system and information processing device | |
| CN110494792A (en) | The vision of peripheral equipment tracks | |
| KR20000017755A (en) | Method for Acquisition of Data About Motion | |
| US11585931B2 (en) | Light direction detector systems and methods | |
| CN112486331A (en) | IMU-based three-dimensional space handwriting input method and device | |
| CN108257177A (en) | Alignment system and method based on space identification | |
| CN109166181A (en) | A kind of mixing motion capture system based on deep learning | |
| CN105630152A (en) | Device and method for processing visual data, and related computer program product | |
| CN110050221A (en) | Virtual rigid frame for sensor subsystem | |
| US20240031678A1 (en) | Pose tracking for rolling shutter camera | |
| US20190394447A1 (en) | Imaging apparatus | |
| KR100563948B1 (en) | Apparatus for attitude determination test of star-sensor and method for display of imaginary constellation | |
| JPWO2020121406A1 (en) | 3D measuring device, mobile robot, push wheel type moving device and 3D measurement processing method | |
| JP6205387B2 (en) | Method and apparatus for acquiring position information of virtual marker, and operation measurement method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210312 |
|
| WD01 | Invention patent application deemed withdrawn after publication |