CN110865704B - Gesture interaction device and method for 360-degree suspended light field three-dimensional display system - Google Patents
Gesture interaction device and method for 360-degree suspended light field three-dimensional display system Download PDFInfo
- Publication number
- CN110865704B CN110865704B CN201911000554.0A CN201911000554A CN110865704B CN 110865704 B CN110865704 B CN 110865704B CN 201911000554 A CN201911000554 A CN 201911000554A CN 110865704 B CN110865704 B CN 110865704B
- Authority
- CN
- China
- Prior art keywords
- gesture
- infrared
- hand
- finger
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Abstract
The invention discloses a gesture interaction device and a gesture interaction method for a 360-degree suspension type light field three-dimensional display system, wherein the gesture interaction device comprises wearable equipment worn by a user, an optical positioning device used for collecting hand position information and a processor used for processing and transmitting signals; an IMU sensor in the wearable device can detect gesture information such as acceleration, bending angle and palm rotation angle of each finger, and the communication is carried out between the Bluetooth module and the processor; the optical positioning device consists of a near-infrared camera array and infrared mark points. The position information of the hand is judged through the infrared camera array, the position relation between the fingers and the three-dimensional image is calculated, the gesture of the hand is detected through the wearable device, the wearable device communicates with the processor, and the gesture is judged, so that the image change of the display system is controlled, and the gesture interaction purpose is achieved. The invention has simple structure, accurate identification and strong expansibility and can be used for the human-computer interaction of various light field three-dimensional display systems.
Description
Technical Field
The invention relates to the technical field of computer vision and gesture recognition, in particular to a gesture interaction device and method for a 360-degree suspended light field three-dimensional display system.
Background
Vision is an important way for humans to obtain information, and 80% of the information obtained by humans every day is obtained visually. The depth information in the physical world is lacked by more two-dimensional display equipment used in daily life, the display image is lacked in reality sense, and the problem is well solved by the appearance of the three-dimensional display technology.
The three-dimensional display technology can provide real three-dimensional sensing effects such as shielding and perspective and provide immersive use experience for an observer. Three-dimensional display technologies are various, and among them, the naked eye three-dimensional display technology, which does not need to wear visual aids such as glasses, is the most popular, and the light field three-dimensional display technology is one of the realization methods. The light field three-dimensional display is a new three-dimensional display technology, and the technology reconstructs information of each field of view of a three-dimensional scene by simulating the light-emitting characteristic of the surface of a three-dimensional object, and can correctly represent the mutual shielding relation between different objects. As with conventional two-dimensional display devices such as televisions and cell phones, good interaction between a user and the three-dimensional display device is also required. Therefore, a method of combining the optical positioning device and the wearable device is provided to realize gesture recognition interactive operation of the three-dimensional display device, an observer can insert a hand into a display space to contact a virtual scene to perform a series of operations, and the telepresence of a user is improved.
Currently, gesture recognition is mainly divided into two research directions based on vision and wearable equipment. The gesture recognition method based on vision can provide a more natural and direct man-machine interaction mode for a user, but has space limitation, and when encountering nonresistible factors such as shielding, accurate recognition cannot be performed, and the vision method is unstable, is greatly limited by a background environment, and has limited recognition accuracy. The wearable device has the advantages of high accuracy and good stability, and the hand position can be determined by light distribution optical positioning, so that accurate identification can be performed in a target area, and misoperation outside the identification area is avoided.
Therefore, optical positioning tracking hand space positions are applied to the existing 360-degree suspended light field three-dimensional display equipment, accurate gesture recognition is carried out by using the wearable equipment, interaction experience of the display equipment can be well improved, and actual use value and prospect of the three-dimensional display equipment are increased.
Disclosure of Invention
The invention aims to provide a gesture interaction device for a 360-degree suspended light field three-dimensional display system, which can realize positioning and tracking of the hand of a user, perform gesture recognition to enable the displayed three-dimensional image to generate corresponding changes such as rotation and the like, realize man-machine interaction and improve the immersive experience of the user.
The purpose of the invention is realized by the following technical scheme: a gesture interaction device for a 360-degree suspended light field three-dimensional display system comprises a wearable device worn by a user, an optical positioning device and a processor;
the wearable device comprises IMU sensors arranged on fingers and a palm and used for detecting acceleration and bending angles of each finger and attitude information such as acceleration and rotation angles of the palm, and a Bluetooth module is arranged in the wearable device and used for communicating with a processor, sending sensor data and receiving control information.
The optical positioning device is arranged above the directional scattering screen of the display system and used for collecting hand position information, the optical positioning device comprises a camera array and a plurality of infrared mark points, the camera array is composed of a plurality of near-infrared cameras, the near-infrared cameras transmit data information to the processor through USB data lines, the infrared mark points are arranged on the wearable device, and at least one of the infrared mark points is arranged in the center of the back of the hand.
Further, each of the wearable devices comprises at least 7 IMU sensors, at least 5 of which are located at the distal interphalangeal joint of each finger, at least 1 of which is located at the base joint of the thumb, and at least 1 of which is located at the center of the back of the hand, and the bluetooth module is mounted at the back of the hand of the wearable device.
Further, the near-infrared camera array in the optical positioning device is symmetrically arranged left and right about the center of the scattering screen, and the direction of the lens faces the direction of the directional scattering screen so as to determine the space position of the hand.
Furthermore, the infrared mark points are active emission type mark points or passive reflection type mark points and are used for determining three-dimensional coordinates of the key points of the hands by the camera.
Furthermore, a visible light filter is arranged in front of the near-infrared camera CCD, so that the CCD can only detect near-infrared light related to positioning and filter background visible light unrelated to positioning.
A gesture interaction method for a 360-degree suspended light field three-dimensional display system comprises the following steps:
1) establishing a world coordinate system and a palm coordinate system, and calibrating a sensor of the wearable device; the palm coordinate system uses the infrared mark point of back of the hand center department as the original point, and the finger tip direction is the positive direction of x axle in pointing to by the original point, and the direction of perpendicular to the x axle in the palm plane and pointing to thumb one side is the positive direction of y axle, obtains the positive direction of z axle according to the right hand rule.
2) The user wears wearable equipment to insert the hand into the detectable area of the optical positioning device, the near-infrared camera array detects the image information of the infrared mark points on the camera imaging surface, the central point of the infrared mark point image is extracted, and the position of the infrared mark points on the corresponding camera imaging surface is determined.
3) And for the image information of the camera imaging surface acquired at the same moment, the image positions of the infrared mark points in different cameras are in one-to-one correspondence, and stereo matching is established.
4) Determining the position coordinates P (x, y, z) of the infrared mark point at the center of the back of the hand under a world coordinate system by a triangulation method according to the world coordinate information of the infrared mark point after the stereo matching on different camera imaging surfaces;
5) calculating the position coordinate P' of each base joint of the finger under a palm coordinate system according to the distance relationship between the infrared mark point at the center of the back of the hand and each base joint of the fingeri(xi，yi，zi) Wherein i is 1, 2, 3, 4, 5;
6) reading IMU sensor data at the finger through the Bluetooth module to obtain the bending angle theta of each joint of the fingerimWherein, i is 1, 2, 3, 4, 5, m is 1, 2, 3, which respectively corresponds to the base joint, middle joint and end joint of each finger according to the joint length l of each fingerimAnd a bending angle thetaimThe projection length delta x of each knuckle on each coordinate axis of the palm coordinate system can be obtainedim，Δyim，ΔzimThe coordinates P 'of each finger tip in the palm coordinate system are obtained based on each finger base joint as a calculation reference'i(xi，yi，zi)；
7) Reading IMU sensor data on the back of a hand on the wearable device to obtain a rotation matrix R of a palm coordinate system relative to a world coordinate system; obtaining a translation vector of the palm coordinate system relative to the world coordinate system according to the coordinates of the infrared mark point at the center of the back of the hand in the world coordinate system determined in the step (4)
Thus, the coordinate P of each fingertip in the world coordinate system is calculatedi(xi，yi，zi)；
P′i(xi，yi，zi) And Pi(xi，yi，zi) Satisfy the relationship
Wherein i is 1, 2, 3, 4, 5;
8) when the obtained fingertip coordinate Pi(xi，yi，zi) When the three-dimensional scene world coordinates displayed by the three-dimensional display system coincide with the world coordinates, the processor extracts the characteristics of the received IMU sensor data information, determines the gesture of a user through a recognition algorithm based on a neural network, and then controls the three-dimensional displayed image to carry out corresponding transformation.
Further, the calibration of the camera is performed by using a luminous dot matrix target board consisting of near-infrared LEDs.
Further, the position of the infrared mark point on the corresponding camera imaging surface is determined by using a fast central point extraction algorithm based on the connected domain mark.
Further, in step 8), the process of performing gesture recognition is as follows:
8-1) integrating the acceleration values of each finger collected by the IMU sensor to obtain the velocity v of each finger tipiWhere i is 1, 2, 3, 4, 5, a rate threshold v is setrAs a decision threshold for a valid gesture starting point, when:
v＝max{vi|i＝1，2，3，4，5}＞vr
and the number of data frames between the first frame and the last frame which satisfy the above formula exceeds 50 frames, and the gesture is determined to be effective;
8-2) for effective gestures, carrying out normalization processing on bending angle and rotation angle data acquired from an IMU sensor in the wearable device, and taking the data as input data of a neural network;
8-3) for all the N IMU sensors for collecting data, each bending angle and rotation angle data can be used as an input node of a neural network, and the input node data form an N-dimensional vector X [ N ], wherein X [ i ] is data after each sensor is normalized, and i is 1, 2, 3.
8-4) determining the number of hidden layer cells according to one of the following three optimal hidden cell number reference formulas:
wherein k is the number of samples, n1The number of hidden units, n is the number of input neurons;
wherein m is the number of output neurons, and a is a constant between [1, 10 ];
n1＝log2n
8-5) inputting the XN into a neural network for training to obtain the trained neural network, wherein the input of the neural network is IMU sensor data after normalization processing, and the output is the probability of a corresponding gesture label; and recording the number of recognized gestures as r, labeling the gestures to be recognized from 1 to r, wherein the data structure of the output layer node is a vector Y [ r ] with r dimension, and an element Y [ i ] in the vector is the product of an input node value of the node and a corresponding weight. For X [ N ] input into the neural network, the i value of the node with the value closest to 1 in the output layer is the mark corresponding to the recognized gesture. And the processor controls corresponding image transformation according to the corresponding gesture to realize a gesture interaction process.
Compared with the prior art, the invention has the beneficial effects that:
1) the interference of visible light on the positioning system can be effectively eliminated by using the near-red optical positioning system, and the accuracy of hand space positioning is improved;
2) compared with a vision-based recognition method, the wearable device can acquire the posture information of the palm and the fingers more accurately, and is higher in accuracy and higher in speed;
3) the wearable device based on the IMU sensor has longer service life and lower system maintenance cost than the wearable device based on the flexible sensor;
4) the scheme that optical positioning and wearable equipment combined together can carry out gesture recognition under the condition that the finger gesture is partly sheltered from, has wider application scope than the gesture recognition based on vision.
Drawings
FIG. 1 is a schematic diagram of a gesture interaction apparatus for a 360-degree floating light field three-dimensional display system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an internal structure of a gesture interaction device for a 360-degree floating light field three-dimensional display system according to an embodiment of the present invention;
FIG. 3 is a schematic view of a working flow of a gesture interaction device for a 360-degree floating light field three-dimensional display system according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the angle relationship between the joints of the human hand according to the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described below with reference to the following embodiments and the accompanying drawings.
The invention provides a gesture interaction device for a 360-degree suspended light field three-dimensional display system, which comprises wearable equipment worn by a user, an optical positioning device and a processor;
the wearable device comprises at least 7 IMU sensors placed on fingers and a palm, wherein at least 5 IMU sensors are positioned at the joints between the distal fingers of each finger, at least 1 IMU sensor is positioned at the joint of the base of the thumb, and at least 1 IMU sensor is positioned at the center of the back of the hand and is used for detecting the acceleration and the bending angle of each finger and the acceleration and the rotation angle of the palm, and the wearable device is internally provided with a Bluetooth module which is used for communicating with a processor, sending sensor data and receiving control information. The Bluetooth module is installed at the back of the hand of the wearable device.
The optical positioning device is arranged above the directional scattering screen of the display system and used for collecting hand position information, the optical positioning device comprises a camera array and a plurality of infrared mark points, the camera array is composed of a plurality of near-infrared cameras, the near-infrared camera array in the optical positioning device is symmetrically arranged on the left and right of the center of the scattering screen, the lens direction faces the directional scattering screen direction, the near-infrared cameras transmit data information to the processor through USB data lines, and a visible light filter is arranged in front of a CCD of the near-infrared camera, so that the CCD can only detect near-infrared light related to positioning, and background visible light unrelated to positioning is filtered. At least one infrared mark point is arranged in the center of the back of the hand of the wearable device, and the infrared mark point is an active emission type mark point or a passive reflection type mark point and is used for determining the three-dimensional coordinates of key points of the hand by a camera.
A gesture interaction method for a 360-degree suspended light field three-dimensional display system comprises the following steps:
1) establishing a world coordinate system and a palm coordinate system, wherein the palm coordinate system takes an infrared mark point at the center of the back of a hand as an origin, the direction pointing to the middle finger tip is the positive direction of an x axis, the direction perpendicular to the x axis and pointing to one side of the thumb in a palm plane is the positive direction of a y axis, the positive direction of a z axis is obtained according to a right hand criterion, and the calibration of the wearable device sensor is carried out;
2) a user wears wearable equipment to insert a hand into a detectable area of the optical positioning device, the calibration of the near-infrared camera is carried out by using a luminous dot matrix target plate consisting of near-infrared LEDs, the near-infrared camera array detects the image information of infrared mark points on a camera imaging surface, the central point of the infrared mark point image is extracted by using a fast central point extraction algorithm based on connected domain marks, and the position of the infrared mark points on the corresponding camera imaging surface is determined;
3) for the image information of the camera imaging surface acquired at the same moment, the image positions of the infrared mark points in different cameras are in one-to-one correspondence, and stereo matching is established;
4) determining the position coordinates P (x, y, z) of the infrared mark points at the back of the hand under a world coordinate system by a triangulation method according to the world coordinate information of the infrared mark points after the stereo matching on different camera imaging surfaces;
5) calculating the position coordinate P' of each finger base joint in the palm coordinate system according to the distance relationship between the infrared mark point at the back of the hand and each finger base jointi(xi，yi，zi) Wherein i is 1, 2, 3, 4, 5;
6) reading data of the IMU sensors at the fingers through a Bluetooth module, and reducing the angle of each joint of each finger through reverse dynamics when only one IMU sensor is installed on each finger; when there are at least two IMU sensors on each finger, the angle of each joint of each finger can also be obtained from the angle at the distal interphalangeal joint being two thirds of the angle at the proximal interphalangeal joint. Recording the bending angle theta of each joint of the fingerimWherein, i is 1, 2, 3, 4, 5, m is 1, 2, 3, which respectively corresponds to the base joint, middle joint and end joint of each finger according to the joint length l of each fingerimAnd a bending angle thetaimThe projection length delta x of each knuckle on each coordinate axis of the palm coordinate system can be obtainedim，Δyim，ΔzimThe coordinates P 'of each finger tip in the palm coordinate system are obtained based on each finger base joint as a calculation reference'i(xi，yi，zi)；
7) Reading IMU sensor data on the back of a hand on the wearable device to obtain a rotation matrix R of a palm coordinate system relative to a world coordinate system; obtaining a translation vector of the palm coordinate system relative to the world coordinate system according to the coordinates of the infrared mark point at the back of the hand in the world coordinate system determined in the step (4)
Thus, the coordinate P of each fingertip in the world coordinate system is calculatedi(xi，yi，zi)，
P′i(xi，yi，zi) And Pi(xi，yi，zi) Satisfy the relationship
Wherein i is 1, 2, 3, 4, 5;
8) when the obtained fingertip coordinate Pi(xi，yi，zi) When the three-dimensional scene world coordinates displayed by the system coincide with the world coordinates, the processor extracts the characteristics of the received IMU sensor data information, determines the gestures of a user through a recognition algorithm based on a neural network, and further controls the three-dimensional displayed image to carry out corresponding transformation;
the specific process of performing gesture recognition is as follows:
8-1) integrating IMU sensor acceleration values to obtain the velocity v of the finger tipiWhere i is 1, 2, 3, 4, 5, a rate threshold v is setrAs a decision threshold for a valid gesture starting point, when:
v＝max{vi|i＝1，2，3，4，5}＞vr
when the total data frame number between the first time and the last time meeting the formula exceeds 50 frames, judging the gesture to be effective, and analyzing gesture data;
8-2) for effective gestures, performing normalization processing on sensor data collected from the wearable device, and taking the sensor data as input data of a neural network;
8-3) for all the N IMU sensors for collecting information, each bending angle and rotation angle data can be used as an input node of a neural network, the input node data form an N-dimensional vector X [ N ], and X [ i ] is data after normalization of each sensor, wherein i is 1, 2, 3.. N;
8-4) determining the number of hidden layer cells according to one of the following three optimal hidden cell number reference formulas:
wherein k is the number of samples, n1The number of hidden units, n is the number of input neurons;
wherein m is the number of output neurons, and a is a constant between [1, 10 ];
n1＝log2n
8-5) inputting the XN into a neural network for training to obtain the trained neural network, wherein the input of the neural network is IMU sensor data, and the output is a corresponding gesture; and recording the number of recognized gestures as r, labeling the gestures to be recognized from 1 to r, wherein the data structure of the output layer node is a vector Y [ r ] with r dimension, and an element Y [ i ] in the vector is the product of an input node value of the node and a corresponding weight. For X [ N ] input into the neural network, the i value of the node with the value closest to 1 in the output layer is the mark corresponding to the recognized gesture. And the processor controls corresponding image transformation according to the corresponding gesture to realize a gesture interaction process.
Examples
As shown in fig. 1, the 360 ° floating three-dimensional display system synchronously projects images onto a directional scattering screen 005 rotating at a high speed through a high-speed projector 003, so as to realize 360 ° retractable floating three-dimensional light field display. Wherein the frame rate of the high-speed projector is 18000fps, and the rotating speed of the directional diffuser screen 005 is 30 r-sThe directional diffusion screen 005 receives 600 projected images by one rotation. The servo motor 006 under the directional diffusion screen 005 drives the screen to rotate at high speed in the horizontal plane, the high-speed projector 003 is positioned right above the rotation center of the screen, and the optical axis of the projector is coaxial with the rotation center of the screen to ensure that the projected image has no distortion. In this way, the system realizes the light field reconstruction of the display points and displays the true three-dimensional scene.
The gesture interaction device for the 360-degree suspended light field three-dimensional display system in the embodiment comprises:
1, two industrial cameras symmetrically arranged above a directional diffuser screen form a binocular camera system, the effective pixel number is 130 ten thousand, the resolution is 1280 × 720, the frame rate is 34fps, a camera 001 lens faces to the directional diffuser screen, and a visible light filter is arranged in front of a light-sensitive surface of the binocular camera 001 and used for filtering visible light;
2 near-infrared LED lamps 002 for auxiliary lighting disposed above the directional diffusion screen 005;
3 wearable device 004 for gathering hand gesture information, comprising: 10 IMU sensors placed at finger joints (two sensors are arranged on each finger, one sensor is positioned at the interphalangeal joint of the far end of the finger, and the other sensor is positioned at the interphalangeal joint of the near end of the finger), 1 IMU sensor positioned at the center of the back of the hand, and 1 IMU sensor positioned at the joint of the base part of the thumb; the back of the hand is provided with a module which comprises a power supply and a Bluetooth transmitting and receiving device and is used for communicating with the processor;
the back of the hand of wearable equipment 004 of 4 department is equipped with 6 infrared mark points, infrared mark point selects as reflection of light mark point, and 6 mark points are located 5 base joints departments of finger and wearable equipment 004 back of the hand center respectively for the spatial position of location finger fingertip.
In this embodiment, the parameters and the number of the devices may be changed according to the actual needs of the system.
In this embodiment, constitute optical locating device by binocular camera 001, infrared LED002 and infrared mark point, when the system began work, optical locating device surveyed the spatial position of palm through binocular camera 001, then calculated finger tip position, when finger tip position and demonstration three-dimensional figure coordinate coincide, began to carry out gesture recognition, and the treater judges gesture information according to the sensor information that wearable equipment 004 transmitted to the control shows the graphic transformation.
Referring to fig. 2 and fig. 3, the specific implementation process of the gesture interaction system is as follows:
(1) adjusting the ambient brightness around the display system to a darker state;
(2) opening the binocular camera 001, and adjusting the posture and parameters of the binocular camera 001 to enable clear imaging;
(3) performing internal and external reference calibration of a camera by using a light-emitting dot matrix target board consisting of near-infrared LEDs, wherein the near-infrared LEDs on the target board are installed at the angular points of a standard checkerboard and calibrated by adopting a Zhang Zhengyou calibration method;
(4) establishing a world coordinate system and a palm coordinate system, wherein the palm coordinate system takes a palm geometric center as an origin, the direction pointing to the middle finger tip is the positive direction of an x axis, the direction perpendicular to the x axis and pointing to one side of the thumb in a palm plane is the positive direction of a y axis, the positive direction of a z axis is obtained according to a right hand criterion, and the parameter calibration of the wearable equipment is carried out;
(5) turning on the near-infrared LED lamp 002, enabling a user to wear the wearable device 004 to insert a hand into a detectable area of the optical positioning device, and enabling the near-infrared camera to detect infrared rays reflected by the reflective mark points to obtain image information of the reflective mark points on an imaging surface of the camera;
(6) extracting characteristic points of the acquired image, extracting the central point of the image of the light reflecting mark point, and determining the position information of the characteristic points on the image surface of the corresponding camera;
(7) for image information acquired at the same time, corresponding the image positions of 6 light-reflecting mark points in different cameras one by one, and establishing stereo matching;
(8) determining the position coordinates P (x, y, z) of the mark points after the stereo matching on the imaging surfaces of different cameras by a triangulation method;
(9) according to the distance relationship between the reflective mark points at the back of the hand and the reflective mark points at the base joints of the fingers, calculating to obtain the position coordinate P' of the base joints of the fingers under the palm coordinate systemi(xi，yi，zi) Wherein i is 1, 2, 3, 4, 5;
(10) as shown in FIG. 4, FPIPIs the proximal interphalangeal joint, FDIPFor the far-end fingertip joint, reading IMU sensor data at the finger position on the wearable device 004 through the Bluetooth module to obtain theta and thetapIPThen according to
The angle values at the three joints of the finger can be obtained. According to the length l of each joint of the finger1，l2，l3And bending angle theta, thetaDIP，θpIPThe projection length delta x of each knuckle on each coordinate axis in the palm coordinate system can be obtainedim，Δyim，ΔzimBased on the joints of the base parts of the fingersCalculating the reference to obtain the coordinates P 'of each finger tip in the palm coordinate system'i(xi，yi，zi)；
(11) Reading IMU sensor data on the back of the hand on the wearable device 004 through a Bluetooth module to obtain rotation angles of the palm relative to the world coordinate system in all coordinate axis directions, and further obtaining a rotation matrix R of the palm coordinate system relative to the world coordinate system; obtaining a translation vector of the palm coordinate system relative to the world coordinate system according to the coordinates of the light-reflecting mark points at the back of the hand under the world coordinate system
Thus, the coordinate P of each fingertip in the world coordinate system is calculatedi(xi，yi，zi)；
(12)P′i(xi，yi，zi) And Pi(xi，yi，zi) Satisfy the relationship
Wherein i is 1, 2, 3, 4, 5;
(13) when the obtained finger tip position coordinate Pi(xi，yi，zi) When the acceleration value of the IMU sensor is coincident with the world coordinate of the three-dimensional scene displayed by the system, the acceleration value of the IMU sensor is integrated to obtain the velocity v of the finger tipiWhere i is 1, 2, 3, 4, 5, a rate threshold v is setrAs a decision threshold for a valid gesture starting point, when: v ═ max { v ═i|i＝1，2，3，4，5}＞vrWhen the total data frame number between the first frame data and the last frame data meeting the formula exceeds 50 frames, judging that the gesture is effective, and analyzing gesture data;
(14) sensor data collected from the wearable device 004 is subjected to normalization processing and used as input data of a neural network;
(15) for all 12 IMU sensors collecting information, each bending angle and rotation angle data can be used as an input node of a neural network, and the input node data form a 12-dimensional vector X [ n ], wherein n is 12, and X [ i ] is data after each sensor is normalized, wherein i is 1, 2, 3.
(16) Determining the number of hidden layer units according to the following optimal hidden unit reference formula:
wherein m is the number of output neurons, n1The number of hidden units, n is the number of input neurons, n is 12, a is [1, 10]]Constant in between. Calculating to obtain n1Is 10;
(17) selecting a BP neural network, using a sigmoid function as an activation function, training the neural network after the neural network is set by using the parameters, finishing the training when the output error of the neural network is less than 0.001, and storing each parameter value of the current neural network; the input of the neural network is IMU sensor data after normalization processing, and the output is the probability of the corresponding gesture label;
(18) in this embodiment, the number of gestures to be recognized is 5, 5 different gestures are corresponding to 1 to 5, and the labels are corresponding to the gestures in order, so that the output layer node data structure is a 5-dimensional vector Y [5], and an element Y [ i ] in the vector is a product of an input node value of the node and a corresponding weight. For X12 input to the neural network, the i value of the node with the value closest to 1 in the output layer is the mark corresponding to the recognized gesture. And the processor controls corresponding image change according to the corresponding gesture to realize a gesture interaction process.
After all the steps and processing are completed, the observer can complete one gesture interaction with the three-dimensional display system. And (4) repeating the steps (5) to (18) to complete the continuous interaction with the display system.
The gesture interaction device and method based on the optical positioning and wearable device are not limited to the embodiment and the display system, and are also suitable for the gesture interaction requirements of a human-computer interaction system and a three-dimensional display system in other wide-angle scene ranges.
While the invention has been further described herein by way of illustration and example, it is to be understood that the invention is not limited to the embodiments and examples described above, and that the foregoing description is intended to be illustrative and not limiting, and that various changes and modifications may be made by one skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims.
Claims (7)
1. A gesture interaction method for a 360-degree suspended light field three-dimensional display system is characterized by being realized based on a gesture interaction device and comprising wearable equipment worn by a user, an optical positioning device and a processor;
the wearable device comprises IMU sensors arranged on fingers and a palm and used for detecting the acceleration and bending angle of each finger and the acceleration and rotation angle of the palm and other attitude information, and a Bluetooth module is arranged in the wearable device and used for communicating with a processor, sending sensor data and receiving control information;
the optical positioning device is arranged above the directional scattering screen of the display system and used for collecting hand position information, the optical positioning device comprises a camera array consisting of a plurality of near-infrared cameras and a plurality of infrared mark points, the near-infrared cameras transmit data information to the processor through USB data lines, and the infrared mark points are arranged on the wearable equipment, and at least one infrared mark point is arranged in the center of the back of a hand;
the gesture interaction comprises the following steps:
1) establishing a world coordinate system and a palm coordinate system, and calibrating a sensor of the wearable device; the palm coordinate system takes an infrared mark point at the center of the back of the hand as an original point, the direction from the original point to the middle finger tip as the positive direction of an x axis, the direction perpendicular to the x axis and to one side of the thumb in the palm plane as the positive direction of a y axis, and the positive direction of a z axis is obtained according to the right hand rule;
2) a user wears wearable equipment to insert a hand into a detectable area of the optical positioning device, the near-infrared camera array detects image information of the infrared mark points on a camera imaging surface, the central point of the infrared mark point image is extracted, and the positions of the infrared mark points on the corresponding camera imaging surface are determined;
3) for the image information of the camera imaging surface acquired at the same moment, the image positions of the infrared mark points in different cameras are in one-to-one correspondence, and stereo matching is established;
4) determining the position coordinates P (x, y, z) of the infrared mark point at the center of the back of the hand under a world coordinate system by a triangulation method according to the world coordinate information of the infrared mark point after the stereo matching on different camera imaging surfaces;
5) calculating the position coordinate P' of each base joint of the finger under a palm coordinate system according to the distance relationship between the infrared mark point at the center of the back of the hand and each base joint of the fingeri(xi，yi，zi) Wherein i is 1, 2, 3, 4, 5;
6) reading IMU sensor data at the finger through the Bluetooth module to obtain the bending angle theta of each joint of the fingerimWherein, i is 1, 2, 3, 4, 5, m is 1, 2, 3, which respectively corresponds to the base joint, middle joint and end joint of each finger according to the joint length l of each fingerimmAnd a bending angle thetaimThe projection length delta x of each knuckle on each coordinate axis of the palm coordinate system can be obtainedim，Δyim，ΔzimThe coordinates P 'of each finger tip in the palm coordinate system are obtained based on each finger base joint as a calculation reference'i(xi，yi，zi)；
7) Reading IMU sensor data on the back of a hand on the wearable device to obtain a rotation matrix R of a palm coordinate system relative to a world coordinate system; obtaining a translation vector of the palm coordinate system relative to the world coordinate system according to the coordinates of the infrared mark point at the center of the back of the hand in the world coordinate system determined in the step (4)
Thus, the coordinate P of each fingertip in the world coordinate system is calculatedi(xi，yi，zi)；
P′i(xi，yi，zi) And Pi(xi，yi，zi) Satisfy the relationship
Wherein i is 1, 2, 3, 4, 5;
8) when the obtained fingertip coordinate Pi(xi，yi，zi) When the three-dimensional scene world coordinates displayed by the three-dimensional display system coincide with the world coordinates, the processor extracts the characteristics of the received IMU sensor data information, determines the gesture of a user through a recognition algorithm based on a neural network, and further controls the three-dimensional displayed image to carry out corresponding transformation;
the process of performing gesture recognition is as follows:
8-1) integrating the acceleration values of each finger collected by the IMU sensor to obtain the velocity v of each finger tipiWhere i is 1, 2, 3, 4, 5, a rate threshold v is setrAs a decision threshold for a valid gesture starting point, when:
v＝max{vi|i＝1，2，3，4，5}＞vr
and the number of data frames between the first frame and the last frame which satisfy the above formula exceeds 50 frames, and the gesture is determined to be effective;
8-2) for effective gestures, carrying out normalization processing on bending angle and rotation angle data acquired from an IMU sensor in the wearable device, and taking the data as input data of a neural network;
8-3) for all the N IMU sensors for collecting data, each bending angle and rotation angle data can be used as an input node of a neural network, and the input node data form an N-dimensional vector X [ N ], wherein X [ i ] is data after each sensor is normalized, and i is 1, 2, 3.
8-4) determining the number of hidden layer cells according to one of the following three optimal hidden cell number reference formulas:
wherein k is the number of samples, n1The number of hidden units, n is the number of input neurons;
wherein m is the number of output neurons, and a is a constant between [1, 10 ];
n1＝log2n
8-5) inputting the XN into a neural network for training to obtain the trained neural network, wherein the input of the neural network is IMU sensor data after normalization processing, and the output is the probability of a corresponding gesture label; recording the number of recognized gestures as r, labeling the gestures to be recognized from 1 to r, wherein the data structure of the output layer node is a vector Y [ r ] of r dimension, and an element Y [ i ] in the vector is the product of an input node value of the node and a corresponding weight; for X [ N ] input into the neural network, the i value of the node with the value closest to 1 in the output layer is the mark corresponding to the recognized gesture; and the processor controls corresponding image transformation according to the corresponding gesture to realize a gesture interaction process.
2. The gesture interaction method according to claim 1, characterized in that: each of the wearable devices comprises at least 7 IMU sensors, at least 5 of which are located at the distal interphalangeal joint of each finger, at least 1 of which is located at the base joint of the thumb, and at least 1 of which is located at the center of the back of the hand, and the Bluetooth module is installed at the back of the hand of the wearable device.
3. The gesture interaction method according to claim 1, characterized in that: the near-infrared camera array in the optical positioning device is symmetrically arranged left and right about the center of the scattering screen, and the direction of the lens faces the direction of the directional scattering screen so as to determine the spatial position of the hand.
4. The gesture interaction method according to claim 1, characterized in that: the infrared mark points are active emission type mark points or passive reflection type mark points and are used for determining three-dimensional coordinates of the hand key points by the camera.
5. The gesture interaction method according to claim 1, characterized in that: the visible light filter is arranged in front of the near-infrared camera CCD, so that the CCD can only detect near-infrared light related to positioning and filter background visible light unrelated to positioning.
6. The gesture interaction method according to claim 1, characterized in that: the calibration of the camera is performed by using a luminous dot matrix target board consisting of near-infrared LEDs.
7. The gesture interaction method according to claim 1, characterized in that: in the step 2), the position of the infrared mark point on the corresponding camera imaging surface is determined by using a fast central point extraction algorithm based on the connected domain mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911000554.0A CN110865704B (en) | 2019-10-21 | 2019-10-21 | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911000554.0A CN110865704B (en) | 2019-10-21 | 2019-10-21 | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110865704A CN110865704A (en) | 2020-03-06 |
| CN110865704B true CN110865704B (en) | 2021-04-27 |
Family
ID=69652283
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911000554.0A Active CN110865704B (en) | 2019-10-21 | 2019-10-21 | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110865704B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111722713A (en) * | 2020-06-12 | 2020-09-29 | 天津大学 | Multi-mode fused gesture keyboard input method, device, system and storage medium |
| CN112306305B (en) * | 2020-10-28 | 2021-08-31 | 黄奎云 | Three-dimensional touch device |
| CN112927290A (en) * | 2021-02-18 | 2021-06-08 | 青岛小鸟看看科技有限公司 | Bare hand data labeling method and system based on sensor |
| CN113093911B (en) * | 2021-04-09 | 2023-06-30 | 浙江大学 | Intelligent ring oriented to three-dimensional interactive input and interaction method thereof |
| CN115812173A (en) * | 2021-05-20 | 2023-03-17 | 京东方科技集团股份有限公司 | Method for dynamically displaying three-dimensional image objects in a stereoscopic display device, dynamic stereoscopic display device and computer program product |
| CN113223344B (en) * | 2021-05-25 | 2022-08-23 | 湖南汽车工程职业学院 | Big data-based professional teaching display system for art design |
| CN114245542B (en) * | 2021-12-17 | 2024-03-22 | 深圳市恒佳盛电子有限公司 | Radar induction lamp and control method thereof |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108256504A (en) * | 2018-02-11 | 2018-07-06 | 苏州笛卡测试技术有限公司 | A kind of Three-Dimensional Dynamic gesture identification method based on deep learning |
| CN109635621A (en) * | 2017-10-07 | 2019-04-16 | 塔塔顾问服务有限公司 | For the system and method based on deep learning identification gesture in first person |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI471757B (en) * | 2013-01-31 | 2015-02-01 | Pixart Imaging Inc | Hand posture detection device for detecting hovering and click |
| CN104298065B (en) * | 2014-05-07 | 2017-01-25 | 浙江大学 | 360-degree three-dimensional display device and method based on splicing of multiple high-speed projectors |
| CN106019592A (en) * | 2016-07-15 | 2016-10-12 | 中国人民解放军63908部队 | Augmented reality optical transmission-type helmet mounted display pre-circuit and control method thereof |
| CN207268857U (en) * | 2017-07-28 | 2018-04-24 | 山东师范大学 | A kind of dynamic 3 D gesture identifying device |
| US10809796B2 (en) * | 2017-09-29 | 2020-10-20 | Apple Inc. | Monitoring a user of a head-wearable electronic device |
| JP6745852B2 (en) * | 2018-01-24 | 2020-08-26 | アップル インコーポレイテッドＡｐｐｌｅ Ｉｎｃ． | Devices, methods, and graphical user interfaces for system-wide behavior of 3D models |
| CN108932500B (en) * | 2018-07-09 | 2019-08-06 | 广州智能装备研究院有限公司 | A kind of dynamic gesture identification method and system based on deep neural network |
| CN109753153B (en) * | 2018-12-26 | 2020-05-22 | 浙江大学 | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system |
-
2019
- 2019-10-21 CN CN201911000554.0A patent/CN110865704B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109635621A (en) * | 2017-10-07 | 2019-04-16 | 塔塔顾问服务有限公司 | For the system and method based on deep learning identification gesture in first person |
| CN108256504A (en) * | 2018-02-11 | 2018-07-06 | 苏州笛卡测试技术有限公司 | A kind of Three-Dimensional Dynamic gesture identification method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110865704A (en) | 2020-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110865704B (en) | Gesture interaction device and method for 360-degree suspended light field three-dimensional display system | |
| US10249090B2 (en) | Robust optical disambiguation and tracking of two or more hand-held controllers with passive optical and inertial tracking | |
| CN107209950B (en) | Automatic generation of virtual material from real world material | |
| CN107004275B (en) | Method and system for determining spatial coordinates of a 3D reconstruction of at least a part of a physical object | |
| EP3035164B1 (en) | Wearable sensor for tracking articulated body-parts | |
| CN117120962A (en) | Controlling two-handed interactions between mapped hand regions of virtual and graphical elements | |
| US9720511B2 (en) | Hand and object tracking in three-dimensional space | |
| CN116724285A (en) | Micro-gestures for controlling virtual and graphical elements | |
| US20170357332A1 (en) | Six dof mixed reality input by fusing inertial handheld controller with hand tracking | |
| US9891435B2 (en) | Apparatus, systems and methods for providing motion tracking using a personal viewing device | |
| US9986228B2 (en) | Trackable glasses system that provides multiple views of a shared display | |
| TWI540461B (en) | Gesture input method and system | |
| US20140168261A1 (en) | Direct interaction system mixed reality environments | |
| EP1074934A2 (en) | Computer input device having six degrees of freedom for controlling movement of a three-dimensional object | |
| CN107646098A (en) | System for tracking portable equipment in virtual reality | |
| CN108885487B (en) | Gesture control method of wearable system and wearable system | |
| CN109804220A (en) | System and method for tracking the fortune dynamic posture on head and eyes | |
| CN109145802B (en) | Kinect-based multi-person gesture man-machine interaction method and device | |
| US10652525B2 (en) | Quad view display system | |
| CN109471533B (en) | Student end system in VR/AR classroom and use method thereof | |
| CN112657176A (en) | Binocular projection man-machine interaction method combined with portrait behavior information | |
| US20190369807A1 (en) | Information processing device, information processing method, and program | |
| US20220256137A1 (en) | Position calculation system | |
| Diaz et al. | Multimodal sensing interface for haptic interaction | |
| Jain et al. | Human computer interaction–Hand gesture recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |