CN111860275B - Gesture recognition data acquisition system and method - Google Patents

Gesture recognition data acquisition system and method Download PDF

Info

Publication number
CN111860275B
CN111860275B CN202010674342.7A CN202010674342A CN111860275B CN 111860275 B CN111860275 B CN 111860275B CN 202010674342 A CN202010674342 A CN 202010674342A CN 111860275 B CN111860275 B CN 111860275B
Authority
CN
China
Prior art keywords
infrared
tracking camera
gesture recognition
gesture
glove
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010674342.7A
Other languages
Chinese (zh)
Other versions
CN111860275A (en
Inventor
吴涛
周锋宜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingdao Xiaoniao Kankan Technology Co Ltd
Original Assignee
Qingdao Xiaoniao Kankan Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingdao Xiaoniao Kankan Technology Co Ltd filed Critical Qingdao Xiaoniao Kankan Technology Co Ltd
Publication of CN111860275A publication Critical patent/CN111860275A/en
Application granted granted Critical
Publication of CN111860275B publication Critical patent/CN111860275B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Vascular Medicine (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The invention provides a gesture recognition data acquisition system, which comprises a closed space, an infrared tracking camera, a glove with an infrared marking point M1 on the surface, and a VR virtual reality headset with an infrared marking point M2 on the surface, wherein the VR virtual reality headset is provided with the gesture recognition tracking camera; the infrared tracking camera is connected with the VR virtual reality head-mounted connection server client; the server client comprises a space positioning module, a fitting module and a shifting module, wherein the positioning module is used for determining the position relation between the centroid of the infrared marking point M1 and the centroid of the infrared marking point M2The fitting module is used for fitting coordinate dataRelationship with positionCurve fitting is carried out to obtain a rotation matrix and a translation vector; the shift module is used for shifting the vector according to the rotation matrix and the translation vectorThe infrared marking point M1 is translated and rotated to a coordinate system taking the gesture recognition tracking camera as an origin coordinate, manual marking is not needed, the accuracy of data marking is improved, and the data marking efficiency is improved.

Description

Gesture recognition data acquisition system and method
Technical Field
The invention relates to the field of computer vision, in particular to a gesture recognition data acquisition system and method.
Background
In order to enhance the sense of immersion of virtual-real combination of VR/AR/MR, the VR/AR/MR has better experience, a human-computer interaction module is indispensable, and particularly, the high-precision real-time restoration of the 3D gesture of the hand in the VR/AR/MR scene greatly influences the sense of immersion of the user experience in the VR/AR/MR scene.
The gesture recognition is very critical in the VR/AR/MR field, particularly, light interaction in VR/AR/MR scene experience plays an important role, so that the precision of bare hand tracking, the requirement on the compatible stability of time delay and environment are high, currently, in the gesture recognition in the VR/AR/MR field, a VR/AR/MR equipment manufacturer gradually tracks and recognizes hand information of a user by considering an environment capture camera arranged at a multiplexing head-mounted integrated machine end, at present, a main scheme of hand tracking mainly adopts an AI-based algorithm architecture, a large amount of image training data needs to be acquired, each image data is marked, then training learning of a convolutional neural network is carried out, and finally, a gesture recognition convolutional neural network model with high precision and high stability can be obtained through multiple training and a large data set.
When a large amount of image data is acquired and each image data is marked, the accuracy of the data marking position of each skeleton point corresponding to a hand on each image data is very critical, and the traditional method at present solves the data marking problem by adopting a manual marking and semi-supervised learning mode. The method comprises the steps of firstly, manually marking a small part of data, then, carrying out network model training based on the data, carrying out identification marking of other data through a trained model, then, carrying out manual supervision and inspection, carrying out manual correction on the wrong position information of the marked point identified by the model, then, continuing to carry out network model training, and repeating the above processes in sequence, wherein a high-precision network model is finally expected to be obtained. By the method, the data labeling accuracy is relatively high in dependence on people, and particularly when some labeling point positions corresponding to some gesture actions are shielded under the view angle of the camera, the position information of the shielded labeling points on the image needs to be estimated manually, so that the data labeling accuracy is difficult to ensure, and the training accuracy of the network model is low.
Therefore, there is a need for a gesture recognition data acquisition system and method that does not require manual labeling, improves the accuracy of data labeling, and improves the efficiency of data labeling.
Disclosure of Invention
In view of the above problems, the present invention aims to provide a gesture recognition data acquisition system, so as to solve the problems that in the existing method, small portions of data are manually marked, network model training is performed based on the small portions of data, recognition and marking of other data are performed through a trained model, then manual supervision and inspection are performed, and error position information of marked points recognized by the model is manually corrected, further network model training is performed, the above processes are sequentially repeated, and finally a high-precision network model is obtained.
The invention provides a gesture recognition data acquisition system which is characterized by comprising a closed space, an infrared tracking camera, a glove with an infrared marking point M1 arranged on the surface and a VR virtual reality head with an infrared marking point M2 arranged on the surface, wherein,
the infrared marking point M1 is arranged at a position corresponding to a hand skeleton point;
the infrared tracking camera is connected with the VR virtual reality headset through the server client;
the infrared tracking camera is arranged on the wall surface of the closed space and is used for scanning the infrared marking points M1 and M2 to obtain the position coordinates of the glove in the closed spacePosition coordinates wearing with the VR virtual reality head in the closed space +.>And coordinates of the position +.>Is +.>Transmitting to the server client;
at least two gesture recognition tracking cameras are arranged on the VR virtual reality head; the VR virtual reality headset is used for shooting the glove through the gesture recognition tracking camera so as to acquire coordinate data of the glove relative to the gesture recognition tracking cameraAnd the coordinate data +.>Transmitting to the server client;
the server client comprises a space positioning module, a fitting module and a shifting module; wherein,,
the positioning module is used for positioning the position coordinatesIs +.>Determining the positional relationship between the centroid of said infrared marking point M1 and the centroid of said infrared marking point M2 +.>
The fitting module is used for taking the gesture recognition tracking camera as an origin coordinate and taking the coordinate dataIs->Performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate;
the shift module is used for translating and rotating the infrared marking point M1 to a coordinate system taking the gesture recognition tracking camera as an origin coordinate according to the rotation matrix and the translation vector, so that the hand skeleton point is marked on a gesture photo shot by the gesture recognition tracking camera.
Preferably, the number of the infrared tracking cameras is 40-50.
Preferably, the infrared tracking camera adopts a high-precision infrared tracking camera with a visual angle range of at least 55 degrees by 45 degrees, a frame rate of at least 180Hz, an exposure mode of Global router and an image resolution of 1080P.
Preferably, the curve fitting estimation is based on a least squares estimation algorithm.
Preferably, the infrared tracking camera is connected with the server client through a switch; the switch is used for transmitting the data acquired by the infrared tracking camera to the server client in real time.
Preferably, the gesture tracking camera adopts a camera with a visual angle range of at least 130×100°, a frame rate of at least 60Hz, an exposure mode of Global camera and an image resolution of VGA.
Preferably, the centroid of the infrared marking point M1 is the centroid of the geometric figure formed by all the infrared marking points M1;
the centroid of the infrared marking point M2 is the centroid of the geometric figure formed by all the infrared marking points M2.
The position coordinatesSaid position coordinates +.>Is->The coordinate system is a coordinate system with an infrared tracking camera as an origin of coordinates for the position relative to the closed space.
The invention also provides a gesture recognition data acquisition method, which comprises the following steps:
the infrared tracking camera is used for scanning the infrared marking points M1 and M2 on the glove and the VR virtual reality head so as to obtain the position coordinates of the glove in the closed spacePosition coordinates of the VR virtual reality head in the closed space>The infrared marking point M1 corresponds to the position of the hand skeleton point;
according to the position coordinatesIs +.>Determining the centroid of the infrared marking point M1 and the infraredPositional relationship between barycenters of the external mark points M2 +.>
Shooting the glove through the gesture recognition tracking camera worn by the VR virtual reality head to acquire coordinate data of the glove relative to the gesture recognition tracking cameraAnd taking the gesture recognition tracking camera as an origin coordinate, and taking the coordinate data +.>Is->Performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate;
and according to the rotation matrix and the translation vector, the infrared marking point M1 is translated and rotated to a coordinate system taking the gesture recognition tracking camera as an origin coordinate, so that the hand skeleton point is marked on a gesture photo shot by the gesture recognition tracking camera.
Preferably, the glove is photographed by a gesture recognition tracking camera worn by the VR virtual reality head to acquire coordinate data of the glove relative to the gesture recognition tracking cameraIn the course of (a) the process,
gesture recognition who VR virtual reality head was worn tracks the camera and shoots 20 at least infrared mark points M1 on the gloves.
According to the technical scheme, the gesture recognition data acquisition system and method provided by the invention are characterized in that the infrared tracking cameras are arranged on five surfaces in the closed space, and then the red is arranged on the glove and the VR virtual reality headThe infrared marking points M1 and M2 are scanned by an infrared tracking camera to obtain the position coordinates of the glove in the closed spacePosition coordinates of the VR virtual reality head in the closed space>And according to the position coordinates +.>Is +.>Determining the positional relationship between the centroid of said infrared marking point M1 and the centroid of said infrared marking point M2 +.>Then shooting the glove by a gesture recognition tracking camera worn by the VR virtual reality head to obtain coordinate data of the glove relative to the gesture recognition tracking camera 121>And the coordinate data->Relation with position->Performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marking point M2 relative to the origin coordinate, and translating and rotating the infrared marking point M1 to a coordinate system taking the gesture recognition tracking camera as the origin coordinate according to the rotation matrix and the translation vector so as to calibrate hand skeleton points to form high-precision picture training data on a gesture picture shot by the gesture recognition tracking camera, thereby reducing manual participation and improving marking efficiencyThe accuracy of the annotation is high, the accuracy of the network model is further improved, and the experience immersion of the user in the VR/AR/MR scene is improved.
Drawings
Other objects and attainments together with a more complete understanding of the invention will become apparent and appreciated by referring to the following description taken in conjunction with the accompanying drawings. In the drawings:
FIG. 1 is a schematic diagram of a gesture recognition data acquisition system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an application of a gesture recognition data acquisition system according to an embodiment of the present invention;
FIG. 3 is a schematic illustration of a glove in a gesture recognition data acquisition system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a gesture recognition data acquisition method according to an embodiment of the invention.
Detailed Description
The traditional method is that small parts of data are marked manually, network model training is carried out based on the data, recognition marking of other data is carried out through the trained model, then manual supervision and inspection are carried out, the error position information of marked points recognized by the model is corrected manually, further network model training is carried out continuously, the above processes are repeated in sequence, and finally a high-precision network model is obtained.
In view of the foregoing, the present invention provides a gesture recognition data acquisition system, and specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
In order to illustrate the gesture recognition data acquisition system provided by the present invention, fig. 1 exemplarily identifies a system structure of the gesture recognition data acquisition system according to an embodiment of the present invention, and fig. 2 exemplarily identifies an application of the gesture recognition data acquisition system according to an embodiment of the present invention.
The following description of the exemplary embodiment(s) is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses. Techniques and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but should be considered part of the specification where appropriate.
As shown in fig. 1, the gesture recognition data acquisition system 100 provided by the invention includes a closed space 110 formed by wall surfaces, and an infrared tracking camera 111, wherein the wall surfaces can be wall surfaces with practical significance, can also be any material with supporting property capable of forming the closed space, the brightness degree of the environment in the closed space 110 is adjustable, the infrared tracking camera is arranged on the wall surfaces of the closed space, can be four wall surfaces, can also be five wall surfaces, and the number is not particularly limited, in order to enhance the tracking effect, in the embodiment, the closed space 110 is provided with a plurality of infrared tracking cameras except the bottom surface, the infrared tracking cameras 111 are disposed on the remaining five surfaces, in this embodiment, the enclosed space 110 adopts a sealed room with a size of 3m by 3m (length by width by height), that is, except for the room floor, the infrared tracking cameras 111 are disposed on the remaining five walls, the specification of the infrared tracking cameras is not particularly limited, in this embodiment, a high-precision infrared tracking camera with a viewing angle range of at least 55 ° by 45 °, a frame rate of at least 180Hz, an exposure mode of Global shift, and an image resolution of 1080P is used to accurately capture each infrared mark point, so as to accurately determine the position of the object marked with the infrared mark point.
In the gesture recognition data acquisition system 100 shown in fig. 1 and fig. 2 together, the number of infrared tracking cameras 111 is 40-50, in this embodiment, 45 infrared tracking cameras 111 are adopted, and the installation of the infrared tracking cameras is performed on the whole enclosed space 110 according to a preset installation position and angle, each infrared tracking camera 111 needs to have a fixed installation angle in a three-dimensional space, so as to ensure that at least one infrared tracking camera of the 45 infrared tracking cameras can scan a visible area (with a length of 3 x 2.5 and a width of 2.5) in the enclosed space 110 under any condition (with a height of more than one meter, the visible area is 3 x 2.5), after the installation of the infrared tracking cameras 111 is completed, a coordinate system of the whole space is established, that is, in a room with a size of 3m x 3m, a fixed position is set as an origin of the enclosed space 110, then one infrared tracking camera in the infrared tracking system is arbitrarily designated as an origin, so that the relative position of the infrared tracking cameras can be acquired, and the relative position of the infrared tracking cameras can be designated.
As shown in fig. 3, in the gesture recognition data acquisition system 100 provided by the present invention, the surface of the glove is provided with the infrared marking points M1, the size and specification of the glove are not particularly limited, each infrared marking point M1 can be accurately set at each joint of the hand of the person, and 26 infrared marking points M1 are set on each glove according to a certain marking position and finger joint distribution, so that the infrared tracking camera 111 can accurately capture 26 infrared marking points M1.
As shown in fig. 1, fig. 2 and fig. 3 together, the gesture recognition data acquisition system 100 provided by the invention comprises a glove with infrared marking points M1 on the surface, wherein the size of the glove can be adjusted according to the sizes of different hands, the infrared marking points M1 are arranged at positions corresponding to hand skeleton points, so that after the glove is worn, the hand and the glove are tightly attached as much as possible, thereby accurately acquiring the skeleton point distribution of the hand, further, only one staff is required to carry the glove to do a plurality of preset hand actions in a closed space 110, the image training data with skeleton point marks can be acquired, further, a convolutional neural network model is trained, and the sense of immersion of virtual-real combination of VR/AR/MR is enhanced.
As shown in fig. 1 and fig. 2 together, the gesture recognition data acquisition system 100 provided by the present invention includes a VR virtual reality headset 120 with an infrared mark point M2 on a surface, the number of the infrared mark points M2 is not specifically limited, in this embodiment, it is ensured that the infrared tracking camera 111 can capture at least 4 infrared mark points M2 on the VR virtual reality headset 120, so that the infrared tracking camera 111 can accurately determine the position information of the VR virtual reality headset 120, at least two gesture recognition tracking cameras 121 are provided on the VR virtual reality headset 120, the model and the specification of the gesture recognition tracking cameras 121 are not specifically limited, in this embodiment, a camera with a viewing angle range of at least 130×100 °, a frame rate of at least 60Hz and an exposure mode of Global timer is adopted, and an image resolution of VGA is ensured, so as to accurately collect hand image data and prevent image distortion.
In addition, in the gesture recognition data acquisition system 100 shown in fig. 1 and fig. 2, the arrangement modes of the infrared tracking camera and the gesture recognition tracking camera at the respective positions are not particularly limited, in this embodiment, the infrared tracking camera and the gesture recognition tracking camera are calibrated cameras, that is, the infrared tracking camera and the gesture recognition tracking camera are both marked cameras with position information, and the calibration method adopts a Zhang Zhengyou calibration method.
In the gesture recognition data acquisition system 100 shown in fig. 1 and fig. 2, the infrared tracking camera 111 is connected with the VR virtual reality headset 120 by a server client 130, and a specific connection manner is not limited, in this embodiment, the VR virtual reality headset 120 is connected with the server client 130 by a wire, and the infrared tracking camera 111 is connected with the server client 130 by a switch; the switch is used for transmitting the data collected by the infrared tracking camera 111 to the server client 130 in real time, so that the server client can timely acquire the picture data shot by the infrared tracking camera.
Specifically, in the gesture recognition data acquisition system 100 shown in fig. 1 and 2, the infrared tracking camera 111 is used to scan the infrared marking points M1 and M2 to obtain the position coordinates of the glove in the enclosed space 110Position coordinates of the VR virtual reality headset 120 within the enclosure 110>And the position coordinates +.>And the position coordinatesTo the server client.
In the gesture recognition data acquisition system 100 shown in fig. 1 and 2, at least two gesture recognition tracking cameras are provided on the VR virtual reality headset 120; the VR virtual reality headset is configured to capture the glove via the gesture recognition tracking camera 121 to obtain coordinate data of the glove relative to the gesture recognition tracking camera 121Here, the coordinate data of the tracking camera 121 relative to the gesture recognition>Namely, the data of the glove in the VR virtual reality headset 120, also can be said to be coordinate data +.>Tracking the image data of the glove photographed by the camera 121 for the gesture recognition; the coordinate data is then->To the server client 130.
In the gesture recognition data acquisition system 100 shown in fig. 1 and 2, the server client 130 includes a spatial positioning module 131, a fitting module 132, and a shifting module 133, where the spatial positioning module 131 is configured to coordinate according to a positionAnd position coordinates->Determining the centroid of the infrared marker point M1 and the centroid of the infrared marker point M2Positional relationshipThe centroid of the infrared marking point M1 is the centroid of the geometric figure formed by all the infrared marking points M1, the centroid of the infrared marking point M2 is the centroid of the geometric figure formed by all the infrared marking points M2, the centroid is the centroid in the traditional sense, and the position coordinates are->Position coordinates->Relation with position->The positions of the glove and the VR virtual headset 120 are relative to the closed space, and the coordinate systems are coordinate systems with the infrared tracking camera 111 as the origin of coordinates, that is, the spatial positioning module 131 is configured to obtain the relative positions of the glove and the VR virtual headset 120 in the whole closed space 110 according to the positions of the infrared marking points M1 and M2.
In the gesture recognition data acquisition system 100 shown in fig. 1 and 2, the fitting module 132 is configured to take the gesture recognition tracking camera 121 as an origin coordinate and to use the coordinate dataIs>And performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate, wherein the curve fitting estimation is based on a least square estimation algorithm, namely, performing curve fitting on the image information of the glove shot by the gesture recognition tracking camera and the relative position information of the VR virtual reality headset 120 and the glove shot by the infrared tracking camera 111 through the least square estimation algorithm to obtain the relation between the centroid of the infrared marker point M2 on the VR virtual reality headset 120 and the gesture recognition tracking camera 121.
In the gesture recognition data acquisition system 100 shown in fig. 1 and 2, the shift module 133 is configured to translate and rotate the infrared marking point M1 to a coordinate system with the gesture recognition tracking camera 121 as an origin coordinate according to the rotation matrix and the translation vector, so that the hand skeleton point is calibrated on the gesture photo taken by the gesture recognition tracking camera 121, that is, the positional relationship between the centroid of the infrared marking point M1 on the glove and the centroid of the infrared marking point M2 on the VR virtual reality headset 120 isThe relation between the centroid of the infrared marking point M2 on the VR virtual reality headset 120 and the gesture recognition tracking camera 121 is obtained by the fitting module 132, so that the shifting module performs translational rotation according to the data obtained by the fitting module, marks the infrared marking point M1 on the glove on the gesture photo captured by the gesture recognition tracking camera, (just like the relation between a and B, the relation between B and C can be obtained, the relation between a and C can be obtained), thereby obtaining hand image data accurately marked with hand skeleton points, and inputting the hand image data as image training data into the convolutional neural network model, so as to complete high-precision training on the convolutional neural network model.
As shown in fig. 1, fig. 2 and fig. 3 together, the gesture recognition data acquisition system provided by the invention only needs a glove with infrared marking points M1 on two hands of a worker, a VR virtual reality head with the infrared marking points M2 is worn on the head, a plurality of specific hand movements are put on the closed space, and an infrared tracking camera arranged on the wall of the closed space scans the infrared marking points M1 and M2, so that the position coordinates of the glove are obtainedWith VR virtual reality head position coordinates +.>And position coordinates +.>And position coordinates->Transmitting to a server client, a gesture recognition tracking camera provided on the VR virtual reality head shoots a glove to acquire coordinate data of the glove with respect to the gesture recognition tracking camera 121->And the coordinate data->Transmitting to the server client, the server client including the coordinate data>Relation with position->And performing curve fitting, and performing rotary translation according to the data obtained by fitting to obtain a large number of hand images with accurate hand skeleton points, so that the hand images are used as image training data to be input into a convolutional neural network model, training of the convolutional neural network model is completed, gestures are automatically identified after the convolutional neural network model, a large number of image training data can be obtained by only swinging a person by a plurality of specific hand actions in the process, the defect of the traditional manual skeleton point identification method is overcome, the accuracy of the image identification skeleton points is improved, the training accuracy of the convolutional neural network model is improved, and the experience immersion of a user in a VR/AR/MR scene is further enhanced.
As shown in fig. 4, corresponding to the gesture recognition data acquisition system, the present invention further provides a gesture recognition data acquisition method, including:
s110: the infrared tracking camera is used for scanning the infrared marking points M1 and M2 on the glove and the VR virtual reality head so as to obtain the position coordinates of the glove in the closed spacePosition coordinates of the VR virtual reality head in the closed space +.>The infrared marking point M1 corresponds to the position of the hand skeleton point;
s120: according to the position coordinatesIs +.>Determining a positional relationship between the centroid of the infrared marker point M1 and the centroid of the infrared marker point M2>
S130: capturing the glove by the gesture recognition tracking camera worn by the VR virtual reality head to obtain coordinate data of the glove relative to the gesture recognition tracking camera 121And the gesture recognition tracking camera is used as an origin coordinate, and the coordinate data is +.>Is>Performing curve fitting estimation based on a least square estimation algorithm to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate;
s140: and translating and rotating the infrared marking point M1 to a coordinate system taking the gesture recognition tracking camera as an origin coordinate according to the rotation matrix and the translation vector, so that the hand skeleton point is marked on a gesture photo shot by the gesture recognition tracking camera.
In step S110, the user wears the glove during data acquisition, and performs several specific gesture actions in the closed space, so that the infrared tracking cameras and the gesture recognition tracking cameras worn by the VR virtual reality head can capture the infrared mark points on the glove corresponding to each gesture action.
In step S120, according to the position coordinatesAnd position coordinates->Determining the positional relationship between the centroid of the infrared marker point M1 and the centroid of the infrared marker point M2>The position coordinates +.>Position coordinates->Relationship with positionThe positions of the infrared tracking cameras are relative to the closed space, and the coordinate systems are coordinate systems with the infrared tracking cameras as the origin of coordinates.
In S130, the gesture recognition tracking camera photographs the glove to acquire coordinate data of the glove with respect to the gesture recognition tracking camera 121Namely, the position relation between the glove (the centroid of the infrared mark point M1) and the gesture recognition tracking camera is obtained, and the position relation is +.>Wearing infrared markers for centroid and VR virtual reality of glove infrared marker point M1 in closed spaceThe positional relationship of the centroid of point M2, coordinate data +.>Relation with position->And performing curve fitting to obtain the relation between the centroid of the VR virtual reality head-mounted infrared marking point M2 and the VR virtual reality head-mounted gesture recognition tracking camera, namely, a rotation matrix and a translation vector of the centroid of the infrared marking point M2 relative to the origin coordinate.
In S140, the infrared marking point M1 is translated and rotated to a coordinate system using the gesture recognition tracking camera as an origin coordinate according to the rotation matrix and the translation vector, so that a distance between the center of mass of the VR virtual reality head-mounted infrared marking point M2 and the relationship between the VR virtual reality head-mounted infrared marking point M2 and the gesture recognition tracking camera is obtained, and the position relationship between the VR virtual reality head-mounted and the glove in the coordinate of the closed space and the position relationship between the VR virtual reality head-mounted gesture recognition tracking camera and the glove in the coordinate of the VR virtual reality head-mounted can be achieved through moving and rotating, so that the infrared marking point M1 is completely covered on the hand image shot by the gesture recognition tracking camera, and the infrared marking point M1 corresponds to the hand skeleton point, thereby completing the skeleton point marking of the hand image.
According to the gesture recognition data acquisition method provided by the invention, the infrared tracking camera scans the infrared marking point M1 on the glove and the infrared marking point M2 on the VR virtual reality head to acquire the position coordinates of the glove in the closed spacePosition coordinates wearing with VR virtual reality head in closed space +.>Thereby determining the positional relationship between the centroid of the infrared marker point M1 and the centroid of the infrared marker point M2>Then, the glove is shot by the gesture recognition tracking camera worn by the VR virtual reality head so as to acquire coordinate data of the glove relative to the gesture recognition tracking camera 121>Coordinate data +.>Relation with position->And performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marking point M2 relative to the origin coordinate, calibrating hand skeleton points on a gesture photo shot by a gesture recognition tracking camera according to the rotation matrix and the translation vector, inputting the hand image as image training data into a convolutional neural network model, completing training of the convolutional neural network model, automatically recognizing the gesture after the convolutional neural network model, and acquiring a large amount of image training data by only swinging a person by a plurality of specific hand actions in the process, so that the method gets rid of the problem of the traditional manual marking skeleton point method, improves the accuracy of the image marking skeleton points, further improves the training accuracy of the convolutional neural network model, and further enhances the experience of a user in VR/AR/MR scenes.
The gesture recognition data acquisition system, method according to the present invention are described above by way of example with reference to the accompanying drawings. However, those skilled in the art will appreciate that various modifications may be made to the gesture recognition data collection system, method, and computer program product presented above without departing from the scope of the present invention. Accordingly, the scope of the invention should be determined from the following claims.

Claims (10)

1. The gesture recognition data acquisition system is characterized by comprising a closed space, an infrared tracking camera, a glove with an infrared marking point M1 arranged on the surface and a VR virtual reality head with an infrared marking point M2 arranged on the surface, wherein,
the infrared marking point M1 is arranged at a position corresponding to a hand skeleton point;
the infrared tracking camera is connected with the VR virtual reality headset through the server client;
the infrared tracking camera is arranged on the wall surface of the closed space and is used for scanning the infrared marking points M1 and M2 to obtain the position coordinates of the glove in the closed spacePosition coordinates wearing with the VR virtual reality head in the closed space +.>And coordinates of the position +.>Is +.>Transmitting to the server client;
at least two gesture recognition tracking cameras are arranged on the VR virtual reality head; the VR virtual reality headset is used for shooting the glove through the gesture recognition tracking camera so as to acquire coordinate data of the glove relative to the gesture recognition tracking cameraAnd the coordinate data +.>Transmitting to the server client;
the server client comprises a space positioning module, a fitting module and a shifting module; wherein,,
the positioning module is used for positioning the position coordinatesIs +.>Determining the positional relationship between the centroid of said infrared marking point M1 and the centroid of said infrared marking point M2 +.>
The fitting module is used for taking the gesture recognition tracking camera as an origin coordinate and taking the coordinate dataIs->Performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate;
the shift module is used for translating and rotating the infrared marking point M1 to a coordinate system taking the gesture recognition tracking camera as an origin coordinate according to the rotation matrix and the translation vector, so that the hand skeleton point is marked on a gesture photo shot by the gesture recognition tracking camera.
2. The gesture-recognition data collection system of claim 1,
the number of the infrared tracking cameras is 40-50.
3. The gesture-recognition data collection system of claim 1,
the infrared tracking camera adopts a high-precision infrared tracking camera with a visual angle range of at least 55 degrees and 45 degrees, a frame rate of at least 180Hz, a Global shift as an exposure mode and an image resolution of 1080P.
4. The gesture-recognition data collection system of claim 1,
the curve fitting estimation is based on a least squares estimation algorithm.
5. The gesture-recognition data collection system of claim 1,
the infrared tracking camera is connected with the server client through a switch; the switch is used for transmitting the data acquired by the infrared tracking camera to the server client in real time.
6. The gesture-recognition data collection system of claim 1,
the gesture tracking camera adopts a camera with a visual angle range of at least 130 degrees by 100 degrees, a frame rate of at least 60Hz, an exposure mode of Global camera and an image resolution of VGA.
7. The gesture-recognition data collection system of claim 1,
the centroid of the infrared marking point M1 is the centroid of the geometric figure formed by all the infrared marking points M1;
the centroid of the infrared marking point M2 is the centroid of the geometric figure formed by all the infrared marking points M2.
8. The gesture-recognition data collection system of claim 1,
the position coordinatesSaid position coordinates +.>And the bitPut relationship->The coordinate system is a coordinate system with an infrared tracking camera as an origin of coordinates for the position relative to the closed space.
9. A method for gesture recognition data acquisition, comprising:
the infrared tracking camera is used for scanning the infrared marking points M1 and M2 on the glove and the VR virtual reality head so as to obtain the position coordinates of the glove in the closed spacePosition coordinates of the VR virtual reality head in the closed space>The infrared marking point M1 corresponds to the position of the hand skeleton point;
according to the position coordinatesIs +.>Determining the positional relationship between the centroid of said infrared marking point M1 and the centroid of said infrared marking point M2 +.>
Shooting the glove through the gesture recognition tracking camera worn by the VR virtual reality head to acquire coordinate data of the glove relative to the gesture recognition tracking cameraAnd the gesture recognition tracking camera is taken as an originCoordinates, the coordinate data +.>Is->Performing curve fitting estimation to obtain a rotation matrix and a translation vector of the centroid of the infrared marker point M2 relative to the origin coordinate;
and according to the rotation matrix and the translation vector, the infrared marking point M1 is translated and rotated to a coordinate system taking the gesture recognition tracking camera as an origin coordinate, so that the hand skeleton point is marked on a gesture photo shot by the gesture recognition tracking camera.
10. The method of claim 9, wherein the glove is captured at a gesture recognition tracking camera worn by the VR virtual reality head to obtain coordinate data of the glove relative to the gesture recognition tracking cameraIn the course of (a) the process,
gesture recognition who VR virtual reality head was worn tracks the camera and shoots 20 at least infrared mark points M1 on the gloves.
CN202010674342.7A 2020-05-12 2020-07-14 Gesture recognition data acquisition system and method Active CN111860275B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN2020103970912 2020-05-12
CN202010397091 2020-05-12

Publications (2)

Publication Number Publication Date
CN111860275A CN111860275A (en) 2020-10-30
CN111860275B true CN111860275B (en) 2023-11-03

Family

ID=72983413

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010674342.7A Active CN111860275B (en) 2020-05-12 2020-07-14 Gesture recognition data acquisition system and method

Country Status (1)

Country Link
CN (1) CN111860275B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112416134A (en) * 2020-12-10 2021-02-26 华中科技大学 Device and method for quickly generating hand key point data set

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103927016A (en) * 2014-04-24 2014-07-16 西北工业大学 Real-time three-dimensional double-hand gesture recognition method and system based on binocular vision
WO2016132371A1 (en) * 2015-02-22 2016-08-25 Technion Research & Development Foundation Limited Gesture recognition using multi-sensory data
CN108196679A (en) * 2018-01-23 2018-06-22 河北中科恒运软件科技股份有限公司 Gesture-capture and grain table method and system based on video flowing

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10496353B2 (en) * 2016-09-29 2019-12-03 Jiang Chang Three-dimensional image formation and color correction system and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103927016A (en) * 2014-04-24 2014-07-16 西北工业大学 Real-time three-dimensional double-hand gesture recognition method and system based on binocular vision
WO2016132371A1 (en) * 2015-02-22 2016-08-25 Technion Research & Development Foundation Limited Gesture recognition using multi-sensory data
CN108196679A (en) * 2018-01-23 2018-06-22 河北中科恒运软件科技股份有限公司 Gesture-capture and grain table method and system based on video flowing

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于数据手套和双目视觉技术的虚拟手势及其空间位置跟踪;吕美玉;侯文君;陈军;;北京邮电大学学报(第06期);全文 *
面向手语自动翻译的基于Kinect的手势识别;付倩;沈俊辰;张茜颖;武仲科;周明全;;北京师范大学学报(自然科学版)(第06期);全文 *

Also Published As

Publication number Publication date
CN111860275A (en) 2020-10-30

Similar Documents

Publication Publication Date Title
CN105809701B (en) Panoramic video posture scaling method
JP5586765B2 (en) Camera calibration result verification apparatus and method
CN107155341B (en) Three-dimensional scanning system and frame
US20070098250A1 (en) Man-machine interface based on 3-D positions of the human body
CN110825234A (en) Projection type augmented reality tracking display method and system for industrial scene
CN103337094A (en) Method for realizing three-dimensional reconstruction of movement by using binocular camera
CN108830906A (en) A kind of camera parameters automatic calibration method based on virtual Binocular Vision Principle
CN112767300A (en) Method for automatically generating labeling data of hand and method for calculating skeleton length
JP5079547B2 (en) Camera calibration apparatus and camera calibration method
CN112949478A (en) Target detection method based on holder camera
JP4948552B2 (en) Camera calibration apparatus, camera calibration method, camera calibration program, and recording medium recording the program
US11386578B2 (en) Image labeling system of a hand in an image
CN114119739A (en) Binocular vision-based hand key point space coordinate acquisition method
CN113077519A (en) Multi-phase external parameter automatic calibration method based on human skeleton extraction
CN114283203A (en) Calibration method and system of multi-camera system
CN105374067A (en) Three-dimensional reconstruction method based on PAL cameras and reconstruction system thereof
JP2023546739A (en) Methods, apparatus, and systems for generating three-dimensional models of scenes
Kassebaum et al. 3-D target-based distributed smart camera network localization
CN111860275B (en) Gesture recognition data acquisition system and method
CN209231985U (en) A kind of camera parameters automatic calibration device based on virtual Binocular Vision Principle
CN112183316B (en) Athlete human body posture measuring method
CN113421286B (en) Motion capturing system and method
JP2006227932A (en) Three-dimensional analysis method from two-dimensional image, and system for executing same
CN104460971A (en) Human motion rapid capturing method
CN113345010A (en) Multi-Kinect system coordinate calibration and conversion method based on improved ICP

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant