CN117519487A - Development machine control teaching auxiliary training system based on vision dynamic capture - Google Patents
Development machine control teaching auxiliary training system based on vision dynamic capture Download PDFInfo
- Publication number
- CN117519487A CN117519487A CN202410015789.1A CN202410015789A CN117519487A CN 117519487 A CN117519487 A CN 117519487A CN 202410015789 A CN202410015789 A CN 202410015789A CN 117519487 A CN117519487 A CN 117519487A
- Authority
- CN
- China
- Prior art keywords
- user
- gesture
- node
- unit
- development machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000011161 development Methods 0.000 title claims abstract description 84
- 238000012549 training Methods 0.000 title claims abstract description 56
- 238000004088 simulation Methods 0.000 claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 53
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 26
- 230000006870 function Effects 0.000 claims description 41
- 238000010586 diagram Methods 0.000 claims description 33
- 238000001514 detection method Methods 0.000 claims description 30
- 238000004891 communication Methods 0.000 claims description 24
- 230000000007 visual effect Effects 0.000 claims description 20
- 230000033001 locomotion Effects 0.000 claims description 18
- 230000009471 action Effects 0.000 claims description 17
- 238000005520 cutting process Methods 0.000 claims description 17
- 238000013507 mapping Methods 0.000 claims description 17
- 239000003245 coal Substances 0.000 claims description 16
- 238000007781 pre-processing Methods 0.000 claims description 14
- 230000008859 change Effects 0.000 claims description 13
- 230000000694 effects Effects 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 12
- 238000001914 filtration Methods 0.000 claims description 10
- 239000002245 particle Substances 0.000 claims description 8
- 238000013461 design Methods 0.000 claims description 6
- 210000000744 eyelid Anatomy 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 239000004575 stone Substances 0.000 claims description 5
- 230000001629 suppression Effects 0.000 claims description 5
- 241000203475 Neopanax arboreus Species 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 238000005507 spraying Methods 0.000 claims description 4
- 238000009412 basement excavation Methods 0.000 claims description 3
- 238000005553 drilling Methods 0.000 claims description 3
- 239000011435 rock Substances 0.000 claims description 3
- 238000010845 search algorithm Methods 0.000 claims description 3
- 239000000779 smoke Substances 0.000 claims description 3
- 239000002689 soil Substances 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 2
- 230000000873 masking effect Effects 0.000 claims 1
- 230000003993 interaction Effects 0.000 abstract description 7
- 230000018109 developmental process Effects 0.000 description 61
- 230000000875 corresponding effect Effects 0.000 description 17
- 230000001960 triggered effect Effects 0.000 description 11
- 238000005065 mining Methods 0.000 description 8
- 238000007689 inspection Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000033228 biological regulation Effects 0.000 description 3
- 210000003128 head Anatomy 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000295 fuel oil Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000012535 impurity Substances 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000007921 spray Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B5/00—Electrically-operated educational appliances
- G09B5/02—Electrically-operated educational appliances with visual presentation of the material to be studied, e.g. using film strip
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention relates to the technical field of teaching auxiliary training, and discloses a development machine control teaching auxiliary training system based on vision dynamic capture, which comprises a camera unit, a display unit and a processing unit; the camera shooting unit is used for shooting hand and eye gestures of a user in real time; the processing unit comprises a gesture recognition processing unit and a simulated heading machine operating system; the gesture recognition processing unit judges gesture commands and concentration degrees of attention of a user by detecting key node space data of hands and eyes of the user; the simulation heading machine operating system is used for receiving a command to realize control and transmitting an operation and environment feedback picture to the display unit. The hand gesture is accurately identified by adopting a plurality of algorithms, and the system robustness is high; the simulation algorithm is combined to simulate the real working condition to the greatest extent, so that the practicability is high; the monocular vision method is adopted to capture the hand gestures to realize human-computer interaction, so that the requirements on hardware equipment are low, a plurality of platforms can be deployed, and the method is easy to popularize and use.
Description
Technical Field
The invention relates to the technical field of teaching auxiliary training, in particular to a development machine control teaching auxiliary training system based on vision dynamic capture.
Background
In recent years, the phenomenon of population aging in China is obvious, labor cost is increased year by year in labor-intensive industries such as coal industry, and the number of skilled laborers is reduced year by year. In actual production, the underground operation working condition is complex, accidents are easily caused by improper operation of a development machine for mining coal mines, and a driver is more required to have enough driving experience, so that the operation strictly accords with national safety production method and coal mine safety training regulations. In this way, the operation technical level and the proficiency of the driver are improved, so that the driver can fully grasp the operation points, the production efficiency is improved, the complex working conditions possibly encountered in the production can be effectively treated, and the production accidents are avoided.
At present, the operation exercise of the heading machine is limited in many ways, the traditional live-action operation adopts a teaching mode of a teacher, the training cost of fuel oil, equipment and the like is high, the utilization rate is low, the requirement on safety guarantee is high, and students cannot directly operate the equipment exercise. The simulation operation teaching device is combined with a device model through computer simulation, for example, a coal mine development machine simulation operation teaching device disclosed in China patent (CN 218214438U) gives physical feedback to a user in a training process through a designed bottom plate device, irregular excessive vibration of a simulation cabin very influences human-computer interaction experience, training efficiency is reduced, a novel development machine operation training system disclosed in China patent (CN 213518763U) comprises a training special space, a simulation development machine model, a mixed reality helmet and a training control platform, operation experience is more real, the whole system has high requirements on equipment and sites, training process is more complex, training cost is greatly improved, a development machine virtual practical training operation teaching device disclosed in China patent (CN 109509374A) and a training method are combined with hardware through computer software, the simulation development machine is controlled through a workbench, but operation is complex, fatigue of the user in the training process is not considered, and the problems of concentration of attention, efficiency and performance are caused. Meanwhile, the teaching equipment is controlled through a physical operation table, and relates to a certain mechanical structure, so that certain requirements are met on equipment and sites, and the problem of cost exists.
For the problems in the related art, no effective solution has been proposed at present.
Disclosure of Invention
Aiming at the problems in the related art, the invention provides a development machine control teaching auxiliary training system based on visual dynamic capturing, which aims to overcome the technical problems in the related art.
For this purpose, the invention adopts the following specific technical scheme:
a development machine control teaching auxiliary training system based on visual dynamic capturing comprises a camera unit, a display unit and a processing unit;
the camera shooting unit is used for shooting hand and eye gestures of a user in real time;
the display unit is used for displaying the operation picture of the simulated heading machine and the feedback effect of the simulation environment in real time;
the processing unit comprises a gesture recognition processing unit and a simulated heading machine operating system;
the gesture recognition processing unit judges a gesture command and a concentration degree of attention of a user by detecting key node space data of hands and eyes of the user on the basis of solving the problem of covering high brightness points of the video and gestures of the user;
the simulation heading machine operating system is used for receiving a command to realize control and transmitting an operation and environment feedback picture to the display unit.
Further, the gesture recognition processing unit comprises a preprocessing unit, a covering processing unit, a key point detection unit and a communication unit;
The preprocessing unit performs high-brightness point removal processing on the collected real-time video data, sends the processed real-time video data to the covering processing unit, divides a screen display area, establishes a mapping relation between the screen area and an actual space, sends the mapping relation to the key point detection unit, detects key node space coordinate data of the hand and the eye of a user, and outputs command signals to the communication unit after judging the concentration degree of gesture commands and the user attention, and the communication unit transmits the signals to the simulation development machine operation system through TCP/IP communication.
Further, the preprocessing unit performs the processing of removing the high brightness point on the collected real-time video data, which comprises the following steps:
converting a video picture from an RGB color space to an HSV color space by analyzing the highlight characteristic of real-time video data, wherein the formula of the pixel value of any point in the video picture image is as follows:
；
wherein, I represents a pixel value,respectively represent the image at +.>Tone value, saturation value, luminance value of a dot,/->Expressed as a set of all pixels in the image;
extracting images of the highlight region of the real-time video surface frame by setting saturation and brightness threshold values, and processing the highlight region by using an image average filtering mode;
And carrying out image fusion on the processed highlight region and the non-highlight region in the original image, and outputting the fused video to a covering processing unit.
Further, the covering processing unit, when dividing the screen display area, establishes a mapping relationship between the screen area and the actual space, and sends the mapping relationship to the key point detection unit, includes:
defining a hand movement area, and dividing a screen display area into a camera shooting areaGesture motion areaPalm frame option area->Screen display area->The gesture action area and the screen display area have a proportional relation m:
；
the lower left corner of the screen display area and the lower left corner of the gesture action area are taken as the origin of coordinates (0, 0), and the space coordinates of the key nodes of the hand are relative to the gesture action areaDenoted as (/ -)>) Is +_ relative to the screen display area>Denoted as (/ -)>) Length and width of screen display areaIs->、/>In the gesture motion, the coordinate position is in the gesture motion zone +.>And Screen display area +.>There is a mapping relation between:
；
；
and establishing gesture block diagrams, sorting all the gesture block diagrams according to the confidence coefficient by using a non-maximum value suppression algorithm, selecting the gesture block diagram with the highest current confidence coefficient, starting to circularly traverse the video, adding the gesture block diagram with the highest current confidence coefficient into a final selection list, removing other gesture block diagrams with high overlap degree with the currently selected gesture block diagram, updating a set of the rest gesture block diagrams, searching the gesture block diagram with the highest confidence coefficient in the updated set, repeating the operation until all the gesture block diagrams are processed, and outputting the gesture block diagram with the highest confidence coefficient after the processing is finished to a key point detection unit.
Further, detecting the space coordinate data of the key nodes of the hand of the user and performing gesture judgment includes:
judging whether a user index finger fingertip node stays for 2s in a single virtual key function area of ten virtual key function areas of a screen display area, wherein the index finger fingertip node has depth change exceeding 5 cm in the direction perpendicular to the camera unit, if so, triggering a corresponding virtual key function, and if not, not triggering the virtual key function;
judging whether a user forms a similar real holding rod action through five-finger aggregation, wherein the whole hand stays for 2s on a single virtual push rod of four push rod function areas of a screen display area, the depth of the whole hand in the direction vertical to the camera unit is changed by more than 5 cm, if so, the corresponding push rod function is triggered, and if not, the push rod function is not triggered;
judging whether a user combines the index finger with the middle finger, and if so, triggering a corresponding feed route setting function, wherein the two-finger fingertip node has a depth change of more than 5 cm in the direction perpendicular to the camera shooting unit in the screen display area, namely, the space coordinate positions of the two-finger fingertip node are set feed starting points and end points, connecting the two-finger fingertip node according to the sequence set by the user by using an A-type search algorithm to obtain a virtual feed route, and if not, not setting the feed starting points and the end points;
Judging whether the user holds a fist with both hands, if so, triggering a scram command, and if not, not triggering the scram command;
and outputting signals corresponding to the key gesture command, the push rod gesture command, the feed gesture command and the scram gesture command to a communication unit for driving the corresponding simulated heading machine function.
Further, detecting the space coordinate data of the key nodes of the hand of the user and performing gesture judgment further includes:
processing the high-frequency jitter problem in the acquisition process by using a self-adaptive centroid-Kalman filtering algorithm;
training a gesture detection model by using a convolutional neural network, and performing auxiliary judgment on the acquired gesture by using the trained gesture detection model.
Further, detecting the space coordinate data of the key nodes of the hands and eyes of the user, and judging the concentration degree of the user comprises:
counting the hand operation frequency of the user in real time, detecting eye key nodes of the user, judging the concentration degree of the user by taking the time interval of the hand operation sending command of the user as a basis, and judging that the attention is not concentrated when the average time interval of the operation command is exceeded;
acquiring space coordinate data of key nodes of two eyes of a user through an image pickup unit, and respectively calculating aspect ratios EAR of the left eye and the right eye of the user:
；
Wherein EAR represents an aspect ratio,representing six node space coordinate data of outer canthus, left and right upper eyelid, left and right inner canthus and left lower eyelid in turn, +.>Represents an L2 norm;
taking the average value of the aspect ratios of the left eye and the right eye of the user as the final eye aspect ratio, judging whether the final eye aspect ratio exceeds a threshold range, if so, judging that the attention is not concentrated, if not, judging whether the attention is concentrated by combining the hand operation frequency of the user, and if the hand operation frequency of the user judges that the attention is not concentrated, judging that the attention is not concentrated.
Furthermore, the simulation development machine operation system comprises a login unit, a theoretical learning unit and a development machine operation training unit;
the user in the login unit selects a virtual keyboard key to log in personal account information by using a command signal sent by the communication unit, the user in the theory learning unit learns the operation theory knowledge, the operation specification and the operation course of the development machine, and after the learning theory knowledge is finished, the development machine operation training unit automatically opens a third visual angle picture of the simulation development machine, the user selects all parts of the development machine and a roadway in the simulation environment by using gestures, and the simulation development machine is controlled by using the command signal sent by the communication unit.
Further, in a feed route setting command in the operation stage of the development machine, the development machine operation training unit automatically prompts rock and soil properties and tunnel design requirement setting, and gives recommended development speed, cutter head rotating speed and propulsion force parameters, a user performs data setting by using index finger tip nodes, and when the feed route nodes set by the user are too concentrated and the effective excavation rate is too low, the simulation development machine operation system automatically optimizes a route according to an A-based route searching algorithm, searches an optimal short route and prompts the user to improve;
wherein, automatically optimizing the path according to the a-routing algorithm to find the optimal short path comprises:
rasterizing a feed route picture set by a user, initializing and setting an open list and a closed list, setting a starting point set by the user to be in the open list, and traversing the open list to calculate a cost function F of each node:
where G is the Euclidean distance from the starting point to the current node, H is the Manhattan distance from the current node to the end point,two-dimensional space coordinates parallel to the display unit;
searching a node a with the minimum cost function F in the list, taking the node a as a node to be processed currently, and repeating the following operations:
firstly), processing all nodes adjacent to the node a to be processed currently, if the nodes are not reachable or in a closed list, ignoring the nodes, otherwise, continuing to process;
Secondly), if the adjacent node b is not in the open list, adding the adjacent node b into the open list, setting the current node a to be processed as a father node, setting the node b as the current node to be processed, calculating F, G, H value of the node b, if the adjacent node b is in the open list, checking whether the path from the starting point to the node b is better, referring to the standard as G value, if the G value is smaller, the path effect is better, automatically taking the father node e as the current node to be processed, and recalculating G, F value;
thirdly), moving the node a processed by the second step to a closed list, and paying no attention to the node a;
the procedure is terminated when any one of the following conditions is satisfied;
adding the condition I and the terminal point into an open list;
condition two, can't find the terminal point, and open the list to be empty at this moment;
if the endpoint has been found, find the shortest path: starting from the end point, each node moves along the father node until the start point, and the shortest path is the optimal short path.
Furthermore, the simulation environment in the display unit utilizes particles and a light source module in the Unity3D to realize the smoke, water spraying and light simulation effect when the heading machine works, utilizes a grid deformation algorithm to control the recession degree of a terrain grid by taking a cutting head as a key element of the terrain grid deformation, and utilizes a reference deformation formula to change the coordinates of the grid vertexes after the vertexes around the collision point of the cutting head are screened, and to preset stone models in different shapes to cooperate with the particle effect to realize the visual simulation of the crushing state of the coal seam after drilling;
Wherein, the reference deformation formula is:
in (x) Variable ，y Variable ) Representing the changed coordinates, (x) Original source ，y Original source ) The coordinates before the change are indicated,for the diameter of the tool>Is the cutting corner->For depth of cut->For the cutting force of the tool->Is the angular velocity of the tool.
The beneficial effects of the invention are as follows:
1) The novel development machine control teaching auxiliary training system provided by the invention combines the visual dynamic capturing and virtual reality technologies, adopts various algorithms to accurately identify gestures, greatly improves the robustness of the system, combines the simulation algorithm to simulate real working conditions to the greatest extent, has strong practicability, fully considers human-computer interaction experience, is simple and convenient to operate, improves the training efficiency, shortens the training period, adopts a monocular visual method to capture hand gestures, has low requirements on hardware equipment, can be deployed on a plurality of platforms, and is easy to popularize and use.
2) According to the method, gestures are accurately identified by adopting various algorithms, and the preprocessing unit optimizes the video quality by adopting the designed algorithm aiming at high-brightness points in the video, so that the influence of high light on subsequent operation is avoided; aiming at the problem of operation failure caused by gesture coverage of a user, a coverage unit establishes a screen mapping area through a non-maximum suppression algorithm, and the corresponding relation of an operation area in a picture is defined, so that the stability and accuracy of identification are greatly improved; the key point detection unit removes space data jitter and reduces errors by using a self-adaptive centroid-Kalman filtering algorithm, the space data of the key nodes of the hand of the user are rapidly and accurately detected and judged by using a neural network, the functions of the virtual keys and the virtual push rods are triggered according to the gestures of the user, and the robustness of the system is greatly improved by combining multiple algorithms.
3) According to the invention, the gesture operation is convenient, the rapid entry operation of a user is convenient, the explanation time of a coach in the traditional live-action operation is greatly shortened by a theoretical learning part, the safety of the user is ensured, each operation item can be repeatedly learned and an operation report is provided for the user to multiplex, the system environment of the simulation development machine simulates the real digging and mining working condition to the greatest extent through a Unity3D particle system and a grid deformation algorithm, the simulation development machine operation system judges the concentration degree of the attention of the user according to the gesture operation frequency and the eye gesture characteristics of the user, the reminding is given to the user, the man-machine interaction experience is improved, the operation efficiency is improved, and the training period is shortened.
4) The system can operate in a cross-platform mode depending on monocular RGB video input, the configuration environment is simple, the computer is supported to generate real-time detection by using a CPU, the frame rate on the notebook is 15-30 fps, the delay is less than 50ms, the designed virtual keys and virtual push rods replace operation hardware in traditional training, the training cost is saved, and compared with a similar frame, the system has obvious advantages in application scenes and is easy to popularize and use.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic block diagram of a development machine manipulation teaching auxiliary training system based on visual dynamic capturing according to an embodiment of the present invention.
Detailed Description
For the purpose of further illustrating the various embodiments, the present invention provides the accompanying drawings, which are a part of the disclosure of the present invention, and which are mainly used to illustrate the embodiments and, together with the description, serve to explain the principles of the embodiments, and with reference to these descriptions, one skilled in the art will recognize other possible implementations and advantages of the present invention, wherein elements are not drawn to scale, and like reference numerals are generally used to designate like elements.
According to the embodiment of the invention, a development machine control teaching auxiliary training system based on visual dynamic capturing is provided.
The invention is further described with reference to the accompanying drawings and the specific implementation, as shown in fig. 1, according to one embodiment of the invention, a development machine control teaching auxiliary training system based on visual dynamic capturing is provided, and the development machine control teaching auxiliary training system comprises a camera unit, a display unit and a processing unit;
the camera shooting unit is used for shooting hand and eye gestures of a user in real time;
the display unit is used for displaying the operation picture of the simulated heading machine and the feedback effect of the simulation environment in real time;
Specifically, the display unit is 1280×720 pixels, and real-time video acquired by the camera unit, the operation flow of the simulated development machine and virtual environment feedback are displayed. And the image pickup unit and the display unit are integrated.
The processing unit comprises a gesture recognition processing unit and a simulated heading machine operating system; in this embodiment, the processing unit is preferably a computer, and the computer processor is an i5-8300H, 64-bit operating system.
The gesture recognition processing unit judges a gesture command and a concentration degree of attention of a user by detecting key node space data of hands and eyes of the user on the basis of solving the problem of covering high brightness points of the video and gestures of the user;
specifically, the gesture recognition processing unit comprises a preprocessing unit, a covering processing unit, a key point detection unit and a communication unit, wherein the preprocessing unit is used for preprocessing a real-time video picture acquired by the camera shooting unit and then sending the real-time video picture to the covering processing unit, the covering processing unit is used for receiving the preprocessed picture to conduct covering processing, dividing a screen display area, establishing a mapping relation between the screen area and an actual space and sending the mapping relation to the key point detection unit, the key point detection unit is used for detecting space coordinate data of key nodes of hand gestures and eye gestures of a user and outputting command signals to the communication unit after judging, and the communication unit is used for sending the signals to an operation system of the simulation development machine.
The simulation heading machine operating system is used for receiving a command to realize control and transmitting an operation and environment feedback picture to the display unit.
Specifically, the simulated development machine operation system comprises a login unit, a theoretical learning unit and a development machine control unit, wherein a user in the login unit selects a virtual keyboard key to log in personal account information by using a command signal sent by the communication unit, the user in the theoretical learning unit learns the development machine operation theoretical knowledge, operation specifications and operation courses, and the development machine operation training unit receives the command signal sent by the communication unit to control the simulated development machine.
In this embodiment, the operation method of the development machine control teaching auxiliary training system based on visual dynamic capturing includes the following steps:
s1, acquiring hand and eye gestures of a user in real time by a camera unit, and transmitting video data to a computer;
s2, the preprocessing unit removes highlight points from video data, the processed video is sent to the covering processing unit for covering processing, a screen display area is divided, a mapping relation between the screen area and an actual space is established and sent to the key point detection unit, the key point detection unit detects space coordinate data of key nodes of hands and eyes of a user, a command signal is output to the communication unit after judging concentration degree of gestures and the user, and the communication unit transmits the signal to an operation system of the simulated development machine through TCP/IP communication;
Specifically, in step S2, the preprocessing unit converts the video frame from the RGB color space to the HSV color space by analyzing the highlight characteristic of the video, and the formula of the pixel value of any point in the image is as follows:
wherein,respectively represent the image at +.>Tone value, saturation value, luminance value of a dot,/->The method comprises the steps of representing the image as a set of all pixel points in the image, extracting the highlight region image of the video surface frame by setting saturation and brightness threshold values, then outputting the processed video to a shade by using an image average filtering mode (namely, overlapping and fusing a current region image and a non-highlight region image in an original image, synthesizing two input images to obtain higher-quality output)And a cover processing unit.
Specifically, in step S2, the covering processing unit defines the mobile region of the camera, and divides the screen display region into the shooting regions of the cameraGesture action zone->Palm frame option area->Screen display area->The gesture action area and the screen display area have a proportional relation +.>：
The lower left corner of the screen display area and the lower left corner of the gesture action area are taken as the origin of coordinates (0, 0), and the space coordinates of the key nodes of the hand are relative to the gesture action areaDenoted as (/ -)>) Is +_ relative to the screen display area >Denoted as (/ -)>) The length and width of the screen display area are recorded as +.>、/>. In the gesture motion, the coordinate position is in the gesture motion area +.>And Screen display area +.>There is a mapping relation between:
the method comprises the steps of guaranteeing the captured integrity in palm movement through establishing a block diagram, sorting the block diagrams with a plurality of blocks according to confidence (the degree that the true value of the block diagram has a certain probability to fall around a measurement result) by using a non-maximum suppression algorithm due to the fact that the number of gesture block diagrams recognized by a system is large, selecting a gesture block diagram with the highest current confidence, starting to circularly traverse videos, adding the gesture block diagram with the highest current confidence into a final selection list, removing other gesture block diagrams with high overlapping degree with the currently selected gesture block diagram, updating a set of the rest gesture block diagrams, searching the gesture block diagram with the highest confidence in the updated set, repeating the operation until all the gesture block diagrams are processed, and outputting the gesture block diagram with the highest confidence after processing to a key point detection unit.
Specifically, in step S2, the key point detection unit detects the space data of the key nodes of the hand of the user according to the prediction frame, solves the problem of high-frequency jitter in the acquisition process by adopting an adaptive centroid-kalman filtering algorithm, weights each identified key point of the single hand, calculates the approximate centroid of the hand by weighted summation, carries out kalman filtering on the space coordinate data of the centroid, predicts the state of the current time step based on the state estimation of the previous time step and the system model, compares the predicted value with the actual measured value, adjusts the predicted value by using the kalman gain to obtain updated state estimation, updates the state covariance and the process noise covariance according to the updated state estimation and the actual measured value, and carries out the prediction-update-covariance update-weight update until the data approaches to the true value, thereby filtering out the significant jump points.
Specifically, in step S2, the key point detection unit triggers the virtual key, the virtual push rod, the virtual feed route and the emergency stop function according to the gesture of the user, the single virtual key stays for 2S in ten virtual key function areas of the screen display area at the tip node of the index finger of the user, the tip node of the index finger has a depth change exceeding 5 cm in the direction perpendicular to the camera shooting unit, the corresponding virtual key function is triggered, the user forms a similar real holding rod action through five-finger aggregation, the whole hand stays for 2S in the single virtual push rod of four push rod function areas of the screen display area, the whole hand has a depth change exceeding 5 cm in the direction perpendicular to the camera shooting unit, the corresponding push rod function is triggered, the user merges through the index finger and the middle finger, the two-finger tip node has a depth change exceeding 5 cm in the direction perpendicular to the camera shooting unit in the screen display area, the corresponding feed route setting function is triggered, the space coordinate positions of the two-finger nodes are the set feed start point and the end point, the virtual feed route is obtained by combining the sequence set by the user according to the a, the gesture seeking algorithm, the user is enabled to connect, the virtual feed route, the user triggers the push rod command, the emergency stop command, the emergency gesture command and the emergency gesture command is sent by the user, the gesture command is simulated gesture command and the gesture is corresponding to the gesture of the hand gesture.
Specifically, in step S2, the key point detection unit trains the gesture detection model by using the convolutional neural network aiming at inaccurate gesture recognition of the user, so as to improve the gesture detection accuracy, the convolutional neural network extracts a large number of keys, push rods, feed and scram gesture actions which are designed in a collecting manner through the camera unit, builds the convolutional neural network for training by using the TensorFlow as a training data set, continuously adjusts the weight matrix and bias parameters to improve the model performance, and guides the final model into the key point detection unit to accurately judge the gesture.
Specifically, in step S2, the key point detection unit counts the hand operation frequency of the user in real time and detects the eye key nodes of the user, the former uses the time interval of the hand operation sending command of the user as the basis, judges the concentration degree of the user, and judges that the attention is not concentrated when the average time interval exceeds the operation command, the latter obtains the space coordinate data of the two eyes of the user through the image capturing unit, and calculates the eye aspect ratio EAR:
in the middle ofRepresenting six node space coordinate data of outer canthus, left and right upper eyelid, left and right inner canthus and left lower eyelid in turn, +.>Represents the L2 norm for calculating the distance between the two vectors. The user's left and right eyes are substituted into the formula to calculate, the average value is taken as the final data, and the average EAR value is 0.2,0.4 after a large amount of data statistics ]In the range, the eye-closing value is close to 0, when the eye-closing degree of the user is close to the threshold value, the user is reminded by combining the hand operation frequency, the concentration degree of the user is ensured, and the human-computer interaction experience is improved.
Specifically, in step S2, the communication module includes: the data packet is transmitted to another network by using the network, the gesture recognition processing unit creates a socket, binds the local network information into a network protocol, and then sends the command signal character string triggered by the gesture by using a sendto function. The method comprises the steps of creating a socket by an operating system end of the simulation development machine, setting a designated port number, creating a user object, binding to a node, connecting a client IP, and creating an array to receive data.
S3, automatically opening a virtual keyboard on a login interface of an operation system of the simulation development machine, enabling a user to click the virtual keyboard by using gestures to send command signals to log in an account, automatically opening a third visual angle picture of the simulation development machine by using the operation system of the simulation development machine in operation practice after learning theoretical knowledge is finished, enabling the user to select each part of the development machine and a roadway in the simulation environment by using gestures to perform safety inspection before operation, automatically opening a virtual key panel by using the operation system after the inspection is finished, enabling a finger tip node of a user to stay for 2 seconds in a single virtual key in ten virtual key function areas of a screen display area, enabling the finger tip node of the index finger to have depth change exceeding 5 cm in a direction perpendicular to a camera unit, triggering a corresponding virtual key function, enabling the user to form a similar real holding rod action through five-finger aggregation, the method comprises the steps that the whole hand stays for 2 seconds in a single virtual push rod of four push rod functional areas of a screen display area, the depth change of the whole hand in the direction perpendicular to a camera shooting unit exceeds 5 cm, the corresponding push rod function is triggered, a user is combined through an index finger and a middle finger, the depth change of a two-finger fingertip node in the direction perpendicular to the camera shooting unit in the screen display area exceeds 5 cm, the corresponding feed route setting function is triggered, the space coordinate positions of the two-finger fingertip node are set feed starting points and end points, connection is conducted according to the sequence set by an A-type search algorithm in combination with the user, a virtual feed route is obtained, the user holds a fist with two hands to trigger an emergency stop command, the key point detection unit outputs a corresponding command signal according to the designed gesture action, and the simulation heading machine generates corresponding action;
Specifically, in step S3, the safety inspection part of the operation preparation stage of the heading machine includes cable, pipeline, gas concentration, roadway support, and inspection of the heading machine components, parameters and faults are randomly set by the system, and the user needs to perform the next actual operation after the inspection is completed.
Specifically, in step S3, the operation flow of the operation phase of the heading machine includes: firstly triggering a power button, sending a test instruction to an operation system of the simulated heading machine, after the test is passed, sequentially triggering an illumination, early warning and spraying button by trained personnel to realize early-stage work, then opening a star wheel and a conveyor to realize a conveying function, utilizing a push rod function to realize left-right rotation of a rotating part, enabling the traveling part to move back and forth, setting an automatic feeding route of the simulated heading machine by utilizing a feeding route setting function, completely cutting a coal bed, and conveying falling coal mine stones out of the pit through the running star wheel and the conveyor.
Specifically, in step S3, in the feeding route setting command in the operation stage of the heading machine, the system automatically prompts the settings of the rock and soil properties and the tunnel design requirements, and gives out recommended heading speed, matched cutter head rotating speed and propulsion parameters, the user uses index finger tip nodes to perform data setting, when the feeding route nodes set by the user are too concentrated and the effective mining rate is too low (the user compares the mining depression degree of the coal seam part in the digital coal seam model with the original volume of the whole model), the operation system of the simulated heading machine automatically optimizes the route according to an a-algorithm, searches the optimal short route, and prompts the user to improve.
The algorithm A rasterizes a feed route picture set by a user, initializes an open list and a closed list, the starting point set by the user is classified into the open list, and the open list is traversed to calculate a cost function F of each node:
where G is the euclidean distance from the starting point to the current node,for two-dimensional spatial coordinates parallel to the display element, the formula is:
h is the Manhattan distance from the current node to the end point, and the formula is:
searching the node a with the minimum F in the list, taking the node a as the node to be processed currently, and repeating the following operations:
and processing all nodes adjacent to the node a to be processed currently, if the node a to be processed is not reachable or is in a closed list, ignoring the node a, otherwise, continuing processing.
If the adjacent node b is not in the open list, the adjacent node b is added into the open list, the node a to be processed currently is set as a father node, the node b is set as the node to be processed currently, the F, G, H value of the node b is calculated, if the adjacent node b is in the open list, whether the path from the starting point to the node b is better or not is checked, the reference standard is the G value, if the G value is smaller, the path effect is better, the father node e of the adjacent node b is automatically used as the node to be processed currently, and the G, F value is recalculated.
And moving the node a after the second) processing to a closed list, and paying no attention.
The procedure is terminated when any of the following conditions is satisfied.
(1) Adding an open list at the end point;
(2) The endpoint cannot be found, and the open list is empty at this time;
if the endpoint has been found, find the shortest path: starting from the end point, each node moves along the parent node to the start point. The shortest path is the feeding path set by the user.
Specifically, in step S3, during the operation flow, the operation system of the simulated heading machine records the sequence, the number of times and the time of triggering the push rod, judges the movement angle, the direction and the distance of the virtual heading machine, judges the correctness of the operation flow of the user according to the comparison with the correct steps of the database, and after the user finishes training, the system automatically scores and invokes the related database data to generate the doc file report.
S4, transmitting the operation process in the step S3 to a display unit to display the feedback effect of the user operation process and the simulation environment.
Specifically, in step S4, the development machine and the simulation environment are simulated, the development machine virtual model is an original model built by referring to a real object in UG NX10.0 software, the three-dimensional model is mapped and rendered by 3DSMAX software and is imported into Unity3D, and the movement is started and stopped by receiving a transmitted character string instruction by using TCP communication in a c# programming script.
Specifically, in step S4, surrounding boxes are set around each component of the simulation heading machine in the Unity3D software as a safe operation range, virtual personnel are randomly set around the heading machine model in consideration of the fact that multiple operators work underground, and when the fact that the movement of the component exceeds the safe range is detected, a user is timely reminded, and an emergency stop command is automatically triggered.
Specifically, in step S4, the simulation environment builds a roadway in the Unity3D in combination with the real underground environment, compresses different stone colors into a digital coal bed model by using a mapping function, sets a virtual camera in the Unity3D to shoot, distinguishes effective coal beds from invalid gangue impurities by colors, sets collision detection for a mining head of the simulation heading machine, records a collision area of the coal beds in the operation process of a user, calculates the mining rate according to the deformation degree of the grid relative to the original model, and outputs the mining rate to the display unit.
Specifically, in step S4, the simulation environment uses particles and a light source module in Unity3D to realize the smoke, water spray and light simulation effect of the heading machine during working, uses a grid deformation algorithm to control the recession degree of the terrain grid by using the cutting head as a key element of the terrain grid deformation, and after the vertices around the collision point of the cutting head are screened, refers to a deformation formula:
In the middle ofIs the diameter of the tool>Is the cutting corner->Is depth of cut,/->Is the cutting force of the cutter>The angular velocity of the cutter is changed, the grid vertex coordinates are changed, stone models with different shapes are preset, and the coal after drilling is realized by matching with the particle effectVisual simulation of the broken state of the layer.
In this embodiment, the above technical solution includes the following steps when applied specifically:
step one, entering an operation system login interface of the simulated development machine, opening a virtual keyboard by the system, displaying in a display unit, triggering a virtual key by a user through a gesture, outputting corresponding content, finishing personal account information login or registration, jumping to an operation content selection interface by a page, and closing the virtual keyboard by the system; the operation content selection interface comprises two parts, namely theoretical learning and actual operation, and a user selects subjects to enter; when the user enters theoretical learning, executing the second step; executing the third step when the user performs the operation exercise of the simulated heading machine;
step two, user theory learning comprises:
the system automatically plays the safety operation standard and legal regulations of special equipment of the development machine, the development mechanism theory knowledge and operation flow, a user watches the development machine operation teaching video, jumps into a simulation development machine model after watching, embeds operation video links into key parts of the model, the user selects the key parts of the simulation development machine model, jumps into relevant teaching video, and exits from a theoretical learning interface after learning;
Step three, simulating operation exercise of the heading machine comprises the following steps:
the initial point position of the simulated heading machine is the digging and mining position of a roadway in a virtual underground environment, and the initial position of the cutting head is the right center vertical to the virtual coal seam;
the simulation development machine model is automatically initialized to a starting point position at the beginning, a cutting head is initialized to the starting point position, a user designs and operates the development machine to carry out tunnel development work according to regulations and operations, firstly triggers a power key of an operation platform of the development machine according to design gestures, sends a test instruction to an operation system of the simulation development machine, and after the test is passed, sequentially triggers an illumination key, an early warning key and a spraying key to realize preparation work, and then opens a star wheel and a conveyor to realize a conveying function; the user triggers the virtual handle through gestures, and the functions of overall left-right steering, forward-backward movement, front support, shovel plate part and cutting head lifting are realized according to the moving distance, speed and direction of the push rod of the simulation heading machine model; finally, the user triggers the virtual cutting head to rotate the key by means of gestures, so that the cutting head rotates positively to completely cut the coal bed, and the dropped coal blocks are conveyed out of the pit through the running star wheels and the conveyor. After the operation is finished, the system automatically scores according to the operation rules of the user, calls all relevant data of the database to generate doc file reports, and the user can multiplex the disk according to the doc file reports.
The operation is operated through the simulation development machine operation system, a user interacts with the simulation development machine model in real time through the operation system, a virtual button and a handle are operated on an interface through gestures, a trigger command signal is sent to the simulation model, the trigger sequence, the times and the time of the keys of the operation system are recorded, whether the motion angle, the motion direction and the motion distance of the simulation development machine and the operation flow of the user are correct or not is judged, and the motion trail is displayed through a display module.
In summary, by means of the above technical solution of the present invention, by accurately identifying gestures by using a plurality of algorithms, the preprocessing unit optimizes the video quality by using the designed algorithm for the high brightness points existing in the video, thereby avoiding the influence of the high brightness on the subsequent operation; aiming at the problem of operation failure caused by gesture coverage of a user, a coverage unit establishes a screen mapping area through a non-maximum suppression algorithm, and the corresponding relation of an operation area in a picture is defined, so that the stability and accuracy of identification are greatly improved; the key point detection unit removes space data jitter and reduces errors by using a self-adaptive centroid-Kalman filtering algorithm, the space data of the key nodes of the hand of the user are rapidly and accurately detected and judged by using a neural network, the functions of the virtual keys and the virtual push rods are triggered according to the gestures of the user, and the robustness of the system is greatly improved by combining multiple algorithms.
In addition, the design gesture operation is convenient and rapid, the user can enter the operation conveniently, the theoretical learning part greatly shortens the explanation time of a coach in the traditional live-action operation, ensures the safety of the user, each operation item can repeatedly learn and provide an operation report for the user to review, the simulation development machine system environment furthest simulates the real excavation working condition through the Unity3D particle system and combines a grid deformation algorithm, the simulation development machine operation system judges the concentration degree of the user according to the gesture operation frequency and the eye gesture characteristic of the user, reminds the user, improves the human-computer interaction experience, improves the operation efficiency and shortens the training period.
In addition, the system can operate in a cross-platform mode depending on monocular RGB video input, the configuration environment is simple, the computer is supported to generate real-time detection by using a CPU, the frame rate on the notebook is 15-30 fps, the delay is less than 50ms, the designed virtual keys and virtual push rods replace operation hardware in traditional training, the training cost is saved, and compared with a similar frame, the system has obvious advantages in application scenes and is easy to popularize and use.
In addition, the gesture is accurately recognized by adopting a plurality of algorithms, the robustness of the system is greatly improved, the simulation algorithm is combined to simulate the real working condition to the greatest extent, the practicability is high, the human-computer interaction experience is fully considered, the operation is simple and convenient, the training efficiency is improved, the training period is shortened, the hand gesture is captured by adopting a monocular vision method, the requirement on hardware equipment is low, the system can be deployed on a plurality of platforms, and the system is easy to popularize and use.
In the present invention, the above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the present invention, but any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. The development machine control teaching auxiliary training system based on visual dynamic capture is characterized by comprising a camera unit, a display unit and a processing unit;
the camera shooting unit is used for shooting hand and eye gestures of a user in real time;
the display unit is used for displaying the operation picture of the simulated heading machine and the feedback effect of the simulation environment in real time;
the processing unit comprises a gesture recognition processing unit and a simulated heading machine operating system;
the gesture recognition processing unit judges a gesture command and a concentration degree of attention of a user by detecting key node space data of hands and eyes of the user on the basis of solving the problem of high brightness points of the video and gesture coverage of the user;
the simulation heading machine operating system is used for receiving a command to realize control and transmitting an operation and environment feedback picture to the display unit.
2. The development machine control teaching auxiliary training system based on visual dynamic capture according to claim 1, wherein the gesture recognition processing unit comprises a preprocessing unit, a covering processing unit, a key point detection unit and a communication unit;
The preprocessing unit performs high-brightness point removal processing on the collected real-time video data, sends the processed real-time video data to the covering processing unit, performs screen display area division, establishes a mapping relation between a screen area and an actual space, and sends the mapping relation to the key point detection unit, the key point detection unit detects key node space coordinate data of the hand and the eye of a user, and outputs command signals to the communication unit after judging the concentration degree of gesture commands and the user attention, and the communication unit transmits signals to the simulation development machine operation system through TCP/IP communication.
3. The development machine control teaching auxiliary training system based on visual dynamic capturing according to claim 1, wherein the preprocessing unit performs high-brightness point removal processing on the collected real-time video data, and the preprocessing unit includes:
converting a video picture from an RGB color space to an HSV color space by analyzing the highlight characteristic of real-time video data, wherein the formula of the pixel value of any point in the video picture image is as follows:
；
wherein, I represents a pixel value,respectively represent the image at +.>Tone value, saturation value, luminance value of a dot,/->Expressed as a set of all pixels in the image;
Extracting images of the highlight region of the real-time video surface frame by setting saturation and brightness threshold values, and processing the highlight region by using an image average filtering mode;
and carrying out image fusion on the processed highlight region and the non-highlight region in the original image, and outputting the fused video to a covering processing unit.
4. The development machine control teaching auxiliary training system based on visual dynamic capturing according to claim 2, wherein the masking processing unit is configured to divide a screen display area, establish a mapping relationship between the screen area and an actual space, and send the mapping relationship to the key point detection unit, and the development machine control teaching auxiliary training system comprises:
defining a hand movement area, and dividing a screen display area into a camera shooting areaGesture action zone->Palm frame option area->Screen display area->The gesture action area and the screen display area have a proportional relation m:
；
the lower left corner of the screen display area and the lower left corner of the gesture action area are taken as the origin of coordinates (0, 0), and the hand is closedSpace coordinates of key nodes relative to gesture action areaDenoted as (/ -)>) Is +_ relative to the screen display area>Denoted as (/ -)>) The length and width of the screen display area are recorded as +.>、/>In the gesture motion, the coordinate position is in the gesture motion zone +. >And Screen display area +.>There is a mapping relation between:
；
；
and establishing gesture block diagrams, sorting all the gesture block diagrams according to the confidence coefficient by using a non-maximum value suppression algorithm, selecting the gesture block diagram with the highest current confidence coefficient, starting to circularly traverse the video, adding the gesture block diagram with the highest current confidence coefficient into a final selection list, removing other gesture block diagrams with high overlap degree with the currently selected gesture block diagram, updating a set of the rest gesture block diagrams, searching the gesture block diagram with the highest confidence coefficient in the updated set, repeating the operation until all the gesture block diagrams are processed, and outputting the gesture block diagram with the highest confidence coefficient after the processing is finished to a key point detection unit.
5. The system of claim 2, wherein detecting key node spatial coordinate data of a user's hand and performing gesture determination comprises:
judging whether a user index finger fingertip node stays for 2s in ten virtual key function areas of a screen display area, wherein the index finger fingertip node has depth change exceeding a threshold value in the direction perpendicular to the camera unit, if so, triggering a corresponding virtual key function, and if not, not triggering the virtual key function;
Judging whether a user forms a similar real holding rod action through five-finger aggregation, wherein the whole hand stays for 2s on a single virtual push rod of four push rod function areas of a screen display area, and the depth of the whole hand exceeds a threshold value in the direction perpendicular to the camera unit, if so, triggering a corresponding push rod function, and if not, not triggering the push rod function;
judging whether a user combines the index finger with the middle finger, and if so, triggering a corresponding feed route setting function, wherein the space coordinate positions of the two-finger fingertip nodes are set feed starting points and end points, and connecting according to an A-type search algorithm and a sequence set by the user to obtain a virtual feed route, if not, not setting the feed starting points and the end points;
judging whether the user holds a fist with both hands, if so, triggering a scram command, and if not, not triggering the scram command;
and outputting signals corresponding to the key gesture command, the push rod gesture command, the feed gesture command and the scram gesture command to a communication unit for driving the corresponding simulated heading machine function.
6. The system of claim 5, wherein the detecting the spatial coordinate data of the key nodes of the hand of the user and performing the gesture determination further comprises:
Processing the high-frequency jitter problem in the acquisition process by using a self-adaptive centroid-Kalman filtering algorithm;
training a gesture detection model by using a convolutional neural network, and performing auxiliary judgment on the acquired gesture by using the trained gesture detection model.
7. The system of claim 6, wherein detecting the spatial coordinate data of key nodes of the hands and eyes of the user and determining the concentration of the user comprises:
counting the hand operation frequency of the user in real time, detecting eye key nodes of the user, judging the concentration degree of the user by taking the time interval of the hand operation sending command of the user as a basis, and judging that the attention is not concentrated when the average time interval of the operation command is exceeded;
acquiring space coordinate data of key nodes of two eyes of a user through an image pickup unit, and respectively calculating aspect ratios EAR of the left eye and the right eye of the user:
；
wherein EAR represents an aspect ratio,representing six node space coordinate data of outer canthus, left and right upper eyelid, left and right inner canthus and left lower eyelid in turn, +.>Represents an L2 norm;
taking the average value of the aspect ratios of the left eye and the right eye of the user as the final eye aspect ratio, judging whether the final eye aspect ratio exceeds a threshold range, if so, judging that the attention is not concentrated, if not, judging whether the attention is concentrated by combining the hand operation frequency of the user, and if the hand operation frequency of the user judges that the attention is not concentrated, judging that the attention is not concentrated.
8. The development machine control teaching auxiliary training system based on visual dynamic capture according to claim 1, wherein the simulation development machine operation system comprises a login unit, a theoretical learning unit and a development machine operation training unit;
the user selects a virtual keyboard key to log in personal account information by using a command signal sent by the communication unit, the user learns the operation theory knowledge, the operation specification and the operation course of the development machine in the theory learning unit, and after the learning theory knowledge is finished, the development machine operation training unit automatically opens a third visual angle picture of the simulation development machine, the user selects all parts of the development machine and a roadway in the simulation environment by using gestures, and the simulation development machine is controlled by using the command signal sent by the communication unit.
9. The development machine control teaching auxiliary training system based on visual dynamic capturing according to claim 1, wherein in a development machine operation stage feed route setting command, a development machine operation training unit automatically prompts rock and soil properties and tunnel design requirement setting, and gives recommended development speed, collocation cutter disc rotating speed and propulsion force parameters, a user performs data setting by using index finger tip nodes, and when a user-set feed route node is too concentrated and the effective excavation rate is too low, a simulation development machine operation system automatically optimizes a route according to an A-type route searching algorithm, searches an optimal short route and prompts the user to improve;
Wherein, automatically optimizing the path according to the a-routing algorithm to find the optimal short path comprises:
rasterizing a feed route picture set by a user, initializing and setting an open list and a closed list, setting a starting point set by the user to be in the open list, and traversing the open list to calculate a cost function F of each node:
；
；
；
where G is the Euclidean distance from the starting point to the current node, H is the Manhattan distance from the current node to the end point,two-dimensional space coordinates parallel to the display unit;
searching a node a with the minimum cost function F in the list, taking the node a as a node to be processed currently, and repeating the following operations:
firstly), processing all nodes adjacent to the node a to be processed currently, if the nodes are not reachable or in a closed list, ignoring the nodes, otherwise, continuing to process;
secondly), if the adjacent node b is not in the open list, adding the adjacent node b into the open list, setting the node a to be processed currently as a father node, setting the node b to be processed currently, calculating F, G, H value of the node b, if the adjacent node b is in the open list, checking whether the path from the starting point to the node b is better, if the G value calculated currently is smaller than the G value of the reference standard, the path effect is better, automatically taking the father node e as the node to be processed currently, and recalculating G, F value;
Thirdly), moving the node a processed by the second step to a closed list, and paying no attention to the node a;
the procedure is terminated when any one of the following conditions is satisfied;
adding the condition I and the terminal point into an open list;
condition two, can't find the terminal point, and open the list to be empty at this moment;
if the endpoint has been found, find the shortest path: starting from the end point, each node moves along the father node until the start point, and the shortest path is the optimal short path.
10. The development machine control teaching auxiliary training system based on visual dynamic capturing according to claim 1, wherein the simulation environment in the display unit utilizes particles and a light source module in Unity3D to achieve smoke, water spraying and light simulation effects when the development machine works, a grid deformation algorithm is utilized to control the sinking degree of a terrain grid by taking a cutting head as a key element of the terrain grid deformation, after vertexes around collision points of the cutting head are screened, a reference deformation formula is utilized to change grid vertex coordinates, stone models in different shapes are preset, and particle effects are matched, so that visual simulation of the crushing state of a coal bed after drilling is achieved;
wherein, the reference deformation formula is:
；
；
in (x) Variable ，y Variable ) Representing the changed coordinates, (x) Original source ，y Original source ) The coordinates before the change are indicated,for the diameter of the tool>Is the cutting corner->For depth of cut->For the cutting force of the tool->Is the angular velocity of the tool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410015789.1A CN117519487B (en) | 2024-01-05 | 2024-01-05 | Development machine control teaching auxiliary training system based on vision dynamic capture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410015789.1A CN117519487B (en) | 2024-01-05 | 2024-01-05 | Development machine control teaching auxiliary training system based on vision dynamic capture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117519487A true CN117519487A (en) | 2024-02-06 |
| CN117519487B CN117519487B (en) | 2024-03-22 |
Family
ID=89753553
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410015789.1A Active CN117519487B (en) | 2024-01-05 | 2024-01-05 | Development machine control teaching auxiliary training system based on vision dynamic capture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117519487B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100306712A1 (en) * | 2009-05-29 | 2010-12-02 | Microsoft Corporation | Gesture Coach |
| US20120110516A1 (en) * | 2010-10-28 | 2012-05-03 | Microsoft Corporation | Position aware gestures with visual feedback as input method |
| CN105536205A (en) * | 2015-12-08 | 2016-05-04 | 天津大学 | Upper limb training system based on monocular video human body action sensing |
| CN106512391A (en) * | 2016-11-09 | 2017-03-22 | 华南理工大学广州学院 | Two-hand gesture recognition method, and simulation driving system and method based on two-hand gesture recognition method |
| CN107331220A (en) * | 2017-09-01 | 2017-11-07 | 国网辽宁省电力有限公司锦州供电公司 | Transformer O＆M simulation training system and method based on augmented reality |
| US20200103980A1 (en) * | 2012-12-13 | 2020-04-02 | Eyesight Mobile Technologies Ltd. | Systems and methods for triggering actions based on touch-free gesture detection |
| DE102018008366A1 (en) * | 2018-10-23 | 2020-05-07 | INPRO Innovationsgesellschaft für fortgeschrittene Produktionssysteme in der Fahrzeugindustrie mbH | Process and system for the gesture-based control of a test process at an industrial, partially automated visual inspection station of an industrial technical process or an industrial technical production plant |
| US20210259601A1 (en) * | 2017-12-07 | 2021-08-26 | Eyefree Assisting Communication Ltd. | Communication methods and systems |
| CN113380088A (en) * | 2021-04-07 | 2021-09-10 | 上海中船船舶设计技术国家工程研究中心有限公司 | Interactive simulation training support system |
| WO2022116282A1 (en) * | 2020-12-04 | 2022-06-09 | 郑州中业科技股份有限公司 | Method and system for human-machine interaction based on dynamic gesture recognition |
| CN114840079A (en) * | 2022-04-27 | 2022-08-02 | 西南交通大学 | High-speed rail driving action simulation virtual-real interaction method based on gesture recognition |
| CN115202479A (en) * | 2022-07-15 | 2022-10-18 | 西安交通大学 | Man-machine co-driving simulation system of man-in-loop and application thereof |
-
2024
- 2024-01-05 CN CN202410015789.1A patent/CN117519487B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100306712A1 (en) * | 2009-05-29 | 2010-12-02 | Microsoft Corporation | Gesture Coach |
| US20120110516A1 (en) * | 2010-10-28 | 2012-05-03 | Microsoft Corporation | Position aware gestures with visual feedback as input method |
| US20200103980A1 (en) * | 2012-12-13 | 2020-04-02 | Eyesight Mobile Technologies Ltd. | Systems and methods for triggering actions based on touch-free gesture detection |
| CN105536205A (en) * | 2015-12-08 | 2016-05-04 | 天津大学 | Upper limb training system based on monocular video human body action sensing |
| CN106512391A (en) * | 2016-11-09 | 2017-03-22 | 华南理工大学广州学院 | Two-hand gesture recognition method, and simulation driving system and method based on two-hand gesture recognition method |
| CN107331220A (en) * | 2017-09-01 | 2017-11-07 | 国网辽宁省电力有限公司锦州供电公司 | Transformer O＆M simulation training system and method based on augmented reality |
| US20210259601A1 (en) * | 2017-12-07 | 2021-08-26 | Eyefree Assisting Communication Ltd. | Communication methods and systems |
| DE102018008366A1 (en) * | 2018-10-23 | 2020-05-07 | INPRO Innovationsgesellschaft für fortgeschrittene Produktionssysteme in der Fahrzeugindustrie mbH | Process and system for the gesture-based control of a test process at an industrial, partially automated visual inspection station of an industrial technical process or an industrial technical production plant |
| WO2022116282A1 (en) * | 2020-12-04 | 2022-06-09 | 郑州中业科技股份有限公司 | Method and system for human-machine interaction based on dynamic gesture recognition |
| CN113380088A (en) * | 2021-04-07 | 2021-09-10 | 上海中船船舶设计技术国家工程研究中心有限公司 | Interactive simulation training support system |
| CN114840079A (en) * | 2022-04-27 | 2022-08-02 | 西南交通大学 | High-speed rail driving action simulation virtual-real interaction method based on gesture recognition |
| CN115202479A (en) * | 2022-07-15 | 2022-10-18 | 西安交通大学 | Man-machine co-driving simulation system of man-in-loop and application thereof |
Non-Patent Citations (1)
| Title |
|---|
| 张秋实;王力农;方雅琪;胡建勋;: "基于手势交互的输电线路运维沉浸式操作训练平台", 电力科学与技术学报, no. 04, 28 December 2017 (2017-12-28) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117519487B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104750397B (en) | A kind of Virtual mine natural interactive method based on body-sensing | |
| US11755041B2 (en) | Objective-based control of an autonomous unmanned aerial vehicle | |
| CN110425005B (en) | Safety monitoring and early warning method for man-machine interaction behavior of belt transport personnel under mine | |
| US20130278501A1 (en) | Systems and methods of identifying a gesture using gesture data compressed by principal joint variable analysis | |
| WO2015103693A1 (en) | Systems and methods of monitoring activities at a gaming venue | |
| CN103977539B (en) | Cervical vertebra rehabilitation health care auxiliary training system | |
| WO2016025697A1 (en) | Visualization of physical interactions in augmented reality | |
| CN109999496A (en) | Control method, device and the electronic device of virtual objects | |
| CN104428732A (en) | Multimodal interaction with near-to-eye display | |
| CN110652726B (en) | Game auxiliary system based on image recognition and audio recognition | |
| CN116259002A (en) | Human body dangerous behavior analysis method based on video | |
| CN109708658A (en) | A kind of visual odometry method based on convolutional neural networks | |
| WO2023104557A1 (en) | Machine-learning for safety rule violation determination | |
| WO2024060978A1 (en) | Key point detection model training method and apparatus and virtual character driving method and apparatus | |
| CN111966217A (en) | Unmanned aerial vehicle control method and system based on gestures and eye movements | |
| CN114972911A (en) | Method and equipment for collecting and processing output data of automatic driving perception algorithm model | |
| CN107329564B (en) | Man-machine finger guessing method based on gesture intelligent perception and man-machine cooperation mechanism | |
| CN117519487B (en) | Development machine control teaching auxiliary training system based on vision dynamic capture | |
| CN114937177A (en) | Automatic marking and detection model training and target recognition method and electronic equipment | |
| KR101862545B1 (en) | Method and system for providing rescue service using robot | |
| CN106611165A (en) | Automobile window detection method and device based on correlation filtering and color matching | |
| JP2021517289A (en) | Object recognition system using distributed neural network | |
| CN117423157A (en) | Mine abnormal video action understanding method combining migration learning and regional invasion | |
| JP2021008754A (en) | Debris detection method, debris subdivision method, debris detection device and debris subdivision device | |
| CN113554682B (en) | Target tracking-based safety helmet detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |