WO2019029100A1 - Multi-interaction implementation method for mining operation based on virtual reality and augmented reality - Google Patents

Abstract

The present invention discloses a multi-interaction implementation method for a mining operation based on virtual reality and augmented reality, belonging to the technical field of virtual reality and augmented reality. The method comprises two modes: a virtual reality mode and an augmented reality mode. The following operations can be implemented in a virtual reality scene: displaying a model, selecting and replacing materials, scene exploration, random movement and placement of the model, embedding videos, QR code generation, realizing natural interactions and voice interactions by means of a trigger; The following operations can be implemented in an augmented reality scene: model selection, voice playback, demonstration of model operation dynamics and model rotation and stop control, screen capturing and functional expansions. Various manners of interaction, such as voice control, hand gesture control and control via a keyboard and a mouse cursor can be implemented in the two modes. The method can be applied to virtual simulations of applicable mine operation scenarios. The method is used to train miners in a mining area and students specializing in mining engineering, to reduce training costs and to improve workers' skills, thereby providing a fast and effective means for guiding production, construction, and technical scientific research.

Description

Multi-interaction realization method of mining operation based on virtual reality and augmented reality Technical field

The invention belongs to the field of virtual reality and augmented reality technology, and particularly relates to a multi-interaction realization method for mining operations based on virtual reality and augmented reality.

Background technique

In 2016, it was called “the first year of virtual reality” by the industry. Some people may mistakenly believe that this technology is a new technology developed in recent years. In fact, Virtual Reality (VR) technology emerged in the 1990s. After 2000, virtual reality technology introduced advanced technologies such as XML and JAVA in the integration and development, applying powerful 3D computing power and interactive technology. , to improve the rendering quality and transmission speed, entered a new era of development. Virtual reality technology is the product of economic and social productivity development and has broad application prospects. The study of virtual reality technology in China started in the early 1990s. With the rapid development of computer graphics and computer system engineering, virtual reality technology has received considerable attention. The National Advertising Research Institute and other organizations jointly released the "China VR User Behavior Research Report for the First Half of 2016", showing that in the first half of 2016, the number of potential users of domestic virtual reality reached 450 million, the number of shallow users was about 27 million, and the number of heavy users was 2.37 million. It is expected that the domestic virtual reality market will usher in explosive growth. Augmented Reality (AR) technology is an emerging technology developed on the basis of virtual reality. Its application fields are also very extensive, and it has shown good application prospects in the fields of industry, medical, military, municipal, television, games, exhibitions and so on.

At present, VR and AR technologies are constantly evolving and the scope of application is more and more extensive, but these two technologies are more used in military, entertainment and other fields. For applications in education, industry, engineering, etc., Multidisciplinary factors such as physics and geography still require more research and development. For the mining industry, the geological conditions of mines in China are complex, and most of them are underground mining. In the mining process, because the mining environment is underground, the process is quite complicated, and gas accidents such as gas and water damage occur. At the same time, mining is another industry with long construction period, large investment and high safety hazard. It is easy to cause safety accidents. Therefore, the safety training of mining employees has always been the top priority of mining work. However, the existing traditional training and teaching system is basically a theoretical introduction to mold display or two-dimensional image display, mainly in the classroom, supplemented by simple animation and audio and video introduction, lack of practice, lack of real scenes. Even if you look at the mold, you can't grasp the actual operation flow of the tool. With the continuous development of technology, various training systems applied to coal mining have been developed accordingly, but there are also problems such as poor authenticity of the system scene, poor immersive effect, and few interactive functions, which can only be simply demonstrated.

Summary of the invention

Aiming at the above technical problems existing in the prior art, the present invention proposes a multi-interaction realization method for mining operations based on virtual reality and augmented reality, which is reasonable in design, overcomes the deficiencies of the prior art, and has good effects.

In order to achieve the above object, the present invention adopts the following technical solutions:

A multi-interaction realization method for mining operation based on virtual reality and augmented reality, adopting a multi-interaction simulation system for underground mining operation, the system includes two modes: virtual reality mode and augmented reality mode; virtual reality mode includes modeling and roaming of specific scenes , model and its material replacement, video embedded virtual scene, model movement, application scene intentional interaction, two-dimensional code generation and voice interaction; augmented reality mode including model selection, model explanation, dynamic model demonstration, gesture control model interaction, screenshot generation Icon, 360 degree rotation and stop, function mode switching and function expansion; the system has two hidden menus, namely the replacement tool in the virtual reality mode, the material selection menu and the model selection menu in the augmented reality mode; the first type The user enters the specific area menu to display, leaving will hide; the second type can display the secondary menu somewhere, click the menu again to hide;

The mining operation multiple interaction implementation method specifically includes the following steps:

Step 1: Construction of the entire environmental scene of the mining operation

According to the real environment of underground mining operation, the modeling tool 3DMax is used for 1:1 equal ratio modeling to realize the environment simulation of the whole underground mining operation; the UE4 engine is used to edit the model including creation, editing texture and material, adding Physical collision, adding lighting, effect lighting and special effects to the overall environment, baking and rendering;

Step 2: Roaming of the virtual reality application scenario

In the UE4 engine, the upper, lower, left, and right keys of the keyboard are set, and the Up, Down, Right, and Left direction control functions are bound, and the Turnaround control function is bound to the mouse to realize the roaming of the virtual reality scene of the entire underground mining operation;

Step 3: Replace the tool model of the underground mining operation and the simulated material of the mine geology

Add hidden menus in the virtual underground mining scene. When roaming to the mining site, the model or material selection menu will appear automatically. Users can select models or materials to replace them from the menu according to their needs.

Step 4: Embed the video material into the 3D application scene and control playback and stop

Embed the video material into the virtual reality scene, play it in the three-dimensional space, simulate the monitoring display device of the mining environment, set the keyboard X key, bind the MediaPlayer media class of the UE4 platform, and control the playing and stopping of the video through the OpenSource and Close functions;

Step 5: Select the model and move to any location

Select the model by mouse and move the model to any position where the simulation operation is needed to achieve the mechanical movement simulation in the real scene;

Step 6: Implement the intent interaction of the application scenario

When the user roams to a specific location in the virtual reality application scenario, and the system detects that the user intentionally enters, the environment light is automatically turned on to implement natural interaction in the virtual scenario;

Step 7: QR code generation

Bind the F key of the keyboard, add the QR code generation function, set the keyboard key control to generate the QR code function, and press the F key of the keyboard to generate a two-dimensional code of the virtual scene panorama with the set sampling points;

Step 8: Implement voice interaction

The user controls the coal mining machine in the virtual reality scene through keywords including forward rotation, reverse rotation, boom raising, lowering arm and stopping, and simulates the operation effect;

Step 9: AR dynamic demonstration function mode switching

The user clicks the AR mode button in the upper right corner of the system to switch to the AR presentation mode.

Preferably, in step 3, the model is instantiated into a specific Actor, the SetMesh function and the SetMaterial function are added to replace the model and the model material, and the Widget Blueprint user interface and the Box collision collision detection are set to realize the hidden menu function of the three-dimensional space.

Preferably, in step 5, a mouse event is added for the model to be operated, the model is selected by the GetHitResult function, and then the coordinates of the SetActorLocation function of the model are changed according to the coordinates of the mouse in the three-dimensional space, and when the mouse clicks again, the time will be The coordinate values of the mouse x, y, and z directions are assigned to the model, and the GetHitResult function sets the model to the unselected mode.

Preferably, in step 6, the TriggerBox trigger is set. When the first person role triggers the TriggerBox, and the system detects that the user intentionally enters an area, a device in the area is automatically enabled.

Preferably, in step 7, the user presses the keyboard F key, and the system generates a virtual scene panorama QR code with the set sampling point, and the user scans the two-dimensional code with the mobile phone, and jumps to the virtual application scene display page of the mobile terminal, On the mobile phone side, the user can enable the gyroscope, switch to the VR split screen mode, set the mobile phone parameters, and then use the VR glasses to experience the virtual underground mining operation environment scene, realize the 720 degree view display, and also realize the multi-scene and multi-angle of the mobile phone end. Roaming experience.

Preferably, in step 8, the speech recognition is implemented based on the Pocket-sphinx library, and the recognition function is implemented through the improved Chinese keyword dictionary, through preprocessing, feature extraction, acoustic model training, language model training, and speech decoding and searching, and finally passes through the UE4. The writing function control function of the engine realizes the control of the speech model in the three-dimensional space; the specific implementation steps of the speech recognition are as follows:

Step 8.1: Pretreatment

Processing the input original speech signal, filtering out unimportant information and background noise, and processing end point detection, speech framing and pre-emphasis of the speech signal;

Pre-emphasis is achieved by a first-order FIR high-pass digital filter. The transfer function of the first-order FIR high-pass digital filter is:

H(z)=1-az ^-1 ;

Where a is the coefficient of the pre-emphasis filter, and the value ranges from 0.9 to 1.0. If the speech sample value at time n is x(n), the pre-emphasized signal is

y(n)=x(n)-a*x(n-1);

Step 8.2: Feature Extraction

Feature extraction is performed by the method of Mel Frequency Cepstral Coefficient (MFCC); the following steps are performed as follows:

Step 8.2.1: using the critical band effect of human hearing, using MEL cepstrum analysis technique to obtain a MEL cepstral coefficient vector sequence for speech signal processing;

Step 8.2.2: use the MEL cepstral coefficient vector sequence to represent the spectrum of the input speech, and set a number of bandpass filters with triangular or sinusoidal filtering characteristics in the speech spectrum range;

Step 8.2.3: Find the output data of each band pass filter through the band pass filter bank;

Step 8.2.4: Logarithmically output data of each band pass filter and perform discrete cosine transform (DCT);

Step 8.2.5: Obtain the MFCC coefficient; the solution formula is as follows:

Where C _i is a characteristic parameter, k is the number of triangular filters, F(k) is the output data of each filter, P is the filter order, and i is the data length;

Step 8.3: Acoustic Model Training

The acoustic model parameters are trained according to the characteristic parameters of the training speech library;

In the identification, the characteristic parameters of the speech to be recognized can be matched with the acoustic model to obtain the recognition result. In this paper, the mixed Gaussian model-Hidden Markov Model (GMM-HMM) is used as the acoustic model, which includes the following steps:

Step 8.3.1: Find the joint probability density function of the mixed Gaussian model:

Where M is the number of Gaussian in the mixed Gaussian model, C _m is the weight, u _m is the mean, ∑ _m is the covariance matrix, D is the observation vector dimension, and the maximum expected value algorithm (EM) is used to mix the Gaussian model parameter variables. Θ={C _m ,u _m ,∑ _m } is estimated and solved by the following formula:

Where j is the number of current iterations, N is the number of elements in the training data set, x ^(t) is the feature vector at time t, h _m (t) is the posterior probability of time C _m ; GMM parameters are performed by EM algorithm It is estimated that it is possible to maximize the probability of generating a speech observation feature on the training data;

Step 8.3.2: Solve the three main components of the HMM

Let the state sequence be q ₁ , q ₂ ,...,q _N , and let the transition probability matrix A=|a _ij |i,j∈[1,N], then the jump probability between the Markov chain states is :a _ij =P(q _t =j|q _t-1 =i); the initial probability of Markov chain π=[π _i ]i∈[1,N], where π _i =P(q ₁ = i); Let the observed probability distribution n _i (o _t )=P(o _t |q _t =i) of each state, use the GMM model to describe the observed probability distribution of the state; according to step 8.3.1, the formula is:

Where N is the number of states, i and j are states, a _ij is the probability of jumping from the i state to the t state at time t-1, o _t is the observed value at time t, and C _i,m is the mixing coefficient , indicating the weight between different Gaussians, u _i,m represents the mean between different Gaussians, ∑ _i,m represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by Baum-Welch algorithm, and finally generated Acoustic model file;

Step 8.4: Language Model Training

The N-Gram model is used to implement the training of the language model; the probability of the occurrence of the i-th word in a sentence depends on the N-1 words in front of it, that is, the context of a word is defined as the N-1 appearing in front of the word. Word whose expression is:

Replace the above expression with the following formula using the conditional probability formula S:

P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w ₁ ,w ₂ ,...,w _n-1 )

Where, P (w ₁₎ is the probability w ₁ appears in the _{article, P (w 1, w 2} ) is w _1, w ₂ consecutive probability _{of, P (w 2 | w 1} ) is already known w ₁ The probability of occurrence of w ₂ occurs in the case of occurrence, assuming that the probability of identifying the sentence is represented by P(s), and P(s)=P(w ₁ , w ₂ , . . . , w _n ) represents the word set w ₁ , w ₂ ,... , the probability that w _n appears continuously and generates S;

Streamlined to the following formula by the Markov assumption:

P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w _n-1 )

Where P(w _i |w _i-1 )=P(w _i-1 ,w _i )/P(w _i ), P(w _i-1 ,w _i ) and P(w _i ) can all be learned from the corpus Statistics, finally, P (sentence), the language model stores the probability statistics of P(w _i-1 , w _i ), and the entire recognition process is realized by finding the maximum value of P (sentence);

Step 8.5: Speech decoding and search algorithm

For the input speech signal, an identification network is established according to the trained acoustic model, the language model and the dictionary mapping file created by the g2p tool, and the best path is found in the network according to the search algorithm. The word string of the speech signal is output with the maximum probability, so that the text contained in the speech sample is determined. The Viterbi algorithm is used to implement speech decoding. The specific process is as follows:

Step 8.5.1: Enter the parameters of the HMM model and the observation sequence O={o ₁ ,o ₂ ,...,o _T }, then all state probabilities when t=1 are:

δ ₁ (i)=π _i b _i (o ₁ )

ψ ₁ (i)=0

Step 8.5.2: Gradually recursive to t=2,3,...,T, then:

Step 8.5.3: Terminate traversal:

Step 8.5.4: Backtracking the optimal path, t=T-1, T-2,...,1;

Step 8.5.5: Output the optimal hidden state path

Where t _t (i) is the joint probability of all nodes through which the optimal path passes when recursively to t, ψ _t (i) is the implicit state at time t, T is time, and P ^* is the probability of the optimal path ,

The endpoint for the optimal path.

Preferably, a takes 0.97.

Preferably, in step 9, specifically, the following steps are included:

Step 9.1: Model Selection

Select the shearer model, the roadheader model, the wind coal drill model and the fully mechanized support model. Each type of model is a 1:1 modeling simulation of the real coal mining tool;

Step 9.2: Model explanation

After the user selects the model, and then selects the tool model option to be learned through this menu, the system will play the corresponding voice explanation, and click the button voice again to stop;

Step 9.3: Model Demo

Import the tool simulation running animation created in the 3DMax modeling process into the Unreal Engine engine, set the corresponding selection menu, and click to demonstrate the running state of the corresponding coal mining tool in AR mode;

Step 9.4: Screenshot generation icon

In the main menu of AR mode, add a button, bind the camera's screenshot function, add a scrolling menu bar on the right side of the menu, when the screenshot function is successfully triggered, the screenshot is displayed to the right by scrolling the menu bar by setting the dynamic material conversion function. During the demonstration, the user clicks the screenshot button, and the system generates an icon on the interface side;

Step 9.5: Rotate

Instantiate the set model as an Actor, add the Rotation function, and implement the model to rotate clockwise;

Step 9.6: Function Extension

Add a secondary UI, control Map switching, and implement running demo functions including Earth, Saturn, Mercury, Atmospheric Planet, and Galaxy; add WidgetBlueprint encoding to display or hide the knowledge panel; design return button to return to AR editing Main module

Step 9.7: The dynamic gesture control model, the real environment is superimposed with the virtual model, and the gesture and the model are interactively controlled, including the following steps:

Step 9.7.1: Initialize video capture, read the logo file and camera camera parameters;

Step 9.7.2: Grab the video frame image;

Step 9.7.3: performing the detection mark and identifying the mark template in the video frame, and performing motion detection on the acquired video frame image by using the OpenCV library function to determine whether the motion track is detected;

If the result of the judgment is that the motion track is detected, proceed to step 9.7.4:

Or the result of the judgment is that no motion track is detected, then the detection mark is continued and the mark template in the video frame is identified, and then step 9.7.12 is performed;

Perform motion detection based on the color histogram and the background difference, and perform background update on the acquired frame and the pixels outside the motion gesture area after detecting each frame motion, and the formula is as follows;

Where u _t is the corresponding pixel point of the background image, u _t+1 is the updated background image pixel point; I _t is the pixel point of the current frame image, and I _f is the mask value of the current frame image pixel point, that is, whether Do background update; a∈[0,1] is the background image model update speed;

Step 9.7.4: Perform preprocessing including denoising on the image;

Through the motion detection step, if motion information is detected, the video frame image containing the motion gesture is preprocessed: median filtering is performed on the image by the medianBlur function of OpenCV to remove the salt and pepper noise;

Step 9.7.5: Convert to HSV space;

The image is color-space converted by the cvtColor function to obtain the data of its HSV space, and the brightness v value in the HSV space is reset as shown below:

Where r, g are red and green pixels of the skin color region, and r>g;

Step 9.7.6: Split the hand area;

Step 9.7.7: Perform morphological processing to remove impurities;

Combining the obtained motion binary image with the binary image obtained by back projection, and performing image morphology closing operation to obtain a relatively complete motion skin color gesture binary image; and removing the noise points in the image;

Step 9.7.8: Obtain the hand contour;

After preliminary morphological operations, removing noise and making the boundary of the hand clearer, the gesture contour is obtained by OpenCV's findContours function, and then the pseudo contour is removed.

Step 9.7.9: Draw the outline of the hand and calibrate the information;

Step 9.7.10: Comparison of contour information, setting direction vector;

Compare the contours obtained in each frame, set the comparison conditions, and assign values to the direction marker variables by comparison;

Step 9.7.11: Perform force simulation on the model according to vector coordinates to realize interaction between dynamic gestures and virtual models;

After the dynamic gesture is judged by the contour, the virtual model is subjected to the force simulation operation according to different judgment results. According to the value of the direction marker in the contour judgment process, the coordinate values of the model in the three-dimensional space will be three axes of x, y and z. The multiplication calculation on the above, through the change of the coordinate value, the change of the model position is achieved to achieve the simulation of the force;

Step 9.7.12: Calculate a conversion matrix of the camera relative to the detected mark;

Step 9.7.13: Superimpose the virtual object on the detected mark and return to step 9.7.2 to realize the superimposed display of the real environment and the virtual model.

The beneficial technical effects brought by the invention:

(1) The three-dimensional model of the present invention is established in equal proportions, and the texture map is closely related to the reality through the editing of the UE4 engine platform, and the ambient lighting of the application scene is simulated by the real lighting simulation. The entire virtual reality scene is more realistic and immersive.

(2) The present invention realizes multiple functional interactions through technical solutions, such as changing tool models through hidden menus during the process of virtual underground mining scene roaming, replacing mining materials to simulate different mining geology, freely moving the location of mining tools, and The video information is embedded in the machine display to show the real scene, and the voice function is used to control the forward, reverse, lift, lowering, and stopping of the shearer.

(3) The invention also generates a two-dimensional code function, and connects the PC end display to the mobile phone end display, and the mobile phone end function can utilize the built-in gyroscope of the mobile phone to generate gravity sensing, and can be utilized simply when set to the VR glasses mode. VR glasses experience real-time scene immersion.

(4) The invention also realizes the AR dynamic demonstration function by using the AR development SDK-ARToolKit. Through the AR editing demonstration function, the user can select the mining tool model in real time, perform 360 rotation display, voice explanation and dynamic running display, save the screenshot, etc. What is important is that it displays the tool model in the AR mode, and the virtual model is combined with the real environment. This not only shows the intuitive stereoscopicity of the model, but also shows its authenticity, making it better for learning and education. effect.

(5) The AR module of the present invention, in addition to its dynamic presentation function, adds processing to the video stream. When the dynamic gesture enters the camera perspective, it generates interaction with the model, and the dynamics of the hand from the far to the near are transmitted to the model. A three-dimensional space forwards an analog force. The top-to-bottom dynamics give the model an upward simulated force. The forward flipping of the hand's dynamics gives the model a downward simulation. Similarly, if the hand is twisted or left or right Tilting gives the model a simulated force with vector direction.

(6) The present invention expands the display function of AR in the field of astronomy in addition to the function realization of the AR module in the coal mine application scenario. The addition of Earth, Saturn, Mercury, the planet with a dynamic atmosphere, and the AR display of the galaxy, while adding the knowledge panel display function to the module of the AR display, enriches the application of AR in the field of educational display.

DRAWINGS

1 is a diagram showing the overall functional structure of an implementation of the present invention.

2 is a schematic diagram of the function of generating a two-dimensional code of the present invention.

3 is a schematic diagram of the interactive function of the speech recognition of the present invention.

4 is a schematic diagram of an AR mode implementation of the present invention.

Figure 5 is a flow chart showing the implementation of the dynamic gesture interaction function of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

The invention provides a multi-interaction realization method for mining operation based on virtual reality and augmented reality. The entire technical function encompassed by the present invention can be understood in conjunction with FIG. The specific implementation steps are as follows:

Step 1: The entire environmental scene of the underground mining operation is built. Create models based on real mining operations using 3DMax modeling tools. The model classification is imported into the UE4 engine. Through the UE4 platform, the model is written, the natural light and the ambient light are simulated, the physical collision detection is added, the parameters of the system are adjusted, and the baking is rendered.

Step 2: Add a first person role in the virtual application scenario and add a mouse and keyboard control event to the character. Bind the up, down, left and right keys of the keyboard to the Up, Down, Right, and Left functions to control the coordinates of the first person character in the virtual three-dimensional space to achieve roaming. Add a Turnaround function to the mouse to control the 720-degree rotation of the first person perspective in the virtual three-dimensional space.

Step 3: Set the interactive menu to realize the function interaction of replacing the tool model of the underground mining operation and the geological material of the mine. First create a Widget Blueprint user interface, set menu options, and add click events to the options. Then add a Box collision collision detection area to the model when the character enters the Box collision collision detection area. The created Widget Blueprint user interface is displayed. Leave the Box collision collision detection area and the Widget Blueprint user interface is hidden. Instantiate the shearer model into an Actor and add the SetMesh function to replace the other tool models. In the same way, the SetMaterial function is added to the mine geological model in the three-dimensional space to realize the replacement of the material. The invention sets four types of mining tool models for the user to select, and sets the mine geology into a material selectable mode, and replaces the model and the material through the displayed style menu. After the replacement is completed, leaving the detection area, the menu is automatically hidden, which does not affect the overall roaming visual effect, and can achieve the function of real-time interaction.

Step 4: Video embedding, playing in three-dimensional space, simulating the monitoring display equipment of the mining environment. The invention sets the keyboard X key binding to the MediaPlayer media class of the UE4 platform, and controls the playing and stopping of the video stream through the Open-Source and Close functions. This operation can simulate the screen display of the underground mining control equipment and the real-time environmental monitoring screen display, highlighting the authenticity and dynamics of the three-dimensional scene, making the simulated virtual scene closer to reality.

Step 5: Select the model to drag to any location that the user wants to place, and to implement the intent interaction function that the device automatically opens. Add a mouse event to the model to be operated, select the model by the GetHitResult function, and then change the coordinate value of the model's SetActorLocation function according to the coordinates of the mouse in the three-dimensional space. When the mouse clicks again, the coordinate values of the three directions of the mouse x, y, and z are assigned to the model at this time, and the GetHitResult function sets the model to the unselected mode. In this embodiment, the user can click on the shearer model in the scene and place it in other mining locations of the mining operation scenario.

The system adds a TriggerBox trigger to a specific area. The first person character enters this area, triggers the TriggerBox trigger, and the corresponding area's ambient light control function SetVisible triggers, and the light is turned on, thereby realizing the automatic induction lamp function set by the present invention. This is also the function of the present invention designed to detect human intent, thereby enabling more natural system interaction.

Step 6: Two-dimensional code generation function. A single PC-side display cannot satisfy the multi-user experience. The present invention can generate a multi-user mobile phone terminal by adding a two-dimensional code and scanning the two-dimensional code, and the mobile phone jumps to the panoramic display of the coal mining operation through the two-dimensional code connection. page. On the mobile phone side, the user can enable the gyroscope, switch to the VR split screen mode, set the mobile phone parameters, and then use the VR glasses to experience the virtual underground coal mining environment to achieve a 720-degree viewing angle display. At the same time, a multi-scene, multi-angle roaming experience on the mobile terminal can be realized. This function mainly adds the QR code generation and hiding function by binding the F and V keys of the keyboard. Add 6 point collection points in the UE4 engine, generate a panorama by collecting the position of the point, and then generate information and related mobile phone settings to generate a network connection in the form of a two-dimensional code to realize end-to-end conversion. The process implemented by this function is shown in Figure 2.

Step 7: Implement voice control. The invention realizes Chinese keyword recognition by using Pocket-sphinx. The specific voice control implementation principle flow is shown in FIG. 3. The present invention adds a speech recognition function to an actor created by a shearer model, by enabling a speech recognition class after system initialization, and saving a reference to such a class. Then create and bind a method to the speech recognition function OnWordSpoken. Whenever the user says the set control words, this method will be triggered to realize the forward, reverse, lift, and descend of the shearer through keyword matching. Related controls such as arm and stop. The speech recognition realized by this method is realized based on the improved speech recognition system Sphinx developed by Carnegie Mellon University. The speech recognition method of the present invention is an isolated word recognition method for a large number of vocabulary, non-specific people, and continuous Chinese syllables. It is a good way to identify the set words from different people. Finally, through the coding technology of UE4, the triggering of the action control function corresponding to the matching words after the speech vocabulary recognition is realized, and the corresponding action control of the model is realized. This recognition system includes five parts: speech preprocessing, feature extraction, acoustic model training, language model training and speech decoding. The following is the specific process of speech recognition:

Step 7.1: Pretreatment.

The input original speech signal is processed, the unimportant information and background noise are filtered out, and the end point detection, speech framing, pre-emphasis and the like of the speech signal are performed. The pre-emphasis of the speech signal is intended to emphasize the high-frequency part of the speech, remove the influence of the lip radiation, and increase the high-frequency resolution of the speech. Pre-emphasis is generally realized by a transfer function of H(z)=1-az ^-1 first-order FIR high-pass digital filter, a is the coefficient of the pre-emphasis filter, and the value range is generally 0.9-1.0, which is 0.97. Let the value of the speech sample at time n be x(n), and the signal after pre-emphasis is

y(n)=x(n)-a*x(n-1)

Step 7.2: Feature extraction.

This paper uses the method of Mel Frequency Cepstral Coefficient (MFCC) to extract. The MFCC parameter is based on the human auditory characteristics. He uses the critical band effect of human hearing, uses the MEL cepstrum analysis technique to process the speech signal to obtain the MEL cepstral coefficient vector sequence, and uses the MEL cepstral coefficient to represent the spectrum of the input speech. Set a number of bandpass filters with triangular or sinusoidal filtering characteristics in the speech spectrum range, then pass the speech energy spectrum through the filter bank, find the output of each filter, take the logarithm of it, and do the discrete cosine transform ( DCT), you can get the MFCC coefficient. The solution formula is as follows:

Where C _i is a characteristic parameter, k is the number of triangular filters, F(k) is the output data of each filter, P is the filter order, and i is the data length.

Step 7.3: Acoustic model training.

The acoustic model parameters are trained according to the characteristic parameters of the training speech library. At the time of identification, the feature parameters of the speech to be recognized can be matched with the acoustic model to obtain a recognition result. In this paper, a mixed Gaussian model-Hidden Markov Model (GMM-HMM) is used as the acoustic model.

Step 7.3.1: Find the joint probability density function of the mixed Gaussian model:

Where M represents the number of Gaussian in the mixed Gaussian model, C _m represents the weight, u _m represents the mean, ∑ _m represents the covariance matrix, and D is the observed vector dimension. The maximum expected value algorithm (EM) is used to estimate the mixed Gaussian model parameter variables: Θ={C _m , u _m , ∑ _m }, which are solved by the following formula:

Where j is the current iteration number, N is the number of elements in the training data set, x ^(t) is the feature vector at time t, and h _m (t) is the posterior probability at time t _m . The GMM parameters are estimated by the EM algorithm, which maximizes the probability of generating speech observation features on the training data.

Step 7.3.2: Solve the three main components of the HMM.

Let the state sequence be q ₁ , q ₂ ,...,q _N , and let the transition probability matrix A=[a _ij ]i,j∈[1,N], then the jump probability between the Markov chain states is :a _ij =P(q _t =j|q _t-1 =i); the initial probability of Markov chain π=[π _i ]i∈[i,N], where π _i =P(q ₁ = i); Let the observed probability distribution b _i (o _t )=P(o _t |q _t =i) of each state, use the GMM model to describe the observed probability distribution of the state; according to step 7.3.1, the formula is:

Where N is the number of states, i and j are states, a _ij is the probability of jumping from the i state to the t state at time t-1, o _t is the observed value at time t, and C _i,m is the mixing coefficient , indicating the weight between different Gaussians, u _i,m represents the mean between different Gaussians, ∑ _i,m represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by Baum-Welch algorithm, and finally generated Acoustic model file;

Step 7.4: Language Model Training.

The language model is used to constrain word search. Language modeling can effectively combine the knowledge of Chinese grammar and semantics, and describe the intrinsic relationship between words, thereby improving the recognition rate and reducing the search range. This paper uses the N-Gram model to implement the training of the language model. The probability that the i-th word appears in a statement depends on the N-1 words in front of it. The context of a word is defined as the N-1 words appearing in front of the word. The expression is:

In this paper, N=2 and N=3 are taken, that is, the probability P(w ₂ |w ₁ ), P(w ₃ |w ₂ , w ₁ ) of the current word appearing is determined by the previous word or two words.

To put it simply, the language model is the model obtained from the statistical corpus. The corpus is the text library used for training. The dictionary file stores the corpus for training and the corresponding speech. The language model is the combined probability of the expressed corpus. The set P (w ₁₎ w ₁ is the probability of appearing in the _{article, P (w 1, w 2} ) are w _1, w ₂ consecutive _{probability, P (w 2 | w 1} ) w ₁ are already known The probability of occurrence of w ₂ occurs in the case of occurrence, assuming that the probability of identifying the sentence is represented by P(s), and P(s)=P(w ₁ , w ₂ , . . . , w _n ) represents the word set w ₁ , w ₂ ,... , w _n appears continuously and generates the probability of S, using the conditional probability formula S to replace the entire formula with:

P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w ₁ ,w ₂ ,...,w _n-1 )

Then use the Markov assumption to streamline:

P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w _n-1 )

We know that P(w _i |w _i-1 )=P(w _i-1 ,w _i )/P(w _i ), P(w _i-1 ,w _i ) and P(w _i ) can all be The corpus is counted and the P (sentence) is finally obtained. The language model stores the probability statistics of P(w _i-1 , w _i ), and the entire recognition process is realized by finding the maximum value of P(sentence).

Step 7.5: Speech decoding and search algorithm.

For the input speech signal, an identification network is established according to the trained acoustic model, language model and dictionary, and the best path is found in the network according to the search algorithm, and the path is capable of outputting the speech signal with maximum probability. The string of words, thus determining the text contained in this speech sample. In this paper, the Viterbi algorithm is used to decode the speech. The specific process is as follows:

(1) Enter the parameters of the HMM model and the observation sequence O={o ₁ ,o ₂ ,...,o _T }, then all state probabilities when t=1 are:

δ ₁ (i)=π _i b _i (o ₁ )

ψ ₁ (i)=0

(2) Gradually recursive to t=2,3,...,T, then:

(3) Terminate traversal:

(4) Backtracking optimal path, t=T-1, T-2,...,1;

Output optimal hidden state path

Where δ _t (i) is the joint probability of all nodes through which the optimal path passes when recursively to t, ψ _t (i) is the implicit state at time t, T is time, and P ^* is the probability of the optimal path ,

The endpoint for the optimal path. Finally, speech recognition is achieved through the optimal path.

After the user speaks the boom, the down arm, the forward rotation, the reverse rotation, and the stop voice, the simulation system realizes the corresponding operation of the shearer, and the system recognizes the keyword spoken by the user and displays it in the upper left corner of the interface.

Step 8: AR dynamic demonstration function mode switching.

Set a widget blueprint on the interface, add the openLevel function, and switch to the new Map, which is AR mode. Enter the AR demonstration mode, which realizes the demonstration of the tool model in the coal mining process, thus realizing the learning and educational application functions of the AR technology.

Step 9: Model selection, model explanation, and dynamic presentation in AR mode.

The AR dynamic demonstration module of the present invention, the user interface is designed to be more concise and convenient for AR display, and the second-level implicit menu is designed. In this embodiment, the additional sub-function selections of model selection, model explanation, model demonstration and function expansion are designed to be hidden. The second-level menu, model selection is divided into coal mining machine, roadheader, wind coal drill, fully mechanized mining bracket and other models. After the user selects, the submenu can be hidden, and the model explanation, model dynamic demonstration and function expansion menu are also realized. The specific implementation includes content can refer to FIG. This paper takes NFT (Natural Feature Tracking) as an example to implement AR technology. The principle is shown in Figure 4. The specific process is as follows:

Step 9.1: Through the camera calibration calibration, the distortion parameter caused by the deviation of the manufacturing process of the camera, that is, the intrinsic matrix of the camera, is obtained to restore the one-to-one correspondence of the 3D space of the camera model to the 2D space.

Step 9.2: According to the hardware parameters of the camera itself, we can calculate the corresponding Projection Matrix.

Step 9.3: Feature extraction of the natural image to be recognized, and a set of feature points {p} is obtained.

Step 9.4: Feature extraction of the image acquired by the camera in real time is also a set of feature points {q}.

Step 9.5: Using the ICP (Iterative Closest Point) algorithm to iteratively solve the R, T matrix (Rotation & Translation) of the two sets of feature points, that is, the Pose matrix, which is the model view matrix commonly referred to in graphics. Suppose the two points in the three-dimensional space are:

Their Euclidean distance is:

For the matrix R and T of p and q changes, for

Where i, j = 1, 2, ..., N, the least squares method is used to find the optimal solution. Make:

The lowest R and T, at this time R, T is the MVP matrix. Where E is the distance sum of the corresponding points in the two points after the transformation, and N is the number of points in the point concentration.

Step 9.6: Obtain the MVP matrix (Model View Projection) for 3D graphics rendering.

Step 10: Screenshot generation icon.

In the main menu of AR mode, add a button, bind the camera's screenshot function, add a scrolling menu bar on the right side of the menu, when the screenshot function is successfully triggered, the screenshot is displayed to the right by scrolling the menu bar by setting the dynamic material conversion function. . During the demonstration process, the user clicks the screenshot button, and the system generates an icon on the left side of the interface, which is convenient for the user to record the difficult points, question points and detailed observations during the learning process, which can enhance the learning effect.

Step 11: Model rotation stops showing.

In AR mode, the user sees the superposition of the real scene and the virtual model. Instantiate the set model into an Actor, add the Rotation function, and implement the model to rotate clockwise. This design, set the model rotation, the user has a 360-degree observation and learning of the tool model, which can better achieve the visual effect. This demonstration learning mode is more authentic and immersive.

Step 12: AR function expansion module.

The invention adds an AR education display extension function, and controls the Map switching by adding a secondary UI to realize demonstration of different objects. These include the Earth, Saturn, Mercury, the Earth's planet and the galaxy's running display function. The planet does the rotation. Through the AR mode, the moving planet is displayed in front of the user, and the knowledge introduction function is added to improve the educational display function of the system. .

Step 13: Dynamic gestures interact with the model.

The AR mode adds OpenCV video information processing. After initializing the video stream, motion detection is performed first. If dynamic hand motion is detected, image processing is performed, and the gesture is subjected to graphics processing, denoising, converting to HSV mode, morphological processing, and drawing outline. The calibration information and the contour information are compared. Finally, the model force simulation is carried out to realize the interaction between the dynamic gesture and the virtual model. The specific implementation principle flow is shown in FIG. 5 . In particular, the dynamic gesture interaction realizes the recognition control of the simulated three-dimensional gesture, and the dynamic hand acquired by the video stream is two-dimensional information, and the matrix calculation is used to compare the calculated camera with the detected transformation matrix of the detected identifier. Obtain a three-dimensional motion gesture motion information, thereby realizing the force simulation of the model in different directions in the three-dimensional space; the specific steps include the following steps:

Step 13.1: Motion Detection

The method is based on the motion detection of the color histogram and the background difference. The program needs a certain time in the process of starting the camera. This time, it is possible to collect images of 20 frames, and the cyclic background of the 20 frames is updated as follows, and each frame is After the motion detection, pixels other than the motion gesture area are also updated as background.

Where u _t is the corresponding pixel point of the background image, u _t+1 is the updated background image pixel point; I _t is the pixel point of the current frame image, and I _f is the mask value of the current frame image pixel point, that is, whether Do background update; a∈[0,1] is the background image model update speed, generally 0.8 to 1, and 0.8 for this method.

Step 13.2: Image Preprocessing

Through the simple motion detection step of step 13.1, if motion information is detected, the video frame image containing the motion gesture is preprocessed: median filtering is performed by the medianBlur function of OpenCV to remove the salt and pepper noise:

Step 13.3: Convert to HSV Space

The color space is converted by the cvtColor function to obtain the data of its HSV space, and the brightness v value in the HSV space is reset to a relatively small brightness value (reducing the interference of static skin color); the brightness in the HSV space is v The value is reset as shown below:

Where r, g are red and green pixels of the skin color region of interest, and r>g;

Step 13.4: Divide the hand area and perform morphological processing

Combining the obtained motion binary image with the binary image obtained by back projection, and performing some image morphology closing operation to obtain a relatively complete motion skin color gesture binary image; removing the noise points in the image;

Step 13.5: Get the gesture outline

After preliminary morphological operations, removing noise and making the boundary of the hand clearer, the gesture contour is obtained by OpenCV's findContours function, and then the pseudo contour is removed.

Step 13.6: Draw outlines, calibration information

Step 13.7: Contour information comparison, set direction vector

Since the hand is constantly moving, the contours we get are constantly changing. The contours obtained for each frame are compared and the comparison conditions are set. Assign values to the direction marker variables by comparison. State comparison and analysis are shown in Table 1:

Table 1: Status Analysis

Step 13.8: Applying a force simulation to the virtual model through the direction vector

After the dynamic gesture is judged by the contour, the virtual model is subjected to the force simulation operation according to different judgment results. According to the value of the direction marker in the contour judgment process, the coordinate values of the model in the three-dimensional space will be multiplied by the x, y, and z coordinate axes, and the change of the coordinate value will realize the change of the model position to achieve the force. Simulation.

In this embodiment, a set of palms are selected from the far-to-near motion, the bottom-up motion, and the palms are twisted in various directions to simulate different force effects generated by the model, and the gesture motion models are respectively moved forward, moved upward, and according to The different twist directions of the hand have a running effect on the forces in all directions. This feature demonstrates the interaction of dynamic gestures with virtual models. This interaction helps users view the model from multiple angles and realizes the interaction between teaching and users, increasing the fun.

The above description is not intended to limit the present invention, and the present invention is not limited to the above examples, and variations, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also belong to the present invention. The scope of protection of the invention.

Claims (8)

1. A multi-interaction realization method for mining operations based on virtual reality and augmented reality, characterized in that: a multi-interaction simulation system is operated by underground mining operation, and the system comprises two modes: a virtual reality mode and an augmented reality mode; the virtual reality mode includes a specific scene. Modeling, roaming, replacement of models and their materials, video embedded virtual scenes, model movement, application scene intentional interaction, two-dimensional code generation and voice interaction; augmented reality mode including model selection, model explanation, dynamic model demonstration, gesture control model Interactive, screenshot generation icons, 360-degree rotation and stop, function mode switching, and function expansion; the system has designed two hidden menus, namely the replacement tool in the virtual reality mode, the material selection menu, and the model selection menu in the augmented reality mode. The first type of user enters the specific area menu will be displayed, leaving will be hidden; the second type of click can display the second level menu somewhere, click the menu again to hide;

   The mining operation multiple interaction implementation method specifically includes the following steps:

   Step 1: Construction of the entire environmental scene of the mining operation

   According to the real environment of underground mining operation, the modeling tool 3DMax is used for 1:1 equal ratio modeling to realize the environment simulation of the whole underground mining operation; the UE4 engine is used to edit the model including creation, editing texture and material, and add Physical collision, adding lighting, effect lighting and special effects to the overall environment, baking and rendering;

   Step 2: Roaming of the virtual reality application scenario

   In the UE4 engine, the upper, lower, left, and right keys of the keyboard are set, and the Up, Down, Right, and Left direction control functions are bound, and the Turnaround control function is bound to the mouse to realize the roaming of the virtual reality scene of the entire underground mining operation;

   Step 3: Replace the tool model of the underground mining operation and the simulated material of the mine geology

   Add hidden menus in the virtual underground mining scene. When roaming to the ore mining area, the model or material selection menu will appear automatically. Users can select models or materials to replace them from the menu according to their needs.

   Step 4: Embed the video material into the 3D application scene and control playback and stop

   Embed the video material into the virtual reality scene, play it in the three-dimensional space, simulate the monitoring display device of the mining environment, set the keyboard X key, bind the MediaPlayer media class of the UE4 platform, and control the playing and stopping of the video through the OpenSource and Close functions;

   Step 5: Select the model and move to any location

   Select the model by mouse and move the model to any position where the simulation operation is needed to achieve the mechanical movement simulation in the real scene;

   Step 6: Implement the intent interaction of the application scenario

   When the user roams to a specific location in the virtual reality application scenario, and the system detects that the user intentionally enters, the environment light is automatically turned on to implement natural interaction in the virtual scenario;

   Step 7: QR code generation

   Bind the F key of the keyboard, add the QR code generation function, set the keyboard key control to generate the QR code function, and press the F key of the keyboard to generate a two-dimensional code of the virtual scene panorama with the set sampling points;

   Step 8: Implement voice interaction

   The user controls the coal mining machine in the virtual reality scene through keywords including forward rotation, reverse rotation, boom raising, lowering arm and stopping, and simulates the operation effect;

   Step 9: AR dynamic demonstration function mode switching

   The user clicks the AR mode button in the upper right corner of the system to switch to the AR presentation mode.
2. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 1, wherein in step 3, the model is instantiated into a specific Actor, and the SetMesh function and the SetMaterial function are added to replace the model and the model. Material, set Widget Blueprint user interface and Box collision collision detection to realize hidden menu function in 3D space.
3. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 1, wherein in step 5, a mouse event is added for the model to be operated, the model is selected by the GetHitResult function, and then according to the mouse. The coordinates of the three-dimensional space change the coordinate value of the SetActorLocation function of the model. When the mouse clicks again, the coordinates of the three directions of the mouse x, y, and z are assigned to the model. At this time, the GetHitResult function sets the model to the unselected mode.
4. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 1, wherein in step 6, a TriggerBox trigger is set, and when the first person role triggers a TriggerBox, the system detects that the user intentionally enters a certain A zone automatically activates a device in this zone.
5. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 1, wherein in step 7, the user presses the keyboard F key, and the system generates a virtual scene panorama with the set sampling points. Code, the user scans the QR code with the mobile phone and jumps to the virtual application scene display page of the mobile phone. On the mobile terminal, the user can enable the gyroscope, switch to the VR split screen mode, set the mobile phone parameters, and then use the VR glasses to experience the virtual underground. The mining operation environment scene realizes the 720-degree view display, and can also realize the multi-scene and multi-angle roaming experience on the mobile phone side.
6. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 1, wherein in step 8, the speech recognition is implemented based on the Pocket-sphinx library, and the Chinese keyword dictionary is improved, and the preprocessing is performed. Feature extraction, acoustic model training, language model training, and speech decoding and search to realize the recognition function. Finally, the UE4 engine's writing function control function is used to realize the control of the speech model in the three-dimensional space. The specific implementation steps of the speech recognition are as follows:

   Step 8.1: Pretreatment

   Processing the input original speech signal, filtering out unimportant information and background noise, and processing end point detection, speech framing and pre-emphasis of the speech signal;

   Pre-emphasis is achieved by a first-order FIR high-pass digital filter. The transfer function of the first-order FIR high-pass digital filter is:

   H(z)=1-az ^-1 ;

   Where a is the coefficient of the pre-emphasis filter, and the value ranges from 0.9 to 1.0. If the speech sample value at time n is x(n), the pre-emphasized signal is

   y(n)=x(n)-a*x(n-1);

   Step 8.2: Feature Extraction

   Feature extraction is performed by the method of Mel Frequency Cepstral Coefficient (MFCC); the following steps are performed as follows:

   Step 8.2.1: using the critical band effect of human hearing, using MEL cepstrum analysis technique to obtain a MEL cepstral coefficient vector sequence for speech signal processing;

   Step 8.2.2: use the MEL cepstral coefficient vector sequence to represent the spectrum of the input speech, and set a number of bandpass filters with triangular or sinusoidal filtering characteristics in the speech spectrum range;

   Step 8.2.3: Find the output data of each band pass filter through the band pass filter bank;

   Step 8.2.4: Logarithmically output data of each band pass filter and perform discrete cosine transform (DCT);

   Step 8.2.5: Obtain the MFCC coefficient; the solution formula is as follows:

   

   Where C _i is a characteristic parameter, k is the number of triangular filters, F(k) is the output data of each filter, P is the filter order, and i is the data length;

   Step 8.3: Acoustic Model Training

   The acoustic model parameters are trained according to the characteristic parameters of the training speech library;

   In the identification, the characteristic parameters of the speech to be recognized can be matched with the acoustic model to obtain the recognition result. In this paper, the mixed Gaussian model-Hidden Markov Model (GMM-HMM) is used as the acoustic model, which includes the following steps:

   Step 8.3.1: Find the joint probability density function of the mixed Gaussian model as follows:

   

   Where M is the number of Gaussian in the mixed Gaussian model, C _m is the weight, u _m is the mean, ∑ _m is the covariance matrix, D is the observation vector dimension, and the maximum expected value algorithm (EM) is used to mix the Gaussian model parameter variables. Θ={C _m ,u _m ,∑ _m } is estimated and solved by the following formula:

   

   

   

   Where j is the number of current iterations, N is the number of elements in the training data set, x ^(t) is the eigenvector at time t, h _m (t) is the posterior probability at time t _m ; GMM parameters are passed EM algorithm Estimating can maximize the probability of generating a speech observation feature on the training data;

   Step 8.3.2: Solve the three components of the HMM

   Let the state sequence be q ₁ , q ₂ ,...,q _N , and let the transition probability matrix A=[a _ij ]i,j∈[1,N], then the jump probability between the Markov chain states is :a _ij =P(q _t =j|q _t-1 =i); the initial probability of Markov chain π=[π _i ]i∈[1,N], where π _i =P(q ₁ = i); Let the observed probability distribution b _i (o _t )=P(o _t |q _t =i) of each state, use the GMM model to describe the observed probability distribution of the state; according to step 8.3.1, the formula is:

   

   Where N is the number of states, i and j are states, a _ij is the probability of jumping from the i state to the t state at time t-1, o _t is the observed value at time t, and C _i,m is the mixing coefficient , indicating the weight between different Gaussians, u _i,m represents the mean between different Gaussians, ∑ _i,m represents the covariance matrix between different Gaussians; the parameters of HMM are estimated by Baum-Welch algorithm, and finally generated Acoustic model file;

   Step 8.4: Language Model Training

   The N-Gram model is used to implement the training of the language model; the probability of the occurrence of the i-th word in a sentence depends on the N-1 words in front of it, that is, the context of a word is defined as the N-1 appearing in front of the word. Word whose expression is:

   

   Replace the above expression with the following formula using the conditional probability formula S:

   P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w ₁ ,w ₂ ,...,w _n-1 )

   Where, P (w ₁₎ is the probability w ₁ appears in the _{article, P (w 1, w 2} ) is w _1, w ₂ consecutive probability _{of, P (w 2 | w 1} ) is already known w ₁ The probability of occurrence of w ₂ occurs in the case of occurrence, assuming that the probability of identifying the sentence is represented by P(s), and P(s)=P(w ₁ , w ₂ , . . . , w _n ) represents the word set w ₁ , w ₂ ,... , the probability that w _n appears continuously and generates S;

   Streamlined to the following formula by the Markov assumption:

   P(sentence)=P(w ₁ )*P(w ₂ |w ₁ )*P(w ₃ |w ₂ )...*P(w _n |w _n-1 )

   Where P(w _i |w _i-1 )=P(w _i-1 ,w _i )/P(w _i ), P(w _i-1 ,w _i ) and P(w _i ) can all be learned from the corpus Statistics, finally, P (sentence), the language model stores the probability statistics of P(w _i-1 , w _i ), and the entire recognition process is realized by finding the maximum value of P (sentence);

   Step 8.5: Speech decoding and search algorithm

   For the input speech signal, an identification network is established according to the trained acoustic model, the language model and the dictionary mapping file created by the g2p tool, and the best path is found in the network according to the search algorithm. The word string of the speech signal is output with the maximum probability, so that the text contained in the speech sample is determined. The Viterbi algorithm is used to implement speech decoding. The specific process is as follows:

   Step 8.5.1: Enter the parameters of the HMM model and the observation sequence O={o ₁ ,o ₂ ,...,o _T }, then all state probabilities when t=1 are:

   δ ₁ (i)=π _i b _i (o ₁ )

   ψ ₁ (i)=0

   Step 8.5.2: Gradually recursive to t=2,3,...,T, then:

   

   

   Step 8.5.3: Terminate traversal:

   

   

   Step 8.5.4: Backtracking the optimal path, t=T-1, T-2,...,1;

   

   Step 8.5.5: Output the optimal hidden state path

   

   Where δ _t (i) is the joint probability of all nodes through which the optimal path passes when recursively to t, ψ _t (i) is the implicit state at time t, T is time, and P ^* is the probability of the optimal path ,

   

   The endpoint for the optimal path.
7. The virtual reality and augmented reality based mining operation multi-interaction implementation method according to claim 6, wherein: a takes 0.97.
8. The virtual reality and augmented reality-based mining operation multi-interaction implementation method according to claim 1, wherein in step 9, the method comprises the following steps:

   Step 9.1: Model Selection

   Select the shearer model, the roadheader model, the wind coal drill model and the fully mechanized support model. Each type of model is a 1:1 modeling simulation of the real coal mining tool;

   Step 9.2: Model explanation

   After the user selects the model, and then selects the tool model option to be learned through this menu, the system will play the corresponding voice explanation, and click the button voice again to stop;

   Step 9.3: Model Demo

   The tool simulation running animation created in the 3DMax modeling process is imported into the UE4 engine, and the corresponding selection menu is set, and the running state of the corresponding coal mining tool can be demonstrated in the AR mode by clicking;

   Step 9.4: Screenshot generation icon

   In the main menu of AR mode, add a button, bind the camera's screenshot function, add a scrolling menu bar on the right side of the menu, when the screenshot function is successfully triggered, the screenshot is displayed to the right by scrolling the menu bar by setting the dynamic material conversion function. During the demonstration, the user clicks the screenshot button, and the system generates an icon on the interface side;

   Step 9.5: Rotate

   Instantiate the set model as an Actor, add the Rotation function, and implement the model to rotate clockwise;

   Step 9.6: Function Extension

   Add a secondary UI, control Map switching, and implement running demo functions including Earth, Saturn, Mercury, Atmospheric Planet, and Galaxies; add WidgetBlueprint encoding to display or hide the Knowledge Profile panel; Design Back button to return to AR Editor Module

   Step 9.7: The dynamic gesture control model, the real environment is superimposed with the virtual model, and the gesture and the model are interactively controlled, including the following steps:

   Step 9.7.1: Initialize video capture, read the logo file and camera camera parameters;

   Step 9.7.2: Grab the video frame image;

   Step 9.7.3: performing the detection mark and identifying the mark template in the video frame, and performing motion detection on the acquired video frame image by using the OpenCV library function to determine whether the motion track is detected;

   If: the judgment result is that the gesture motion track is detected, then step 9.7.4 is performed;

   Or the result of the judgment is that no motion track is detected, then the detection mark is continued and the mark template in the video frame is identified, and then step 9.7.12 is performed;

   Perform motion detection based on the color histogram and the background difference, and perform background update on the acquired frame and the pixels outside the motion gesture area after detecting each frame motion, and the formula is as follows;

   

   Where u _t is the corresponding pixel point of the background image, u _t+1 is the updated background image pixel point; I _t is the pixel point of the current frame image, and I _f is the mask value of the current frame image pixel point, that is, whether Do background update; a∈[0,1] is the background image model update speed, this paper takes 0.8;

   Step 9.7.4: Perform preprocessing including denoising on the image;

   Through the motion detection step, if motion information is detected, the video frame image containing the motion gesture is preprocessed: median filtering is performed on the image by the medianBlur function of OpenCV to remove the salt and pepper noise;

   Step 9.7.5: Convert to HSV space;

   The image is color-space converted by the cvtColor function to obtain the data of its HSV space, and the brightness v value in the HSV space is reset as shown below:

   

   Where r, g are red and green pixels of the skin color region, and r>g;

   Step 9.7.6: Split the hand area;

   Step 9.7.7: Perform morphological processing to remove impurities;

   Combining the obtained motion binary image with the binary image obtained by back projection, and performing image morphology closing operation to obtain a relatively complete motion skin color gesture binary image; and removing the noise points in the image;

   Step 9.7.8: Obtain the hand contour;

   After preliminary morphological operations, removing noise and making the boundary of the hand clearer, the gesture contour is obtained by OpenCV's findContours function, and then the pseudo contour is removed.

   Step 9.7.9: Draw the outline of the hand and calibrate the information;

   Step 9.7.10: Comparison of contour information, setting direction vector;

   Compare the contours obtained in each frame, set the comparison conditions, and assign values to the direction marker variables by comparison;

   Step 9.7.11: Perform force simulation on the model according to vector coordinates to realize interaction between dynamic gestures and virtual models;

   After the dynamic gesture is judged by the contour, the virtual model is subjected to the force simulation operation according to different judgment results. According to the value of the direction marker in the contour judgment process, the coordinate values of the model in the three-dimensional space will be three axes of x, y and z. The multiplication calculation on the above, through the change of the coordinate value, the change of the model position is achieved to achieve the simulation of the force;

   Step 9.7.12: Calculate a conversion matrix of the camera relative to the detected mark;

   Step 9.7.13: superimpose the virtual object on the detected mark, and return to step 9.7.2 to realize superimposed display of the real environment and the virtual model;

   Step 9.7.14: When the VR mode is clicked, the system switches the display mode, the camera is turned off, and the above steps are stopped.

Images