CN113411546B - AR-based coal mining machine visual monitoring method - Google Patents
AR-based coal mining machine visual monitoring method Download PDFInfo
- Publication number
- CN113411546B CN113411546B CN202110539085.0A CN202110539085A CN113411546B CN 113411546 B CN113411546 B CN 113411546B CN 202110539085 A CN202110539085 A CN 202110539085A CN 113411546 B CN113411546 B CN 113411546B
- Authority
- CN
- China
- Prior art keywords
- coal mining
- mining machine
- data
- monitoring
- coal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/156—Mixing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
Abstract
The invention provides an AR-based coal mining machine visual monitoring method, which comprises the following steps: firstly, arranging a sensor and an airborne camera on a coal mining machine, and acquiring operation data of the coal mining machine and monitoring data of a monitored coal wall in a working area; secondly, constructing a 3D coal mining machine monitoring virtual model according to the fully mechanized coal mining face equipment matching drawing; processing the continuous monitoring video data to enable the video data to be consistent with a virtual coal wall object in the 3D coal mining machine monitoring virtual model; then, projecting the processed video data on a corresponding virtual coal wall object, and driving the position and the geometric posture of a monitoring virtual model of the 3D coal mining machine in real time to enable the monitoring virtual model of the 3D coal mining machine to be fused with the monitoring data and the video data in a space-time mode; and finally, reproducing an underground coal mining scene in real time in a real environment by means of an AR technology. By means of AR and visualization technology, the underground mining environment is dynamically and vividly reproduced in real time, and the efficiency and accuracy of monitoring data cognition of monitoring personnel are improved.
Description
Technical Field
The invention relates to the technical field of coal mining, in particular to an AR-based coal mining machine visual monitoring method.
Background
The coal mining environment is complex and dangerous, and the research and development targets of the coal comprehensive mechanized mining are less humanization and unmanned, so that the informatization, digitization and intellectualization are the main directions of the coal mine fully-mechanized mining development. The coal mining machine is used as important equipment for comprehensive mechanized mining, bears key tasks of coal cutting and coal loading, and the starting rate and the reliability of the coal mining machine directly influence the mining efficiency and the economic benefit of a coal mine, so that the coal mining machine is required to have high reliability. The reliability of the coal mining machine depends on the product quality of the coal mining machine and whether the coal mining machine is operated and maintained correctly in the operation process, and an operator needs to monitor the operation data of the coal mining machine in real time and adjust the operation parameters of the coal mining machine at any time to enable the coal mining machine to be in the optimal working condition, so that the operation efficiency is improved, and the service life is prolonged. The traditional coal mining machine monitoring system is positioned in a crossheading or ground dispatching command center to monitor the operation parameters and the operation environment of the coal mining machine.
The mining environment data monitoring of the coal mining machine adopts a video monitoring mode, namely a plurality of cameras are arranged on a hydraulic support of a fully mechanized mining face at fixed intervals, video acquisition is carried out on the production field environment around the coal mining machine, then the video acquisition is transmitted to a control center, along with the advance of the coal mining machine, a monitoring system needs to continuously switch the monitoring cameras to realize the follow-up monitoring, and the operation monitoring data and the geometric posture of the coal mining machine are displayed by a 2D display.
The coal mining machine monitoring data comprises 3D physical data (coal mining machine position, geometric attitude data, relative coal wall position data and the like) and 2D abstract data (operation data, monitoring video and the like). In a traditional coal mining machine monitoring system, 3D physical graphic data are converted into 2D graphics through multiple projection transformation and are displayed in parallel with 2D abstract data, monitoring personnel need to reversely convert the 2D graphics into the 3D physical graphic data in real time in the brain and then associate the 3D physical graphic data with the 2D abstract data, so that the problems that the monitoring personnel are heavy in thinking burden, the data association is easy to be disordered and the like are caused, and further, the decision making is slow and even misjudgment is caused. Therefore, research on a visualization monitoring system based on 3D space display and fusion of various data by utilizing augmented reality and data visualization technologies is urgent and necessary.
The development direction of the coal mine automatic mining technology is less humanization, digitalization and intellectualization, and the mainstream fully-mechanized mining working face system adopts an automatic equipment production and remote operator intervention mode. The coal mining machine is an important device of the fully mechanized coal mining face, and monitoring personnel need to observe the running state, mining process parameters and surrounding mining environment in real time. The existing remote monitoring system of the coal mining machine adopts a plurality of plane displays to display monitoring data in parallel, and mainly has the following problems:
1) The geometric pose data of the coal mining machine are not visual
The existing coal mining machine monitoring system has two types, namely a 2D monitoring system and a VR-based transparent working face system, and has the problem that the geometric pose data of the coal mining machine is not intuitive.
The 2D monitoring system transforms the 3D geometric figure data of the coal mining machine into a 2D projection figure with a fixed visual angle in a projection manner, the projection figure is displayed by a plurality of display screens in a split screen manner, monitoring personnel can only observe the pose of the coal mining machine from a plurality of fixed visual angles, and the display effect is not visual.
The transparent working face system is developed based on the Internet of things, VR and digital twin technology, a three-machine 3D virtual model of the fully mechanized working face is constructed, the fully mechanized working face is reproduced in a 3D space created by a computer, and monitoring data are displayed by multiple 2D displays in a split screen mode. The transparent working face system has the advantage that monitoring personnel can observe the geometric pose and mining environment data of the coal mining machine from any visual angle. The disadvantages are that the 3D geometry data of the shearer can only be projected into 2D graphics to be presented on the display due to the 2D display used as the display output device, and it is difficult to manipulate the 3D model in the 2D display space.
2) The monitoring video is not associated with the geometric pose data of the coal mining machine in time and space
The traditional coal mining machine mining environment monitoring system consists of a camera, a transmission network, video processing and data storage, and is shown in figure 1. The cameras are arranged under the top beam of the hydraulic support, and 1 camera is arranged at intervals. The monitoring system starts the corresponding camera according to the position of the coal mining machine, and automatically switches the camera along with the movement of the coal mining machine, so that the coal mining of the coal mining machine is continuously tracked and monitored in the whole process. In the reciprocating coal mining process of the coal mining machine, the monitoring video of the coal mining machine obtained by alternately fixing the machine position cameras is discontinuous, the monitoring video is mostly shot at an inclined angle, and a certain deformation exists after the monitoring video picture is changed through projection; the geometric poses of the monitoring video pictures and the 3D virtual model of the coal mining machine are not matched in spatial position and size, and the mining scene of the coal mining machine cannot be presented in a time-space consistent mode. Therefore, monitoring personnel can not visually judge whether the working state of the coal mining machine and the surrounding production environment meet the requirements of safe production.
3) Difficulty in association of 3D model and 2D monitoring data of coal mining machine
A traditional coal mining machine visual monitoring system adopts a plurality of 2D flat panel displays to display monitoring data in a split screen mode, and a 3D virtual model needs to be projected and transformed into a 2D image to be displayed with the 2D monitoring data in parallel. The 3D geometric model of the coal mining machine is difficult to associate with the 2D monitoring data after being transformed by forward and backward projection, and the abstract metadata is difficult to directly associate with the representative equipment object in a 3D space, so that the problems of heavy thinking burden, disordered data association and the like of monitoring personnel are caused, and slow decision and even misjudgment are caused.
The external mining environment data of the coal mining machine mainly comprises monitoring videos of the peripheral coal wall and equipment of the coal mining machine, the traditional working face mining environment monitoring scheme is shown in figure 1, along with uninterrupted reciprocating coal mining of the coal mining machine, a monitoring video near the coal mining machine is obtained by alternately starting fixed machine position cameras, the monitoring video is discontinuous, is not matched with the geometric shape, size and geometric pose data of the coal mining machine in space, cannot be associated with 3D physical graphic data and abstract monitoring data of the coal mining machine, the intuitiveness of the monitoring data is poor, and the information acquisition efficiency is low.
Disclosure of Invention
Aiming at the defects in the background technology, the invention provides an AR-based coal mining machine visual monitoring method, which takes monitoring data as a research object, and by means of AR and a visual technology, solves the problems that the 3D geometric data in the traditional coal mining machine monitoring system is difficult to be associated with 2D abstract data after projection transformation, the cross-space-time association between metadata and the corresponding 3D object is difficult to realize, and the like, improves the cognitive efficiency of monitoring personnel on the monitoring data, and ensures timely and reliable decision.
The technical scheme of the invention is realized as follows:
a coal mining machine visualization monitoring method based on AR comprises the following steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to acquire video monitoring data; acquiring the geometric pose and working data of equipment by using an airborne sensor; transmitting the continuous monitoring video data, the geometric pose of the equipment and the operation data to a fully mechanized mining face monitoring system;
step two: visually presenting a 3D coal mining machine virtual model in a real environment by using AR equipment, wherein the virtual model comprises a coal mining machine, a scraper and a monitored coal wall;
step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; driving the position of the virtual coal wall by the distance data between the coal mining machine and the monitored coal wall; the position data of the coal mining machine drives a scraper conveyor model; the coal mining machine model is driven by the coal mining machine posture, the geometric posture and the warning information;
step four: displaying the attribute data and the operation data of the coal mining machine on a touch display, and associating the attribute data and the operation data with a 3D coal mining machine virtual model presented by the AR equipment through a network;
step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine virtual model in time and space, the posture of the 3D coal mining machine virtual model is completely consistent with that of a coal mining machine in the monitoring video, and an underground coal mining scene is visually reproduced in real time in a real space.
The video monitoring data comprise embedded abstract data, and the embedded abstract data comprise coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring video data and coal mining machine position data; the 3D coal mining machine monitoring virtual model is embedded and driven by coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring data and coal mining machine position data.
The geometric pose and the working data of the equipment are associated abstract data, and the geometric pose and the working data of the equipment are associated with the monitoring virtual model of the 3D coal mining machine.
The 3D coal mining machine monitoring virtual model comprises a coal mining machine position virtual model, a monitored coal wall virtual model and a coal mining machine virtual model; embedding the position data of the coal mining machine into the virtual model of the position of the coal mining machine; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding the coal mining machine attitude data, the geometric attitude data and the key component warning data into a coal mining machine virtual model; the virtual model of the coal mining machine is respectively associated with the geometric pose of the equipment and the working data; the virtual model of the position of the coal mining machine is a scraper model.
The operation method of the touch module of the coal mining machine visual monitoring system comprises the following steps:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data retrieval: connecting the fully mechanized coal mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized coal mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
The operation method of the AR module of the visualized monitoring system of the coal mining machine comprises the following steps:
initialization: starting the AR head-mounted display equipment, automatically scanning the working environment around the helmet in the monitoring room by the equipment, and carrying out spatial modeling; determining positioning information of a space where the coal mining machine equipment is located by means of an SLAM technology and a space anchor technology of AR equipment; starting an AR monitoring program of a coal mining machine on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR equipment last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, a monitoring video and warning information of a 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time to enable the monitoring video and the operation data of the coal mining machine to be in space-time synchronization with the geometric posture of the coal mining machine.
Compared with the prior art, the invention has the following beneficial effects:
1) The AR-based visualization monitoring technology for the coal mining machine takes AR, visualization and man-machine interaction technologies as a core, combines multidisciplinary technologies such as 3D virtual model construction, video monitoring and digital twinning, fuses and associates 3D physical graphic data and 2D abstract monitoring data of the coal mining machine in a 3D space, dynamically reproduces underground mining scenes of the coal mining machine in real time, and efficiently and visually presents running data and running states of the coal mining machine;
2) The invention assists the existing fully mechanized coal mining face remote monitoring system, realizes remote virtual visual mining, provides technical support for less people and intelligent production of coal mine comprehensive mechanical mining, and has important guiding significance and reference value for digital operation and maintenance and intelligent mining of the coal mining machine.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a block diagram of a mining environment monitoring system of a conventional coal mining machine.
Fig. 2 is a three-machine matching diagram of the fully mechanized mining face.
Fig. 3 is a real view of the fully mechanized coal mining face.
Fig. 4 is an external mining environment monitoring model of a coal mining machine of the present invention.
Fig. 5 is a schematic diagram of 2D object projection mapping for 2D image.
FIG. 6 is a flow chart of the present invention.
FIG. 7 is a coal mining machine monitoring video mapping model established by the present invention.
Fig. 8 is a simplified model of electromechanical components of a coal mining machine constructed in accordance with the present invention.
Fig. 9 is a position coordinate system of the coal mining machine established by the present invention.
Fig. 10 is a 3D digital model of a scraper conveyor built by the present invention.
FIG. 11 is a shearer position visualization model.
FIG. 12 is a block diagram of a fully mechanized coal mining face process simulation system of the present invention.
FIG. 13 shows a drawing C of the present invention1A camera position schematic; wherein (a) is C1A front view of the camera, and (b) is C1Side view of the camera.
FIG. 14 is a view of the onboard camera of the present invention; wherein (a) is C1The camera takes pictures, and (b) is C2The camera shoots pictures, and (C) is C3The camera shoots pictures, and (d) is C4The camera takes pictures.
FIG. 15 is a schematic diagram of fusion of surveillance video data and physical three-dimensional data according to the present invention.
Figure 16 is the main menu of the AR device.
Fig. 17 is a schematic diagram of the operation of moving the position of the shearer scene model according to the present invention.
Fig. 18 is a schematic view of the rotating operation of the shearer scene model of the present invention.
Fig. 19 is an example of the operation of the immersive human-computer interaction module of the present invention.
Fig. 20 shows an example of the operation of the visualization monitoring system of the coal mining machine according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art based on the embodiments of the present invention without inventive step, are within the scope of the present invention.
The fully mechanized mining face is enclosed by a coal mining machine, a hydraulic support, a scraper conveyor and a coal wall to form a safe mining space, the working environment is severe, and the space is narrow, as shown in fig. 2 and 3. The coal mining machine carries out reciprocating mining along the coal wall, the working space is limited and closed, and the relative position parameters of the monitored coal wall (which can be approximate to a 2D plane) and the coal mining machine can be solved by a three-machine matching diagram of the fully mechanized mining face and a remote control system. The external mining environment of the coal mining machine is visually monitored by an onboard camera. Arranging a plurality of onboard monitoring cameras on a coal mining machine body, wherein the shooting direction of the cameras is vertical to the coal wall, continuously shooting the coal wall and related equipment around the coal mining machine, and monitoring the external mining environment of the coal mining machine, as shown in figure 4, CnBeing an onboard camera, PnFor monitoring video sequences, PnFor coal wall pictures of the monitored working area, Pn ′Is a corresponding virtual coal wall object.
The embodiment of the invention provides an AR-based coal mining machine visual monitoring method, which comprises the following specific steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to obtain continuous monitoring video data; acquiring the geometric pose and working data of the equipment by using an airborne sensor; transmitting the continuous monitoring video data, the equipment geometric pose and the operation data to a fully mechanized coal mining face monitoring system; the video monitoring data comprises embedded abstract data, and the embedded abstract data comprises coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring video data and coal mining machine position data; embedding and driving the 3D coal mining machine monitoring virtual model into the coal mining machine attitude data, the geometric attitude data, the key component warning data, the mining area position data, the mining area monitoring data and the coal mining machine position data. The geometric pose and the working data of the equipment are associated abstract data, and the geometric pose and the working data of the equipment are associated with the monitoring virtual model of the 3D coal mining machine.
The coal mining machine attitude data comprises a heading angle alpha, a pitch angle beta and a roll angle gamma of the machine body relative to the absolute coordinates of the earth. The three rotation angle parameters of the 3 data-driven coal mining machine virtual models can embed the machine body attitude data into the coal mining machine virtual models, and visualization of the machine body attitude data is achieved.
The geometric pose data comprises the rotation angle theta of the rocker arm and the rotation angle theta of the heightening oil cylinder4And the telescopic distance delta Y of the movable column of the heightening oil cylinder; the geometric attitude of the coal mining machine is composed of theta and theta4Driving the delta Y parameter, wherein theta drives the angle of the rocker arm of the coal mining machine, and the numerical value of theta drives the angle of the rocker arm of the coal mining machine from rotary encoders arranged at the left rocker arm and the right rocker arm of the coal mining machine; theta.theta.4Driving to increase the angle of the oil cylinder; and the delta Y drives the extending amount of the movable column of the heightening oil cylinder, and the coal mining machine virtual model can be driven by the left and right 6 parameters. And embedding the 6 geometrical attitude data to realize the visualization of the geometrical attitude of the coal mining machine.
The safety state data of the core machine, the electric, the hydraulic and the control component of the coal mining machine can adopt color identification. According to the standard "GB 2893-2008 security color", the colors that convey security information are red (dangerous, prohibited), blue (instruction), yellow (attention, warning) and green (security). The corresponding safety status data (danger, warning, safety) is expressed by changing the color of the component (red, yellow, green), as shown in fig. 7. When the part is in a dangerous and warning state, the brightness of the part is set to be flashing, and the transparency of the rest parts is reduced so as to highlight the part in the abnormal state and simultaneously send out warning sound to further warn a user. The equipment warning data is embedded into the physical virtual body of the coal mining machine through color, flicker and sound, so that the visualization of the warning data of key components of the coal mining machine is realized.
An absolute encoder is arranged on a low-speed shaft of a traction part of a travelling mechanism of the coal mining machine, and absolute coordinates of the coal mining machine on a scraper conveyor can be solved by measuring the number of rotation turns of a gear. A 2D reference coordinate system shown in fig. 9 is established, with the coal wall facing direction as the X coordinate direction, the scraper conveyor head as the origin of coordinates, and the scraper conveyor length direction as the Y coordinate direction. Acquiring the push coordinate data of the scraping plate groove from the existing fully mechanized mining face monitoring system, namely the X coordinate of the scraping plate groove, wherein the Y coordinate of the scraping plate groove can be approximate to the normal coordinateAnd measuring to obtain the spatial distribution shape of the scraper groove in a two-dimensional coordinate system. A3D digital model of the scraper conveyor is built by replacing a scraper groove with a complicated shape with a cuboid, and is shown in figure 10. Neglecting the rotation angle of the scraper trough, positioning the scraper trough by pushing X coordinate data, and driving a 3D virtual model of the scraper conveyor to obtain a 3D virtual model approximate to the actual geometric posture. The pushing size of the scraper blade groove is too small relative to the length size of the whole machine, so that the geometric form of a 3D virtual model in a monitoring system is not visual, and the pushing size of the scraper blade conveyor is enlarged by 3 times for displaying. Total length of coal mining machine is L1Length L of scraper trough of scraper conveyor2N = int (L)1/L2) The block scraper trough represents the position of the shearer. Taking a scraper on which a bilateral symmetry plane of the coal mining machine is positioned as a reference, and taking n = int (L) on the left and right sides respectively1/L2And/2) modifying the color of the scraper to be yellow, setting continuous flashing, and visually presenting the position data of the coal mining machine, as shown in FIG. 11. The appearance of the scraper virtual model in the direction to be mined of the coal mining machine is set to be green and represents the running direction of the coal mining machine; the appearance of the virtual model of the scraper on the mined side of the coal mining machine is set to be red; and refreshing the color of the corresponding scraper virtual model along with the continuous change of the position of the coal mining machine. Embedding the position data and mining direction data of the coal mining machine into the 3D position model in a color form, driving the X coordinate of the 3D virtual model of the scraper blade by using the pushing data, and changing the position of the scraper blade so as to visualize the position of the coal mining machine in an intuitive mode.
A coal mining machine monitoring video mapping model shown in the figure 8 is established, based on the SAR technology, a plurality of airborne cameras are adopted to shoot a monitored coal wall virtual model, 2D monitoring videos are amplified in equal proportion and cut, and are aligned and superposed to a plane where a corresponding coal wall is located through space registration and registration, so that a highly real and three-dimensional dynamic coal mining scene can be generated in a virtual-real fusion space created by AR. Amplifying and cutting the monitoring image according to a certain size proportion, respectively mapping the monitoring image to the monitored coal wall planes of the coal mining machine corresponding to the graph 7, and calculating to obtain the distance L between the 4 monitored coal wall planes and the coal mining machinenVideo amplification ratio lambdanVideo cropping size AnAnd BnCan be drivenAnd (4) monitoring the plane of the coal wall, and reproducing the coal mining working scene of the coal mining machine in real time. The monitoring video is embedded into the corresponding monitored coal wall virtual model, the size and the position of the coal wall virtual model and the video amplification ratio are driven through related parameters, and visualization of the mining environment monitoring data of the coal mining machine is achieved.
Step two: using AR equipment to visually present a 3D coal mining machine virtual model in a real environment, wherein the virtual model comprises a coal mining machine, a scraper and a monitored coal wall; the 3D coal mining machine monitoring virtual model comprises a coal mining machine position virtual model (or a scraper machine model), a monitored coal wall virtual model and a coal mining machine virtual model; embedding the position data of the coal mining machine into the virtual model of the position of the coal mining machine; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding the coal mining machine attitude data, the geometric attitude data and the key component warning data into a coal mining machine virtual model; the virtual model of the coal mining machine is respectively associated with the geometric pose of the equipment and the working data.
Step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; the distance data between the coal mining machine and the monitored coal wall drives the position of the virtual coal wall; the position data of the coal mining machine drives a scraper conveyor model; the coal mining machine model is driven by the coal mining machine posture, the geometric posture and the warning information; the coal mining environment monitoring model monitors the coal wall and related equipment around the coal mining machine through the onboard camera, and in a 3D space created by AR equipment, an onboard monitoring video is accurately superposed on a corresponding coal wall object of the 3D model of the coal mining machine through projection transformation and space registration and registration technologies, so that a 3D coal mining scene with high third dimension and reality sense can be generated. When the monitored coal wall virtual model is a 2D object, the plane where the monitored coal wall virtual model is located is parallel to the imaging plane of the monitoring camera, the size of the monitoring video shot by the airborne camera and the shot 2D plane object are in an equal proportional amplification relation, and the proportionality coefficient is k. After the 2D monitoring video is cut in an equal-scale scaling mode, the 2D monitoring video is registered with a 2D virtual coal wall object space in an AR space, namely, the 2D monitoring video is registered and aligned to the 2D virtual coal wall object, and real-time dynamic space-time fusion of monitoring data of a coal mining machine and a virtual digital model of the coal mining machine is achieved, and the method is shown in figure 5.
Step four: and displaying the attribute data and the operation data of the coal mining machine on the touch display, and associating the attribute data and the operation data with the 3D coal mining machine virtual model presented by the AR equipment through a network.
Step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine virtual model in time and space, the posture of the 3D coal mining machine virtual model is completely consistent with that of a coal mining machine in the monitoring video, and an underground coal mining scene is visually reproduced in real time in a real space.
The visualized monitoring system of the coal mining machine is based on AR and visualization technology, monitoring data are obtained from an existing fully mechanized mining face monitoring system, and abstract monitoring data of the coal mining machine and an image-bearing physical virtual model are combined in a correlation mode through 3D visualization technology. The visual monitoring system of coal-winning machine based on AR is as auxiliary system, satisfies the demand that monitoring personnel directly perceived, high-efficient monitoring coal-winning machine operating condition. The coal mining machine visual monitoring system is composed of an AR module and a flat touch man-machine interaction module, the AR module uses AR equipment to visually display in a 3D mode, the flat touch man-machine interaction module uses a traditional touch display to display, and the two modules are in data association through a network. The flow of the visualization monitoring system of the AR-based coal mining machine is shown in FIG. 6.
As shown in fig. 6, the design process of the flat touch-control human-computer interaction module (coal mining machine visual monitoring system) is composed of initialization, data retrieval, and abstract data visualization 3 parts:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data calling: connecting the fully mechanized mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
As shown in fig. 6, the AR module (coal mining machine monitoring virtual model) design process is composed of three parts, namely initialization, 3D virtual model loading and model visualization driving:
initialization: starting the AR equipment, automatically scanning and monitoring the working environment around the indoor helmet by the equipment, and performing spatial modeling; determining positioning information of the space where the coal mining machine equipment is located by means of an SLAM technology and a space anchor technology of the AR equipment; starting an AR program on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR monitoring program of the coal mining machine last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, monitoring video and warning information of the 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time.
There are two modes of embedding and associating 3D-like physical virtual objects and 2D abstract data of the shearer, as shown in fig. 6. Direct contact is established between the two data, the thinking burden of monitoring personnel can be reduced, the decision making efficiency is improved, and the data visualization effect is excellent. When the panel touch interface selects certain abstract data, the panel touch interface highlights the data, and meanwhile, the AR human-computer interaction interface automatically highlights the 3D virtual model of the corresponding device. When the AR human-computer interaction interface selects an image 3D virtual model of a certain device, the AR interface highlights the 3D virtual model, meanwhile, the flat touch type interaction interface automatically highlights corresponding abstract monitoring data, and the contact between the two is directly and visually prompted.
System test and run case
By taking a certain type of coal mining machine as an example, a coal mining machine visual monitoring system is designed, and visual fusion effects like 3D physical virtual bodies and 2D abstract data are verified. The machine-mounted camera monitoring video of the coal mining machine is acquired by a fully mechanized mining face process simulation system. The fully mechanized coal mining face process simulation system consists of a coal mining machine, a scraper conveyor, a hydraulic support and a coal wall, and is used for simulating and simulating a fully mechanized coal mining face end beveling feed coal mining process, as shown in fig. 12.
Referring to the external mining environment monitoring model of the coal mining machine shown in fig. 4, 4 virtual cameras C are arranged at corresponding positions of the coal mining machine in the fully mechanized mining face process simulation system1～C4In which C is1The camera positions are shown in figure 13. And adjusting relevant parameters of the virtual camera in the system until an expected monitoring video picture is obtained.
And starting the fully mechanized mining face process simulation system, and recording the videos by the 4 cameras in turn to respectively obtain the monitoring videos of the onboard cameras shown in the figure 14.
After the airborne monitoring video picture is scaled and cut in an equal proportion, the picture is mapped to a coal wall object corresponding to the environmental scene model of the coal mining machine, and the visual effect of the monitoring video is shown in fig. 15.
The abstract 2D monitoring data of the coal mining machine is obtained from an existing coal mining machine monitoring system, the position and posture data of the coal mining machine is obtained from a fully mechanized mining face process simulation system, and the monitoring data is arranged into a video file and an Excel table and stored to a specified position for later use.
The coal mining machine visual monitoring system based on the AR is composed of an AR human-computer interaction module and a flat touch control type human-computer interaction module, and the two modules are respectively started. And reading the monitoring data of the coal mining machine from the designated position, and driving the respective modules.
An AR man-machine interaction module of the coal mining machine visual monitoring system runs by AR equipment and is directly presented in a 3D space, and the running process of the system is as follows:
1) Starting AR man-machine interaction module
The augmented reality helmet is worn and started, the system main menu is opened, and the shearer AR monitoring program is selected, as shown in fig. 16.
2) Positioning of the coal mining machine monitoring scene model is performed by using gestures of AR equipment, the 3D coal mining machine monitoring virtual model is positioned to the specified position of the workbench, the scene model moving operation is shown in figure 17, and the scene model rotating operation is shown in figure 18. Note: the system uses the space anchor technology of the AR equipment, the program can automatically memorize the position of the model when exiting, and when the program is started for the second time, if the space is not changed, the secondary positioning is not needed.
3) The AR man-machine interaction module of the visualized monitoring system of the coal mining machine driven by the monitoring scene model data of the coal mining machine calls preset simulation data and simulation monitoring videos, and the operation effect of the system is shown in figure 19.
4) AR interaction module and flat-panel touch module associated operation
The visual monitoring system of the coal mining machine based on the AR is composed of a flat touch control type module and an AR module, data association and data exchange are carried out on the two modules through a network, and the effect of the two modules in cooperation and parallel operation is shown in figure 20.
The system test data is acquired by the fully mechanized mining face virtual simulation system, the onboard monitoring video of the coal mining machine and the operation data of the coal mining machine are simulated, and the system test and operation are carried out by taking the coal mining machine as a monitoring example. 3D avatar monitoring data and 2D abstract monitoring data of the coal mining machine are visually and dynamically presented, seamless dynamic fusion, interconnection and intercommunication are achieved, operation is convenient and fast, and expected effects are achieved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (3)
1. A coal mining machine visual monitoring method based on AR is characterized by comprising the following steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to obtain continuous monitoring video data; acquiring the geometric pose and operation data of the equipment by using an airborne sensor; transmitting the continuous monitoring video data, the equipment geometric pose and the operation data to a fully mechanized coal mining face monitoring system;
step two: visually presenting a 3D coal mining machine monitoring virtual model in a real environment by using AR equipment, wherein the 3D coal mining machine monitoring virtual model comprises a scraper machine model, a monitored coal wall virtual model and a coal mining machine virtual model; embedding the position data of the coal mining machine into a scraper model; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding attitude data, geometric attitude data and key component warning data of the coal mining machine into a virtual model of the coal mining machine; the virtual model of the coal mining machine is respectively associated with the geometric pose and the operation data of the equipment;
the attitude data of the coal mining machine comprises a course angle alpha, a pitch angle beta and a roll angle gamma of the machine body relative to the absolute coordinates of the earth; three rotation angle parameters of the 3 data-driven coal mining machine virtual model can embed the machine body attitude data into the coal mining machine virtual model, so that the visualization of the machine body attitude data is realized;
the geometric attitude data includes a rotation angle of the rocker arm
And the rotation angle of the oil cylinder is increased
And the telescopic distance of the movable column of the heightening oil cylinder
(ii) a The geometrical posture of the shearer is composed of
A parameter drive, wherein,
the angle for driving the rocker arm of the coal mining machine is obtained from rotary encoders arranged at the left rocker arm and the right rocker arm of the coal mining machine;
the angle of the oil cylinder is driven to be increased;
the extension of the movable column of the heightening oil cylinder is driven, and the virtual model of the coal mining machine can be driven by 6 parameters on the left and right sides; embedding the 6 geometric attitude data to realize the visualization of the geometric attitude of the coal mining machine;
the safety state data of the core machine, the electric, the hydraulic and the control components of the coal mining machine can adopt color identification; when the component is in a dangerous and warning state, the brightness of the component is set to flash, and the transparency of the other components is reduced so as to highlight the abnormal component and simultaneously give out warning sound to further warn a user; the equipment warning data is embedded into a physical virtual body of the coal mining machine through color, flicker and sound, so that the visualization of the warning data of key components of the coal mining machine is realized;
step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine monitoring virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; the distance data between the coal mining machine and the monitored coal wall drives the position of the virtual coal wall; driving a scraper model by coal mining machine position data; driving a virtual model of the coal mining machine by using attitude data, geometric attitude data and key component warning data of the coal mining machine;
step four: the method comprises the following steps that device attribute data and operation data of the coal mining machine are displayed on a touch display and are associated with a 3D coal mining machine monitoring virtual model presented by AR devices through a network;
step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine monitoring virtual model in time and space, the posture of the 3D coal mining machine monitoring virtual model is completely consistent with the posture of a coal mining machine in the continuous monitoring video data, and an underground coal mining scene is visually reproduced in real space in real time.
2. The AR-based visualization monitoring method for the coal mining machine according to claim 1, wherein the operation method of the touch module of the visualization monitoring system for the coal mining machine is as follows:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data retrieval: connecting the fully mechanized coal mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized coal mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
3. The AR-based visualization monitoring method for the coal mining machine according to claim 2, wherein the AR module of the visualization monitoring system for the coal mining machine is operated by:
initialization: starting the AR head-mounted display equipment, automatically scanning the working environment around the helmet in the monitoring room by the AR head-mounted display equipment, and carrying out spatial modeling; determining positioning information of a space where the coal mining machine is located by means of an SLAM technology and a space anchor technology of AR equipment; starting an AR monitoring program of a coal mining machine on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR equipment last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, a monitoring video and warning information of a 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time to enable continuous monitoring video data and running data to be in space-time synchronization with the geometric posture of the coal mining machine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539085.0A CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539085.0A CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113411546A CN113411546A (en) | 2021-09-17 |
| CN113411546B true CN113411546B (en) | 2022-11-01 |
Family
ID=77678922
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110539085.0A Active CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113411546B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114282620A (en) * | 2021-12-29 | 2022-04-05 | 汇鲲化鹏(海南)科技有限公司 | Multi-source information physical twin data fusion management method and management system |
| CN115527013B (en) * | 2022-09-06 | 2023-06-27 | 北京龙软科技股份有限公司 | Multi-dimensional visual inspection virtual-real coordination method for mine |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000049475A2 (en) * | 1999-02-17 | 2000-08-24 | Westinghouse Savannah River Company | Systems and methods for interactive virtual reality process control, simulation, and training (ivrpcst) |
| CN103050045A (en) * | 2012-10-01 | 2013-04-17 | 河南理工大学 | Multi-work-type collaborative virtual practical training operation system for coal mine fully mechanized coal face |
| CN103244184A (en) * | 2013-05-07 | 2013-08-14 | 西南石油大学 | Real-time mine pressure monitoring system for downhole fully-mechanized coal mining face |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102759909A (en) * | 2012-06-14 | 2012-10-31 | 中国矿业大学 | Work state virtual simulation system for electric haulage shearer based on different geological conditions |
| CN107066313B (en) * | 2017-05-10 | 2020-08-04 | 太原理工大学 | Fully mechanized coal mining face virtual monitoring method based on local area network cooperation |
| CN109989751B (en) * | 2019-05-06 | 2020-10-16 | 西安科技大学 | Cross-platform remote real-time motion tracking method for fully mechanized three-machine |
| SE543979C2 (en) * | 2019-09-20 | 2021-10-12 | Metso Outotec Finland Oy | Mining Equipment Inspection System, Mining Equipment Inspection Method, and Mining Equipment Inspection Device |
| CN112392485B (en) * | 2020-11-12 | 2021-08-17 | 临沂矿业集团菏泽煤电有限公司 | Transparent digital twin self-adaptive mining system and method for fully mechanized coal mining face |
-
2021
- 2021-05-18 CN CN202110539085.0A patent/CN113411546B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000049475A2 (en) * | 1999-02-17 | 2000-08-24 | Westinghouse Savannah River Company | Systems and methods for interactive virtual reality process control, simulation, and training (ivrpcst) |
| CN103050045A (en) * | 2012-10-01 | 2013-04-17 | 河南理工大学 | Multi-work-type collaborative virtual practical training operation system for coal mine fully mechanized coal face |
| CN103244184A (en) * | 2013-05-07 | 2013-08-14 | 西南石油大学 | Real-time mine pressure monitoring system for downhole fully-mechanized coal mining face |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113411546A (en) | 2021-09-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113411546B (en) | AR-based coal mining machine visual monitoring method | |
| US10157502B2 (en) | Method and apparatus for sharing augmented reality applications to multiple clients | |
| US9639987B2 (en) | Devices, systems, and methods for generating proxy models for an enhanced scene | |
| CN103485784B (en) | Based on coal-winning machine remote control system and the long-range control method of virtual reality technology | |
| CN101939765B (en) | A computer implemented method and system for remote inspection of an industrial process | |
| CN106843124A (en) | A kind of Automatic Warehouse three-dimensional real-time monitoring method and system | |
| CN102759909A (en) | Work state virtual simulation system for electric haulage shearer based on different geological conditions | |
| Xie et al. | Framework for a closed-loop cooperative human Cyber-Physical System for the mining industry driven by VR and AR: MHCPS | |
| CN104268939A (en) | Transformer substation virtual-reality management system based on three-dimensional panoramic view and implementation method of transformer substation virtual-reality management system based on three-dimensional panoramic view | |
| US20110109730A1 (en) | Stereoscopic overlay offset creation and editing | |
| CN105405335A (en) | Remote control virtual training system of fully mechanized coal mining face | |
| CN102945563A (en) | Showing and interacting system and method for panoramic videos | |
| WO2020213426A1 (en) | Image processing device, image processing method, and program | |
| CN111222190B (en) | Ancient building management system | |
| US11164377B2 (en) | Motion-controlled portals in virtual reality | |
| CN113746936B (en) | VR and AR distributed cooperation fully-mechanized coal mining face intelligent monitoring system | |
| US20200077021A1 (en) | Image processing apparatus and method, and program | |
| WO2022121909A1 (en) | Visualized twin factory system controlled in real time | |
| CN114861262A (en) | Visualized large-scale building operation platform system based on Unity3D | |
| CN104345885A (en) | Three-dimensional tracking state indicating method and display device | |
| CN115828620B (en) | Mine disaster emergency simulation method and system based on augmented reality | |
| CN108549479B (en) | Method and system for realizing multi-channel virtual reality and electronic equipment | |
| CN115527013A (en) | Mine multi-dimensional visual inspection virtual-actual cooperative method | |
| JPH09297672A (en) | Monitor screen display system | |
| US11308670B2 (en) | Image processing apparatus and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230827 Address after: 719000 Daliuta town houliuta village, Shenmu City, Yulin City, Shaanxi Province Patentee after: Shaanxi Guangtai Mine Electromechanical Equipment Co.,Ltd. Address before: 451191 Xianghe Road, Longhu Town, Zhengzhou City, Henan Province Patentee before: HENAN INSTITUTE OF ENGINEERING |