CN113411546A - AR-based coal mining machine visual monitoring method - Google Patents
AR-based coal mining machine visual monitoring method Download PDFInfo
- Publication number
- CN113411546A CN113411546A CN202110539085.0A CN202110539085A CN113411546A CN 113411546 A CN113411546 A CN 113411546A CN 202110539085 A CN202110539085 A CN 202110539085A CN 113411546 A CN113411546 A CN 113411546A
- Authority
- CN
- China
- Prior art keywords
- coal mining
- mining machine
- data
- monitoring
- coal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/156—Mixing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
Abstract
The invention provides an AR-based coal mining machine visual monitoring method, which comprises the following steps: firstly, arranging a sensor and an airborne camera on a coal mining machine to acquire operation data of the coal mining machine and monitoring data of a monitored coal wall in a working area; secondly, constructing a 3D coal mining machine monitoring virtual model according to the fully mechanized coal mining face equipment matching drawing; processing the continuous monitoring video data to enable the video data to be consistent with a virtual coal wall object in the 3D coal mining machine monitoring virtual model; then, projecting the processed video data on a corresponding virtual coal wall object, and driving the position and the geometric posture of a monitoring virtual model of the 3D coal mining machine in real time to enable the monitoring virtual model of the 3D coal mining machine to be fused with the monitoring data and the video data in a space-time mode; and finally, reproducing an underground coal mining scene in real time in a real environment by means of an AR technology. By means of AR and visualization technology, the underground mining environment is dynamically and vividly reproduced in real time, and the efficiency and accuracy of monitoring data cognition of monitoring personnel are improved.
Description
Technical Field
The invention relates to the technical field of coal mining, in particular to an AR-based coal mining machine visual monitoring method.
Background
The coal mining environment is complex and dangerous, and the research and development targets of the coal comprehensive mechanized mining are less humanization and unmanned, so that the informatization, digitization and intellectualization are the main directions of the coal mine fully-mechanized mining development. The coal mining machine is used as important equipment for comprehensive mechanized mining, bears key tasks of coal cutting and coal charging, and the operation rate and reliability of the coal mining machine directly influence the mining efficiency and economic benefit of a coal mine, so that the coal mining machine is required to have high reliability. The reliability of the coal mining machine depends on the product quality of the coal mining machine and whether the coal mining machine is operated and maintained correctly in the operation process, and an operator needs to monitor the operation data of the coal mining machine in real time and adjust the operation parameters of the coal mining machine at any time to enable the coal mining machine to be in the optimal working condition, so that the operation efficiency is improved, and the service life is prolonged. The traditional coal mining machine monitoring system is positioned in a crossheading or ground dispatching command center to monitor the operation parameters and the operation environment of the coal mining machine.
The mining environment data monitoring of the coal mining machine adopts a video monitoring mode, namely a plurality of cameras are arranged on a hydraulic support of a fully mechanized mining face at fixed intervals, video acquisition is carried out on the production field environment around the coal mining machine, then the video acquisition is transmitted to a control center, along with the advance of the coal mining machine, a monitoring system needs to continuously switch the monitoring cameras to realize the follow-up monitoring, and the operation monitoring data and the geometric posture of the coal mining machine are displayed by a 2D display.
The shearer monitoring data includes 3D physical data (shearer position, geometric attitude data, relative coal wall position data, etc.) and 2D abstract data (operational data, surveillance video, etc.). In a traditional coal mining machine monitoring system, 3D physical graph data are converted into 2D graphs through multiple projection transformation and are displayed with 2D abstract data in parallel, monitoring personnel need to reversely convert the 2D graphs into the 3D physical graph data in real time in the brain and then associate the 3D physical graph data with the 2D abstract data, so that the problems that the monitoring personnel are heavy in thinking burden and the data association is easy to be disordered are caused, and further, decision making is slow or even misjudgment is caused. Therefore, the research of the visualization monitoring system based on 3D space display and multi-type data fusion by utilizing the augmented reality and data visualization technology is urgent and necessary.
The development direction of the coal mine automatic mining technology is less humanization, digitalization and intellectualization, and the mainstream fully-mechanized mining working face system adopts an automatic equipment production and remote operator intervention mode. The coal mining machine is an important device of the fully mechanized coal mining face, and monitoring personnel need to observe the running state, mining process parameters and surrounding mining environment in real time. The existing remote monitoring system of the coal mining machine adopts a plurality of plane displays to display monitoring data in parallel, and mainly has the following problems:
1) the geometric pose data of the coal mining machine are not intuitive
The existing coal mining machine monitoring system has two types, namely a 2D monitoring system and a VR-based transparent working surface system, and has the problem that the geometric pose data of the coal mining machine is not directly observed.
The 2D monitoring system is used for projecting and converting 3D geometric figure data of the coal mining machine into a 2D projection figure with a fixed visual angle, the projection figure is displayed in a split screen mode through multiple display screens, monitoring personnel can only observe the pose of the coal mining machine from a plurality of fixed visual angles, and the display effect is not visual.
The transparent working face system is developed based on the Internet of things, VR and digital twin technology, a three-machine 3D virtual model of the fully mechanized working face is constructed, the fully mechanized working face is reproduced in a 3D space created by a computer, and monitoring data are displayed by multiple 2D displays in a split screen mode. The transparent working face system has the advantage that monitoring personnel can observe the geometric pose and mining environment data of the coal mining machine from any visual angle. The shortcomings are that the 3D geometric data of the coal mining machine can only be projected and transformed into 2D graphs to be presented on the display due to the adoption of the 2D display as the display output device, and the 3D model is difficult to be manipulated in the 2D display space.
2) The monitoring video is not associated with the geometric pose data of the coal mining machine in time and space
The traditional coal mining machine mining environment monitoring system consists of a camera, a transmission network, video processing and data storage, and is shown in figure 1. The cameras are arranged under the top beam of the hydraulic support, and 1 camera is arranged at intervals. The monitoring system starts the corresponding camera according to the position of the coal mining machine, and automatically switches the camera along with the movement of the coal mining machine, so that the coal mining of the coal mining machine is continuously tracked and monitored in the whole process. In the reciprocating coal mining process of the coal mining machine, the monitoring video of the coal mining machine obtained by alternately fixing the machine position cameras is discontinuous, the monitoring video is mostly shot at an inclined angle, and a certain deformation exists after the monitoring video picture is changed through projection; the geometric poses of the monitoring video pictures and the 3D virtual model of the coal mining machine are not matched in spatial position and size, and the mining scene of the coal mining machine cannot be presented in a time-space consistent mode. Therefore, monitoring personnel can not visually judge whether the working state of the coal mining machine and the surrounding production environment meet the requirements of safe production.
3) 3D model and 2D monitoring data association difficulty of coal mining machine
A traditional coal mining machine visual monitoring system adopts a plurality of 2D flat panel displays to display monitoring data in a split screen mode, and a 3D virtual model needs to be projected and transformed into a 2D image to be displayed with the 2D monitoring data in parallel. The 3D geometric model of the coal mining machine is difficult to associate with the 2D monitoring data after being transformed by forward and backward projection, and the abstract metadata is difficult to directly associate with the representative equipment object in a 3D space, so that the problems of heavy thinking burden, disordered data association and the like of monitoring personnel are caused, and slow decision and even misjudgment are caused.
The external mining environment data of the coal mining machine mainly comprises monitoring videos of the peripheral coal wall and equipment of the coal mining machine, the traditional working face mining environment monitoring scheme is shown in figure 1, along with uninterrupted reciprocating coal mining of the coal mining machine, a monitoring video near the coal mining machine is obtained by alternately starting fixed machine position cameras, the monitoring video is discontinuous, is not matched with the geometric shape, size and geometric pose data of the coal mining machine in space, cannot be associated with 3D physical graphic data and abstract monitoring data of the coal mining machine, the intuitiveness of the monitoring data is poor, and the information acquisition efficiency is low.
Disclosure of Invention
Aiming at the defects in the background technology, the invention provides an AR-based coal mining machine visual monitoring method, which takes monitoring data as a research object, and by means of AR and a visual technology, solves the problems that the 3D geometric data in the traditional coal mining machine monitoring system is difficult to be associated with 2D abstract data after projection transformation, the cross-space-time association between metadata and the corresponding 3D object is difficult to realize, and the like, improves the cognitive efficiency of monitoring personnel on the monitoring data, and ensures timely and reliable decision.
The technical scheme of the invention is realized as follows:
an AR-based coal mining machine visual monitoring method comprises the following steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to acquire video monitoring data; acquiring the geometric pose and working data of the equipment by using an airborne sensor; transmitting the continuous monitoring video data, the equipment geometric pose and the operation data to a fully mechanized coal mining face monitoring system;
step two: visually presenting a 3D coal mining machine virtual model in a real environment by using AR equipment, wherein the virtual model comprises a coal mining machine, a scraper and a monitored coal wall;
step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; the distance data between the coal mining machine and the monitored coal wall drives the position of the virtual coal wall; the position data of the coal mining machine drives a scraper conveyor model; the coal mining machine model is driven by the coal mining machine posture, the geometric posture and the warning information;
step four: the attribute data and the operation data of the coal mining machine equipment are displayed on a touch display and are associated with a 3D coal mining machine virtual model presented by the AR equipment through a network;
step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine virtual model in time and space, the posture of the 3D coal mining machine virtual model is completely consistent with that of a coal mining machine in the monitoring video, and an underground coal mining scene is intuitively reproduced in real space in real time.
The video monitoring data comprises embedded abstract data, and the embedded abstract data comprises coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring video data and coal mining machine position data; the 3D coal mining machine monitoring virtual model is embedded and driven by coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring data and coal mining machine position data.
The geometric pose and the working data of the equipment are associated abstract data, and the geometric pose and the working data of the equipment are associated with the monitoring virtual model of the 3D coal mining machine.
The 3D coal mining machine monitoring virtual model comprises a coal mining machine position virtual model, a monitored coal wall virtual model and a coal mining machine virtual model; embedding the position data of the coal mining machine into the virtual model of the position of the coal mining machine; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding the attitude data, the geometric attitude data and the warning data of key components of the coal mining machine into a virtual model of the coal mining machine; the virtual model of the coal mining machine is respectively associated with the geometric pose of the equipment and the working data; the virtual model of the position of the coal mining machine is a scraper model.
The operation method of the touch module of the coal mining machine visual monitoring system comprises the following steps:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data retrieval: connecting the fully mechanized coal mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized coal mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
The operation method of the AR module of the visualized monitoring system of the coal mining machine comprises the following steps:
initialization: starting the AR head-mounted display equipment, automatically scanning and monitoring the working environment around the indoor helmet by the equipment, and performing spatial modeling; determining positioning information of a space where the coal mining machine equipment is located by means of an SLAM technology and a space anchor technology of AR equipment; starting an AR monitoring program of a coal mining machine on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR equipment last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, a monitoring video and warning information of a 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time to enable the monitoring video and the operation data of the coal mining machine to be in space-time synchronization with the geometric posture of the coal mining machine.
Compared with the prior art, the invention has the following beneficial effects:
1) the AR-based visualization monitoring technology for the coal mining machine takes AR, visualization and man-machine interaction technologies as a core, combines multidisciplinary technologies such as 3D virtual model construction, video monitoring and digital twinning, fuses and associates 3D physical graphic data and 2D abstract monitoring data of the coal mining machine in a 3D space, dynamically reproduces underground mining scenes of the coal mining machine in real time, and efficiently and visually presents running data and running states of the coal mining machine;
2) the invention assists the existing fully mechanized coal mining face remote monitoring system, realizes remote virtual visual mining, provides technical support for less people and intelligent production of coal mine comprehensive mechanical mining, and has important guiding significance and reference value for digital operation and maintenance and intelligent mining of the coal mining machine.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a block diagram of a mining environment monitoring system of a conventional coal mining machine.
Fig. 2 is a three-machine matching diagram of the fully mechanized mining face.
Fig. 3 is a real view of the fully mechanized coal mining face.
Fig. 4 is an external mining environment monitoring model of a coal mining machine of the present invention.
Fig. 5 is a schematic diagram of 2D object projection mapping for 2D image.
FIG. 6 is a flow chart of the present invention.
FIG. 7 is a coal mining machine monitoring video mapping model established by the present invention.
Fig. 8 is a simplified model of electromechanical components of a coal mining machine constructed in accordance with the present invention.
Fig. 9 is a position coordinate system of the coal mining machine established by the present invention.
Fig. 10 is a 3D digital model of a scraper conveyor built by the present invention.
FIG. 11 is a shearer position visualization model.
FIG. 12 is a block diagram of a fully mechanized coal mining face process simulation system of the present invention.
FIG. 13 shows a drawing C of the present invention1A camera position schematic; wherein (a) is C1A front view of the camera, and (b) is C1A side view of the camera.
FIG. 14 is a view of the onboard camera of the present invention; wherein (a) is C1The camera shoots pictures, and (b) is C2The camera shoots pictures, and (C) is C3The camera takes a picture of the scene,(d) is C4The camera takes pictures.
Fig. 15 is a schematic diagram of the fusion of the surveillance video data and the physical three-dimensional data according to the present invention.
Figure 16 is the main menu of the AR device.
Fig. 17 is a schematic diagram of the operation of moving the position of the shearer scene model according to the present invention.
Fig. 18 is a schematic view of the rotating operation of the shearer scene model of the present invention.
Fig. 19 is an example of the operation of the immersive human-computer interaction module of the present invention.
Fig. 20 shows an example of the operation of the visualization monitoring system of the coal mining machine according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
The fully mechanized mining face is enclosed by a coal mining machine, a hydraulic support, a scraper conveyor and a coal wall to form a safe mining space, the working environment is severe, and the space is narrow, as shown in fig. 2 and 3. The coal mining machine carries out reciprocating mining along the coal wall, the working space is limited and closed, and the relative position parameters of the monitored coal wall (which can be approximate to a 2D plane) and the coal mining machine can be solved by a three-machine matching diagram of the fully mechanized mining face and a remote control system. The external mining environment of the coal mining machine is visualized by adopting an airborne camera monitoring scheme. Arranging a plurality of onboard monitoring cameras on a coal mining machine body, wherein the shooting direction of the cameras is vertical to the coal wall, continuously shooting the coal wall and related equipment around the coal mining machine, and monitoring the external mining environment of the coal mining machine, as shown in figure 4, CnBeing an onboard camera, PnFor monitoring video sequences, PnFor coal wall pictures of the monitored working area, Pn ′Is a corresponding virtual coal wall object.
The embodiment of the invention provides an AR-based coal mining machine visual monitoring method, which comprises the following specific steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to acquire continuous monitoring video data; acquiring the geometric pose and working data of the equipment by using an airborne sensor; transmitting the continuous monitoring video data, the equipment geometric pose and the operation data to a fully mechanized coal mining face monitoring system; the video monitoring data comprises embedded abstract data, and the embedded abstract data comprises coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring video data and coal mining machine position data; the 3D coal mining machine monitoring virtual model is embedded and driven by coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring data and coal mining machine position data. The geometric pose and the working data of the equipment are associated abstract data, and the geometric pose and the working data of the equipment are associated with the monitoring virtual model of the 3D coal mining machine.
The coal mining machine attitude data comprises a heading angle alpha, a pitch angle beta and a roll angle gamma of the machine body relative to the absolute coordinates of the earth. The three rotation angle parameters of the 3 data-driven coal mining machine virtual models can embed the machine body attitude data into the coal mining machine virtual models, and visualization of the machine body attitude data is achieved.
The geometric pose data comprises the rotation angle theta of the rocker arm and the rotation angle theta of the heightening oil cylinder4And the telescopic distance delta Y of the movable column of the heightening oil cylinder; the geometric attitude of the coal mining machine is composed of theta and theta4Driving the delta Y parameter, wherein theta drives the angle of the rocker arm of the coal mining machine, and the numerical value of theta is from rotary encoders arranged at the left rocker arm and the right rocker arm of the coal mining machine; theta4Driving to increase the angle of the oil cylinder; and the delta Y drives the extending amount of the movable column of the heightening oil cylinder, and the coal mining machine virtual model can be driven by the left and right 6 parameters. And embedding the 6 geometrical attitude data to realize the visualization of the geometrical attitude of the coal mining machine.
The safety state data of the core machine, the electric, the hydraulic and the control components of the coal mining machine can adopt color identification. According to the standard "GB 2893-2008 security color", the colors that convey security information are red (dangerous, prohibited), blue (instruction), yellow (attention, warning) and green (security). The corresponding safety status data (danger, warning, safety) is expressed by changing the color of the component (red, yellow, green), as shown in fig. 7. When the component is in a dangerous and warning state, the brightness of the component is set to be flashing, and the transparency of the rest components is reduced so as to highlight the abnormal component and simultaneously give out warning sound to further warn a user. The equipment warning data is embedded into the physical virtual body of the coal mining machine through color, flicker and sound, so that the visualization of the warning data of key components of the coal mining machine is realized.
An absolute encoder is arranged on a low-speed shaft of a traction part of a traveling mechanism of the coal mining machine, and the absolute coordinate of the coal mining machine on a scraper conveyor can be solved by measuring the number of turns of the gear. A 2D reference coordinate system shown in fig. 9 is established, with the coal wall facing as the X coordinate direction, the head of the scraper conveyor as the origin of coordinates, and the length direction of the scraper conveyor as the Y coordinate direction. The method comprises the steps of obtaining transition coordinate data of a scraper groove from an existing fully mechanized mining face monitoring system, namely an X coordinate of the scraper groove, and obtaining a spatial distribution shape of the scraper groove in a two-dimensional coordinate system, wherein a Y coordinate of the scraper groove can be approximate to a constant. A3D digital model of the scraper conveyor is built by replacing a scraper groove with a complicated shape with a cuboid, and is shown in figure 10. Neglecting the rotation angle of the scraper trough, positioning the scraper trough by pushing X coordinate data, and driving a 3D virtual model of the scraper conveyor to obtain a 3D virtual model similar to the actual geometric attitude. Due to the fact that the pushing size of the scraper groove is too small compared with the length size of the whole machine, the geometric form of a 3D virtual model in the monitoring system is not visual, and the pushing size of the scraper conveyor is enlarged by 3 times to be displayed. Total length of coal mining machine is L1Length L of scraper trough of scraper conveyor2Taking n as int (L)1/L2) The block scraper trough represents the position of the shearer. Taking a scraper on which left and right symmetrical surfaces of the coal mining machine are located as a reference, and taking n as int (L) on the left and right sides respectively1/L2And/2) modifying the color of the scraper to be yellow, setting continuous flashing, and visually presenting the position data of the coal mining machine, as shown in FIG. 11. The appearance of the scraper virtual model in the direction to be mined of the coal mining machine is set to be green and represents the running direction of the coal mining machine; scraper virtualization of the mined side of a coal mining machineThe model appearance is set to red; and refreshing the color of the corresponding scraper virtual model along with the continuous change of the position of the coal mining machine. Embedding the position data and mining direction data of the coal mining machine into the 3D position model in a color form, driving the X coordinate of the 3D virtual model of the scraper blade by using the pushing data, and changing the position of the scraper blade so as to visualize the position of the coal mining machine in an intuitive mode.
A coal mining machine monitoring video mapping model shown in the figure 8 is established, based on the SAR technology, a plurality of airborne cameras are adopted to shoot a monitored coal wall virtual model, 2D monitoring videos are amplified and cut in equal proportion, and are aligned and superposed to a plane where a corresponding coal wall is located through space registration and registration, so that a highly real and three-dimensional dynamic coal mining scene can be generated in a virtual-real fusion space created by AR. Amplifying and cutting the monitoring image according to a certain size proportion, respectively mapping the monitoring image to the monitored coal wall planes of the coal mining machine corresponding to the graph 7, and calculating to obtain the distance L between the 4 monitored coal wall planes and the coal mining machinenVideo amplification ratio lambdanVideo cropping size AnAnd BnThe plane of the monitored coal wall can be driven, and the coal mining working scene of the coal mining machine can be reproduced in real time. The monitoring video is embedded into the corresponding monitored coal wall virtual model, the size, the position and the video amplification scale of the coal wall virtual model are driven by related parameters, and the visualization of the mining environment monitoring data of the coal mining machine is realized.
Step two: visually presenting a 3D coal mining machine virtual model in a real environment by using AR equipment, wherein the virtual model comprises a coal mining machine, a scraper and a monitored coal wall; the 3D coal mining machine monitoring virtual model comprises a coal mining machine position virtual model (or a scraper machine model), a monitored coal wall virtual model and a coal mining machine virtual model; embedding the position data of the coal mining machine into the virtual model of the position of the coal mining machine; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding the attitude data, the geometric attitude data and the warning data of key components of the coal mining machine into a virtual model of the coal mining machine; the virtual model of the coal mining machine is respectively associated with the geometric pose of the equipment and the working data.
Step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; the distance data between the coal mining machine and the monitored coal wall drives the position of the virtual coal wall; the position data of the coal mining machine drives a scraper conveyor model; the coal mining machine model is driven by the coal mining machine posture, the geometric posture and the warning information; the coal mining environment monitoring model monitors the coal wall and related equipment around the coal mining machine through the onboard camera, and in a 3D space created by the AR equipment, the onboard monitoring video is accurately superposed on a corresponding coal wall object of the 3D model of the coal mining machine through projection transformation and space registration technology, so that a 3D coal mining scene with high reality and three-dimensional sense can be generated. When the monitored coal wall virtual model is a 2D object, the plane where the monitored coal wall virtual model is located is parallel to the imaging plane of the monitoring camera, the size of the monitoring video shot by the airborne camera and the shot 2D plane object are in an equal proportional amplification relation, and the proportionality coefficient is k. After the 2D monitoring video is cut by scaling in equal proportion, the 2D monitoring video is registered with the 2D virtual coal wall object space in the AR space, that is, the 2D monitoring video is registered and aligned to the 2D virtual coal wall object, so that real-time dynamic space-time fusion of the monitoring data of the coal mining machine and the virtual digital model of the coal mining machine is realized, as shown in fig. 5.
Step four: and displaying the attribute data and the operation data of the coal mining machine on the touch display, and associating the attribute data and the operation data with the 3D coal mining machine virtual model presented by the AR equipment through a network.
Step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine virtual model in time and space, the posture of the 3D coal mining machine virtual model is completely consistent with that of a coal mining machine in the monitoring video, and an underground coal mining scene is intuitively reproduced in real space in real time.
The coal mining machine visual monitoring system is based on AR and visualization technology, acquires monitoring data from the existing fully mechanized coal face monitoring system, and associates and fuses the abstract monitoring data of the coal mining machine and the physical virtual model with the image through 3D visualization technology. The visual monitoring system of coal-winning machine based on AR is as auxiliary system, satisfies the demand that monitoring personnel directly perceived, high-efficient monitoring coal-winning machine operating condition. The coal mining machine visual monitoring system is composed of an AR module and a flat touch man-machine interaction module, the AR module uses AR equipment to visually display in a 3D mode, the flat touch man-machine interaction module uses a traditional touch display to display, and the two modules are in data association through a network. The flow of the visualization monitoring system of the AR-based coal mining machine is shown in FIG. 6.
As shown in fig. 6, the design process of the flat touch-control human-computer interaction module (coal mining machine visual monitoring system) is composed of initialization, data retrieval, and abstract data visualization 3 parts:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data retrieval: connecting the fully mechanized coal mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized coal mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
As shown in fig. 6, the AR module (coal mining machine monitoring virtual model) design process is composed of three parts, namely initialization, 3D virtual model loading and model visualization driving:
initialization: starting the AR equipment, automatically scanning and monitoring the working environment around the indoor helmet by the equipment, and performing spatial modeling; determining positioning information of a space where the coal mining machine equipment is located by means of an SLAM technology and a space anchor technology of AR equipment; starting an AR program on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR monitoring program of the coal mining machine last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, monitoring video and warning information of the 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time.
There are two modes of embedding and associating 3D-like physical virtual objects and 2D abstract data of the shearer, as shown in fig. 6. Direct contact is established between the two data, the thinking burden of monitoring personnel can be reduced, the decision-making efficiency is improved, and the data visualization effect is excellent. When the panel touch interface selects certain abstract data, the panel touch interface highlights the data, and meanwhile, the AR human-computer interaction interface automatically highlights the 3D virtual model of the corresponding device. When the AR human-computer interaction interface selects an image 3D virtual model of a certain device, the AR interface highlights the 3D virtual model, meanwhile, the flat touch type interaction interface automatically highlights corresponding abstract monitoring data, and the contact between the two is directly and visually prompted.
System test and run case
By taking a certain type of coal mining machine as an example, a coal mining machine visual monitoring system is designed, and visual fusion effects like 3D physical virtual bodies and 2D abstract data are verified. The machine-mounted camera monitoring video of the coal mining machine is acquired by a fully mechanized coal mining face process simulation system. The fully mechanized coal mining face process simulation system consists of a coal mining machine, a scraper conveyor, a hydraulic support and a coal wall, and is used for simulating and simulating a fully mechanized coal mining face end beveling feed coal mining process, as shown in fig. 12.
Referring to the external mining environment monitoring model of the coal mining machine shown in fig. 4, 4 virtual cameras C are arranged at corresponding positions of the coal mining machine in the fully mechanized mining face process simulation system1～C4In which C is1The camera position is shown in figure 13. And adjusting relevant parameters of the virtual camera in the system until an expected monitoring video picture is obtained.
Starting the fully mechanized mining face process simulation system, and recording the videos by the 4 cameras in turn to respectively obtain the monitoring videos of the airborne cameras shown in the figure 14.
After the airborne monitoring video picture is scaled and cut in an equal proportion, the picture is mapped to a coal wall object corresponding to the environmental scene model of the coal mining machine, and the visual effect of the monitoring video is shown in fig. 15.
The abstract 2D monitoring data of the coal mining machine is obtained from an existing coal mining machine monitoring system, the position and posture data of the coal mining machine is obtained from a fully mechanized mining face process simulation system, and the monitoring data is arranged into a video file and an Excel table and stored to a specified position for later use.
The coal mining machine visual monitoring system based on the AR is composed of an AR human-computer interaction module and a flat touch control type human-computer interaction module, and the two modules are respectively started. And reading the monitoring data of the coal mining machine from the designated position, and driving the respective modules.
The AR man-machine interaction module of the coal mining machine visual monitoring system is operated by AR equipment and directly presented in a 3D space, and the operation flow of the system is as follows:
1) starting AR man-machine interaction module
The augmented reality helmet is worn and started, the system main menu is opened, and the shearer AR monitoring program is selected, as shown in fig. 16.
2) Positioning of the coal mining machine monitoring scene model is performed by using gestures of AR equipment, the 3D coal mining machine monitoring virtual model is positioned to the specified position of the workbench, the scene model moving operation is shown in figure 17, and the scene model rotating operation is shown in figure 18. Note: the system uses the space anchor technology of the AR equipment, the program can automatically memorize the position of the model when exiting, and when the program is started for the second time, if the space is not changed, the secondary positioning is not needed.
3) The data of the monitoring scene model of the coal mining machine drives an AR man-machine interaction module of the visualized monitoring system of the coal mining machine to call preset simulation data and simulation monitoring videos, and the operation effect of the system is shown in figure 19.
4) AR interaction module and flat-panel touch module associated operation
The visual monitoring system of the coal mining machine based on the AR is composed of a flat touch control type module and an AR module, data association and data exchange are carried out on the two modules through a network, and the effect of the two modules in cooperation and parallel operation is shown in figure 20.
The system test data is acquired by the fully mechanized mining face virtual simulation system, the onboard monitoring video of the coal mining machine and the operation data of the coal mining machine are simulated, and the system test and operation are carried out by taking the coal mining machine as a monitoring example. The 3D aspect monitoring data and the 2D abstract monitoring data of the coal mining machine are visually and dynamically displayed, seamless dynamic fusion, interconnection and intercommunication are achieved, operation is convenient and fast, and the expected effect is achieved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (6)
1. A coal mining machine visual monitoring method based on AR is characterized by comprising the following steps:
the method comprises the following steps: arranging an airborne camera and an airborne sensor on a coal mining machine, and continuously shooting the coal wall of a monitored working area by using the airborne camera to acquire video monitoring data; acquiring the geometric pose and working data of the equipment by using an airborne sensor; transmitting the continuous monitoring video data, the equipment geometric pose and the operation data to a fully mechanized coal mining face monitoring system;
step two: visually presenting a 3D coal mining machine virtual model in a real environment by using AR equipment, wherein the virtual model comprises a coal mining machine, a scraper and a monitored coal wall;
step three: acquiring monitoring data from a fully mechanized coal mining face monitoring system, and driving an AR device to visually present a 3D coal mining machine virtual model, wherein a monitoring video is projected on a virtual coal wall object after being processed; the distance data between the coal mining machine and the monitored coal wall drives the position of the virtual coal wall; the position data of the coal mining machine drives a scraper conveyor model; the coal mining machine model is driven by the coal mining machine posture, the geometric posture and the warning information;
step four: the attribute data and the operation data of the coal mining machine equipment are displayed on a touch display and are associated with a 3D coal mining machine virtual model presented by the AR equipment through a network;
step five: by means of the AR technology, the continuous monitoring video data and the operation data are perfectly fused with the 3D coal mining machine virtual model in time and space, the posture of the 3D coal mining machine virtual model is completely consistent with that of a coal mining machine in the monitoring video, and an underground coal mining scene is intuitively reproduced in real space in real time.
2. The AR-based visual monitoring method of a shearer as recited in claim 1, wherein the video monitoring data includes embedded abstract data including shearer attitude data, geometric attitude data, critical component warning data, mining zone location data, mining zone monitoring video data, and shearer location data; the 3D coal mining machine monitoring virtual model is embedded and driven by coal mining machine attitude data, geometric attitude data, key component warning data, mining area position data, mining area monitoring data and coal mining machine position data.
3. The AR-based visualization monitoring method for the coal mining machine according to claim 2, wherein the device geometric pose and the working data are associated abstract data, and the device geometric pose and the working data are associated with the 3D coal mining machine monitoring virtual model.
4. The AR-based shearer visual monitoring method according to claim 2 or 3, wherein the 3D shearer monitoring virtual model comprises a shearer location virtual model, a monitored coal wall virtual model and a shearer virtual model; embedding the position data of the coal mining machine into the virtual model of the position of the coal mining machine; embedding the mining area position data and the mining area monitoring video data into the monitored coal wall virtual model; embedding the attitude data, the geometric attitude data and the warning data of key components of the coal mining machine into a virtual model of the coal mining machine; the virtual model of the coal mining machine is respectively associated with the geometric pose of the equipment and the working data; the virtual model of the position of the coal mining machine is a scraper model.
5. The AR-based visualization monitoring method for the coal mining machine according to claim 4, wherein the operation method of the touch control module of the visualization monitoring system for the coal mining machine is as follows:
initialization: opening a flat touch control computer, starting a visual monitoring system of the coal mining machine, and connecting a network system;
data retrieval: connecting the fully mechanized coal mining face monitoring system, extracting required embedded abstract data from a database of the fully mechanized coal mining face monitoring system, and processing the embedded abstract data into a data form called by a virtual module;
visualization of abstract data: according to the visualization requirement, the associated abstract data and the embedded abstract data are distributed on the touch display in the form of a chart, data and a curve.
6. The AR-based visualization monitoring method for a coal mining machine according to claim 5, wherein the AR module of the visualization monitoring system for a coal mining machine is operated by:
initialization: starting the AR head-mounted display equipment, automatically scanning and monitoring the working environment around the indoor helmet by the equipment, and performing spatial modeling; determining positioning information of a space where the coal mining machine equipment is located by means of an SLAM technology and a space anchor technology of AR equipment; starting an AR monitoring program of a coal mining machine on a main menu of the AR equipment;
3D virtual model loading: when the AR monitoring program of the coal mining machine is started for the first time, loading a 3D coal mining machine monitoring virtual model, and positioning the 3D coal mining machine monitoring virtual model to a workbench by an operator; when the AR monitoring program of the coal mining machine is started again, if the environmental space information of the AR equipment is the same, repositioning the 3D coal mining machine monitoring virtual model according to the space anchor recorded by the AR equipment last time;
and (3) model visualization driving: the method comprises the steps of acquiring embedded abstract data from a visual monitoring system of the coal mining machine, driving the machine body posture, the geometric posture of a moving part, the position of the coal mining machine, parameters of a monitored coal wall, a monitoring video and warning information of a 3D coal mining machine monitoring virtual model in real time, and dynamically and visually driving the 3D coal mining machine monitoring virtual model in real time to enable the monitoring video and the operation data of the coal mining machine to be in space-time synchronization with the geometric posture of the coal mining machine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539085.0A CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110539085.0A CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113411546A true CN113411546A (en) | 2021-09-17 |
| CN113411546B CN113411546B (en) | 2022-11-01 |
Family
ID=77678922
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110539085.0A Active CN113411546B (en) | 2021-05-18 | 2021-05-18 | AR-based coal mining machine visual monitoring method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113411546B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114282620A (en) * | 2021-12-29 | 2022-04-05 | 汇鲲化鹏(海南)科技有限公司 | Multi-source information physical twin data fusion management method and management system |
| CN115527013A (en) * | 2022-09-06 | 2022-12-27 | 北京龙软科技股份有限公司 | Mine multi-dimensional visual inspection virtual-actual cooperative method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000049475A2 (en) * | 1999-02-17 | 2000-08-24 | Westinghouse Savannah River Company | Systems and methods for interactive virtual reality process control, simulation, and training (ivrpcst) |
| CN102759909A (en) * | 2012-06-14 | 2012-10-31 | 中国矿业大学 | Work state virtual simulation system for electric haulage shearer based on different geological conditions |
| CN103050045A (en) * | 2012-10-01 | 2013-04-17 | 河南理工大学 | Multi-work-type collaborative virtual practical training operation system for coal mine fully mechanized coal face |
| CN103244184A (en) * | 2013-05-07 | 2013-08-14 | 西南石油大学 | Real-time mine pressure monitoring system for downhole fully-mechanized coal mining face |
| CN107066313A (en) * | 2017-05-10 | 2017-08-18 | 太原理工大学 | A kind of fully-mechanized mining working virtual monitoring method cooperateed with based on LAN |
| CN109989751A (en) * | 2019-05-06 | 2019-07-09 | 西安科技大学 | A kind of cross-platform long-range real time kinematics tracking system of three machine of fully mechanized mining and method |
| CN112392485A (en) * | 2020-11-12 | 2021-02-23 | 临沂矿业集团菏泽煤电有限公司 | Transparent digital twin self-adaptive mining system and method for fully mechanized coal mining face |
| SE1951064A1 (en) * | 2019-09-20 | 2021-03-21 | Metso Sweden Ab | Mining Equipment Inspection System, Mining Equipment Inspection Method, and Mining Equipment Inspection Device |
-
2021
- 2021-05-18 CN CN202110539085.0A patent/CN113411546B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000049475A2 (en) * | 1999-02-17 | 2000-08-24 | Westinghouse Savannah River Company | Systems and methods for interactive virtual reality process control, simulation, and training (ivrpcst) |
| CN102759909A (en) * | 2012-06-14 | 2012-10-31 | 中国矿业大学 | Work state virtual simulation system for electric haulage shearer based on different geological conditions |
| CN103050045A (en) * | 2012-10-01 | 2013-04-17 | 河南理工大学 | Multi-work-type collaborative virtual practical training operation system for coal mine fully mechanized coal face |
| CN103244184A (en) * | 2013-05-07 | 2013-08-14 | 西南石油大学 | Real-time mine pressure monitoring system for downhole fully-mechanized coal mining face |
| CN107066313A (en) * | 2017-05-10 | 2017-08-18 | 太原理工大学 | A kind of fully-mechanized mining working virtual monitoring method cooperateed with based on LAN |
| CN109989751A (en) * | 2019-05-06 | 2019-07-09 | 西安科技大学 | A kind of cross-platform long-range real time kinematics tracking system of three machine of fully mechanized mining and method |
| SE1951064A1 (en) * | 2019-09-20 | 2021-03-21 | Metso Sweden Ab | Mining Equipment Inspection System, Mining Equipment Inspection Method, and Mining Equipment Inspection Device |
| CN112392485A (en) * | 2020-11-12 | 2021-02-23 | 临沂矿业集团菏泽煤电有限公司 | Transparent digital twin self-adaptive mining system and method for fully mechanized coal mining face |
Non-Patent Citations (5)
| Title |
|---|
| AHMAD KALHOR等: "Online modeling of real-world time series through evolving AR models", 《 2012 IEEE INTERNATIONAL CONFERENCE ON FUZZY SYSTEMS》 * |
| GREGORY KRAMIDA: "Resolving the Vergence-Accommodation Conflict in Head-Mounted Displays", 《IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS》 * |
| 季晓刚等: "基于XNA的采煤机3DVR远程监控系统研究", 《自动化与仪表》 * |
| 李万高等: "煤矿安全生产网络视频监控系统的设计与实现", 《煤炭技术》 * |
| 葛世荣等: "互联网+采煤机智能化关键技术研究", 《煤炭科学技术》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114282620A (en) * | 2021-12-29 | 2022-04-05 | 汇鲲化鹏(海南)科技有限公司 | Multi-source information physical twin data fusion management method and management system |
| CN115527013A (en) * | 2022-09-06 | 2022-12-27 | 北京龙软科技股份有限公司 | Mine multi-dimensional visual inspection virtual-actual cooperative method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113411546B (en) | 2022-11-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113411546B (en) | AR-based coal mining machine visual monitoring method | |
| US20220398844A1 (en) | Shared Augmented Reality Session Recording and Playback | |
| US10157502B2 (en) | Method and apparatus for sharing augmented reality applications to multiple clients | |
| US10591880B2 (en) | Method for repairing with 3D printing | |
| CN103485784B (en) | Based on coal-winning machine remote control system and the long-range control method of virtual reality technology | |
| CN101939765B (en) | A computer implemented method and system for remote inspection of an industrial process | |
| CN102497568B (en) | Interactive three-dimensional virtual scene and visualized information integrating system | |
| Xie et al. | Framework for a closed-loop cooperative human Cyber-Physical System for the mining industry driven by VR and AR: MHCPS | |
| CN102759909A (en) | Work state virtual simulation system for electric haulage shearer based on different geological conditions | |
| CN106843124A (en) | A kind of Automatic Warehouse three-dimensional real-time monitoring method and system | |
| CN104268939A (en) | Transformer substation virtual-reality management system based on three-dimensional panoramic view and implementation method of transformer substation virtual-reality management system based on three-dimensional panoramic view | |
| CN105405335A (en) | Remote control virtual training system of fully mechanized coal mining face | |
| CN111222190B (en) | Ancient building management system | |
| CN102945563A (en) | Showing and interacting system and method for panoramic videos | |
| CN109989751B (en) | Cross-platform remote real-time motion tracking method for fully mechanized three-machine | |
| CN109268010A (en) | A kind of long-range inspection interference method of virtual reality Mechanized | |
| CN113746936B (en) | VR and AR distributed cooperation fully-mechanized coal mining face intelligent monitoring system | |
| US20200077021A1 (en) | Image processing apparatus and method, and program | |
| WO2022121909A1 (en) | Visualized twin factory system controlled in real time | |
| CN114861262A (en) | Visualized large-scale building operation platform system based on Unity3D | |
| CN104345885A (en) | Three-dimensional tracking state indicating method and display device | |
| CN115527013A (en) | Mine multi-dimensional visual inspection virtual-actual cooperative method | |
| JPH09297672A (en) | Monitor screen display system | |
| CN113840008A (en) | Coal mining machine immersion type remote control device, coal mining machine remote control system and method | |
| Sommer et al. | View-dependent Virtual and Augmented Reality for Machine Tools |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230827 Address after: 719000 Daliuta town houliuta village, Shenmu City, Yulin City, Shaanxi Province Patentee after: Shaanxi Guangtai Mine Electromechanical Equipment Co.,Ltd. Address before: 451191 Xianghe Road, Longhu Town, Zhengzhou City, Henan Province Patentee before: HENAN INSTITUTE OF ENGINEERING |