CN112163986A - Distributed processing method for underground logging and mining three-dimensional data of metal mine - Google Patents
Distributed processing method for underground logging and mining three-dimensional data of metal mine Download PDFInfo
- Publication number
- CN112163986A CN112163986A CN202011040169.1A CN202011040169A CN112163986A CN 112163986 A CN112163986 A CN 112163986A CN 202011040169 A CN202011040169 A CN 202011040169A CN 112163986 A CN112163986 A CN 112163986A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- dimensional data
- gpu
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 239000002184 metal Substances 0.000 title claims abstract description 12
- 238000005065 mining Methods 0.000 title claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000009877 rendering Methods 0.000 claims abstract description 15
- 238000003860 storage Methods 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 9
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 5
- 238000012805 post-processing Methods 0.000 claims abstract description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 4
- 238000013461 design Methods 0.000 claims description 3
- 230000006855 networking Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 description 8
- 238000000034 method Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 230000006872 improvement Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005553 drilling Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a distributed processing method for underground logging and mining three-dimensional data of a metal mine, which is based on the following steps: the system comprises a main node, a CPU node, a GPU node, a storage node and an RDMA high-speed network, and comprises the steps of data loading, task decomposition, preprocessing, three-dimensional data processing, three-dimensional data rendering, post-processing and result submission and storage. The invention adopts common computing nodes to form an extensible computing cluster, which not only can deal with larger data volume, but also can provide better performance and has lower cost.
Description
Technical Field
The invention relates to a distributed processing method, in particular to a distributed processing method for massive three-dimensional data of a mine, which is mainly used for processing massive business data generated by geology, measurement and mining in the three-dimensional modeling and visual management and control processes of mine production.
Background
With the development of the digital technology of the mining industry, the application of three-dimensional mining software gradually becomes a trend, and further more and more geological, measurement and mining three-dimensional data are generated. Some of this data comes directly from the three-dimensional laser scanner, some from the borehole data, and some from conversion from other data sources. Compared with traditional two-dimensional data and relational data, the three-dimensional measurement and acquisition data has the characteristics of large data volume and large processing difficulty, and various data sources need to be extracted, converted and loaded on a platform so as to improve the data processing efficiency.
After the integration of multi-source heterogeneous data is solved and the data standardization is solved, the traditional data management and processing solution cannot process huge data.
The current mainstream processing method is still to process serial or data stream three-dimensional data on a high-performance computer, and with the increase of data volume, the processing architecture has presented certain limitations in future technical development, and a higher-performance three-dimensional data processing method is urgently needed for the analysis and processing of three-dimensional data needing to be processed in real time and huge amount of unstructured three-dimensional data.
Disclosure of Invention
The invention provides a distributed processing method for underground logging and mining three-dimensional data of a metal mine, which aims to: the method can deal with larger data volume, provide better performance and reduce cost.
The technical scheme of the invention is as follows:
a distributed processing method for underground logging and mining three-dimensional data of a metal mine comprises the following steps:
step 1, data loading: loading ground measuring and acquiring three-dimensional original data;
step 2, task decomposition: the main node decomposes the task into a plurality of CPU nodes and GPU nodes for processing respectively according to the task characteristics;
step 3, pretreatment: preprocessing the three-dimensional data by a CPU node;
step 4, three-dimensional data processing: performing main body processing on the three-dimensional data by a GPU node;
step 5, rendering three-dimensional data: finishing rendering of the three-dimensional data by the GPU node;
and 6, post-treatment: post-processing the three-dimensional data by the CPU node;
step 7, result submission and storage: the master node aggregates all the processing results and stores or submits the processing results.
As a further improvement of the invention: the CPU node and the GPU node are respectively connected with the main node through Ethernet, so that the transmission of a control command of three-dimensional data processing is realized;
the CPU node and the GPU node are in communication connection through an RDMA data exchange network, and transmission of three-dimensional data result data is achieved.
As a further improvement of the invention: the system also comprises a storage node accessed to the RDMA data exchange network, wherein the storage node is used for storing the three-dimensional logging data.
As a further improvement of the invention: the main node is used for managing the computing cluster and has the functions of cluster networking, task scheduling and system.
As a further improvement of the invention: the data source of the main node is three-dimensional design data.
As a further improvement of the invention: the GPU nodes comprise a main GPU node and a distribution GPU node, the main GPU node is only used for displaying the rendering result, and the distribution GPU node is used for rendering processing.
Compared with the prior art, the invention has the following beneficial effects: the method adopts common computing nodes to form an extensible computing cluster, which not only can deal with larger data volume, but also can provide better performance and has lower cost.
Drawings
FIG. 1 is a schematic block diagram of the present invention;
FIG. 2 is a flow chart of the method.
Detailed Description
The technical scheme of the invention is explained in detail in the following with the accompanying drawings:
referring to fig. 1, a distributed processing architecture for the subsurface exploration of three-dimensional data in a metal mine comprises: a master node, a CPU node, a GPU node, a storage node, and an RDMA high-speed network.
1. A master node: the main functions are that the system is used for managing the whole computing cluster, including cluster networking, adding and quitting of computing nodes and the like; managing the scheduling of the calculation tasks, including the receiving of new tasks, the decomposition of the tasks, the scheduling of the tasks, the result submission of the tasks and the like; and thirdly, some other system functions such as logging, performance monitoring, and the like.
The heterogeneous data of various specialties such as mining, measurement, geology (short for ground survey and mining) and the like are integrated into structured data capable of being calculated in real time through data extraction and cleaning, and the structured data are used as data sources of main nodes and injected into a distributed processing system.
2. CPU computing node: the CPU calculation node is a general CPU calculation type server and is mainly used for preprocessing and post-processing three-dimensional measurement and acquisition data, such as the preprocessing of drilling data, the preprocessing of block data and the like.
By combining the characteristics of three-dimensional processing, a large amount of preprocessing data which do not relate to drawing are processed on the CPU computing nodes, for example, a large amount of numerical computation which is irrelevant to drawing and processing computation are distributed on each CPU computing node for distributed processing, the problem of data processing bottleneck of the traditional serial data is solved, and the preprocessing speed is greatly increased.
3. GPU computing nodes: the calculation capacity of the GPU calculation node is mainly provided by a special GPU, and the GPU has incomparable advantages compared with a CPU in the aspects of processing point cloud data, grid fixed point data, three-dimensional visual rendering and the like in the operation of massive three-dimensional measurement and acquisition data. Therefore, the tasks are disassembled and sent to the GPU node for processing.
The GPU nodes process three-dimensional drawing data and perform data visualization rendering, the performance of the GPU is the key of three-dimensional display performance and effect generally, after distributed processing is adopted, the main GPU is only used for displaying rendering results, a large amount of rendering processing is performed on the distributed GPU, and when the data of a large mine are processed, the performance can be improved by more than 10 times compared with the traditional performance.
4. A storage node: the storage nodes are used for storing three-dimensional measurement and acquisition data.
Similarly, for a large amount of three-dimensional data, a mechanism for synchronously providing distributed storage is needed to ensure the high efficiency and consistency of the data, and at the same time, a function for real-time data sharing is needed to be provided at the presentation and design level.
5. RDMA high-speed network: in the system, a main node and a computing node are connected through a common Ethernet, and management data are transmitted through the Ethernet. The data volume of the three-dimensional measurement and acquisition data is very large, which can reach the scale of hundreds of G or even T, and the efficiency of transmitting the data on the Ethernet is very low. Thus, a connection is made between a compute node and a storage node using an RDMA high speed network, and between a CPU and a GPU node, a connection is also made using a high speed RDMA network for fast data exchange between the two types of nodes
The distributed processing method comprises the following steps:
step 1, data loading: loading ground measuring and acquiring three-dimensional original data;
step 2, task decomposition: the main node decomposes the task into a plurality of CPU nodes and GPU nodes for processing respectively according to the task characteristics;
step 3, pretreatment: preprocessing the three-dimensional data by a CPU node;
step 4, three-dimensional data processing: performing main body processing on the three-dimensional data by a GPU node;
step 5, rendering three-dimensional data: finishing rendering of the three-dimensional data by the GPU node;
and 6, post-treatment: post-processing the three-dimensional data by the CPU node;
step 7, result submission and storage: the master node aggregates all the processing results and stores or submits the processing results.
Claims (6)
1. A distributed processing method for underground logging and mining three-dimensional data of a metal mine is characterized by comprising the following steps:
step 1, data loading: loading ground measuring and acquiring three-dimensional original data;
step 2, task decomposition: the main node decomposes the task into a plurality of CPU nodes and GPU nodes for processing respectively according to the task characteristics;
step 3, pretreatment: preprocessing the three-dimensional data by a CPU node;
step 4, three-dimensional data processing: performing main body processing on the three-dimensional data by a GPU node;
step 5, rendering three-dimensional data: finishing rendering of the three-dimensional data by the GPU node;
and 6, post-treatment: post-processing the three-dimensional data by the CPU node;
step 7, result submission and storage: the master node aggregates all the processing results and stores or submits the processing results.
2. The distributed processing method for metal mine downhole logging three-dimensional data as recited in claim 1, wherein: the CPU node and the GPU node are respectively connected with the main node through Ethernet, so that the transmission of a control command of three-dimensional data processing is realized;
the CPU node and the GPU node are in communication connection through an RDMA data exchange network, and transmission of three-dimensional data result data is achieved.
3. The distributed processing method for metal mine downhole logging three-dimensional data as recited in claim 2, wherein: the system also comprises a storage node accessed to the RDMA data exchange network, wherein the storage node is used for storing the three-dimensional logging data.
4. The distributed processing method for metal mine downhole logging three-dimensional data as recited in claim 1, wherein: the main node is used for managing the computing cluster and has the functions of cluster networking, task scheduling and system.
5. The distributed processing method for metal mine downhole logging three-dimensional data as recited in claim 1, wherein: the data source of the main node is three-dimensional design data.
6. A distributed processing method for metal mine downhole logging three-dimensional data according to any of claims 1 to 5, wherein: the GPU nodes comprise a main GPU node and a distribution GPU node, the main GPU node is only used for displaying the rendering result, and the distribution GPU node is used for rendering processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011040169.1A CN112163986A (en) | 2020-09-28 | 2020-09-28 | Distributed processing method for underground logging and mining three-dimensional data of metal mine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011040169.1A CN112163986A (en) | 2020-09-28 | 2020-09-28 | Distributed processing method for underground logging and mining three-dimensional data of metal mine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112163986A true CN112163986A (en) | 2021-01-01 |
Family
ID=73861902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011040169.1A Pending CN112163986A (en) | 2020-09-28 | 2020-09-28 | Distributed processing method for underground logging and mining three-dimensional data of metal mine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112163986A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101587583A (en) * | 2009-06-23 | 2009-11-25 | 长春理工大学 | The farm of playing up based on the GPU cluster |
| CN102592315A (en) * | 2011-01-12 | 2012-07-18 | 上海库达数字信息技术有限公司 | 3D rendering platform based on GPU cloud cluster |
| CN102609971A (en) * | 2012-01-11 | 2012-07-25 | 南京大学 | Quick rendering system using embedded GPU (Graphics Processing Unit) for realizing 3D-GIS (Three Dimensional-Geographic Information System) |
| CN103049926A (en) * | 2012-12-24 | 2013-04-17 | 广东威创视讯科技股份有限公司 | Distributed three-dimensional rendering system |
| US20140168230A1 (en) * | 2012-12-19 | 2014-06-19 | Nvidia Corporation | Asynchronous compute integrated into large-scale data rendering using dedicated, separate computing and rendering clusters |
| CN104952096A (en) * | 2014-03-31 | 2015-09-30 | 中国电信股份有限公司 | CPU and GPU hybrid cloud rendering method, device and system |
| CN105263050A (en) * | 2015-11-04 | 2016-01-20 | 山东大学 | Mobile terminal real-time rendering system and method based on cloud platform |
| CN106209997A (en) * | 2016-06-30 | 2016-12-07 | 上海上大海润信息系统有限公司 | Heterogeneous Cluster Management System that a kind of facing cloud renders and method |
| CN106910234A (en) * | 2015-12-18 | 2017-06-30 | 普联软件股份有限公司 | One kind is based on improved 3 d rendering engine Distributed Rendering Environment method and system |
-
2020
- 2020-09-28 CN CN202011040169.1A patent/CN112163986A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101587583A (en) * | 2009-06-23 | 2009-11-25 | 长春理工大学 | The farm of playing up based on the GPU cluster |
| CN102592315A (en) * | 2011-01-12 | 2012-07-18 | 上海库达数字信息技术有限公司 | 3D rendering platform based on GPU cloud cluster |
| CN102609971A (en) * | 2012-01-11 | 2012-07-25 | 南京大学 | Quick rendering system using embedded GPU (Graphics Processing Unit) for realizing 3D-GIS (Three Dimensional-Geographic Information System) |
| US20140168230A1 (en) * | 2012-12-19 | 2014-06-19 | Nvidia Corporation | Asynchronous compute integrated into large-scale data rendering using dedicated, separate computing and rendering clusters |
| CN103049926A (en) * | 2012-12-24 | 2013-04-17 | 广东威创视讯科技股份有限公司 | Distributed three-dimensional rendering system |
| CN104952096A (en) * | 2014-03-31 | 2015-09-30 | 中国电信股份有限公司 | CPU and GPU hybrid cloud rendering method, device and system |
| CN105263050A (en) * | 2015-11-04 | 2016-01-20 | 山东大学 | Mobile terminal real-time rendering system and method based on cloud platform |
| CN106910234A (en) * | 2015-12-18 | 2017-06-30 | 普联软件股份有限公司 | One kind is based on improved 3 d rendering engine Distributed Rendering Environment method and system |
| CN106209997A (en) * | 2016-06-30 | 2016-12-07 | 上海上大海润信息系统有限公司 | Heterogeneous Cluster Management System that a kind of facing cloud renders and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107704608A (en) | A kind of OLAP multidimensional analyses and data digging system | |
| CN104205039A (en) | Interest-driven business intelligence systems and methods of data analysis using interest-driven data pipelines | |
| CN108718345A (en) | A kind of digitlization workshop industrial data Network Transmitting system | |
| CN110990467B (en) | BIM model format conversion method and conversion system | |
| CN104504047A (en) | Estimation system of reserve of solid mineral resources based on two-tier database | |
| CN113378011B (en) | Construction method and system of complex product assembly digital twin body | |
| CN113570275A (en) | Water resource real-time monitoring system based on BIM and digital elevation model | |
| CN111753034A (en) | One-stop type geographical big data platform | |
| Zhang et al. | A quick survey on large scale distributed deep learning systems | |
| CN111061732A (en) | Report generation method based on big data processing | |
| CN110059138A (en) | One kind being based on big data platform data analysis domain architecting method | |
| CN112163986A (en) | Distributed processing method for underground logging and mining three-dimensional data of metal mine | |
| CN117591516A (en) | Supervision report data analysis system, supervision report data analysis method, supervision report data analysis equipment and storage medium | |
| CN115984503B (en) | Geological profile generation method, system, electronic equipment and medium | |
| CN112667704A (en) | Coal mine industry internet data middle platform system structure | |
| CN111914014A (en) | Big data platform and application thereof | |
| CN106327153A (en) | Over-cloud scientific workflow excavation method based on event direct prior relation | |
| CN110297597B (en) | Storage method and reading method of seismic data | |
| CN109460599A (en) | A kind of the transmitting quantization analysis method and system of assembly features deviation | |
| CN116523328A (en) | Intelligent decision-making method for cooperation of aviation equipment and manufacturing industry chain | |
| CN115047833A (en) | Mine digital twin factory and construction method thereof | |
| CN114707651A (en) | Topology selection method, device, equipment and medium for protocol operation | |
| CN113052968A (en) | Knowledge graph construction method of three-dimensional structure geological model | |
| CN116433857B (en) | Three-dimensional data visualization method and system for bridge engineering geology | |
| CN106708838A (en) | Flow data query method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210101 |