CN117422581A - Mineral resource safety monitoring and early warning method, system, equipment and medium - Google Patents
Mineral resource safety monitoring and early warning method, system, equipment and medium Download PDFInfo
- Publication number
- CN117422581A CN117422581A CN202311440675.3A CN202311440675A CN117422581A CN 117422581 A CN117422581 A CN 117422581A CN 202311440675 A CN202311440675 A CN 202311440675A CN 117422581 A CN117422581 A CN 117422581A
- Authority
- CN
- China
- Prior art keywords
- mineral resource
- data
- nodes
- historical data
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 229910052500 inorganic mineral Inorganic materials 0.000 title claims abstract description 165
- 239000011707 mineral Substances 0.000 title claims abstract description 165
- 238000012544 monitoring process Methods 0.000 title claims abstract description 47
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000004088 simulation Methods 0.000 claims abstract description 32
- 239000013598 vector Substances 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 11
- 238000003058 natural language processing Methods 0.000 claims abstract description 11
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000002372 labelling Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 5
- 238000013138 pruning Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 11
- 239000010410 layer Substances 0.000 description 66
- 230000009193 crawling Effects 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000008451 emotion Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000004140 cleaning Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 238000009966 trimming Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 235000014510 cooky Nutrition 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012502 risk assessment Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 238000013068 supply chain management Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000035899 viability Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/02—Agriculture; Fishing; Mining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F18/24155—Bayesian classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
Abstract
The invention discloses a mineral resource safety monitoring and early warning method, a system, equipment and a medium, and relates to the field of mineral resource supply chain monitoring and early warning; the method comprises the following steps: writing a crawler program to crawl a target website, and collecting multi-element mineral resource data; performing feature extraction on historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; a naive Bayes classifier is built according to the feature vectors of the marked historical data; labeling unlabeled historical data by adopting a naive Bayes classifier to obtain labeled multi-element mineral resource data; and inputting the marked multi-element mineral resource data and risk characteristics of historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result. The invention monitors the whole process of the mineral resource supply chain and accurately pre-warns the mineral resource supply chain, thereby improving the operation efficiency of the supply chain.
Description
Technical Field
The invention relates to the field of mineral resource supply chain monitoring and early warning, in particular to a mineral resource safety monitoring and early warning method, system, equipment and medium.
Background
With the continuous development of social economy and the improvement of the living standard of people, the demand of mineral resources has shown an increasing trend. However, the management and monitoring of the mineral resource supply chain presents challenges due to the wide distribution of mineral resources and the complex mining process.
Currently, for management and monitoring of mineral resource supply chains, the following problems exist: (1) The traditional monitoring means mainly depend on manual inspection and data statistics, and has low efficiency and high cost; (2) The existing monitoring system can only monitor a certain link of the supply chain, and cannot comprehensively know the overall condition of the supply chain; (3) The lack of an effective early warning mechanism often requires a significant amount of time and resources to discover and resolve once a problem occurs.
Therefore, developing a high-efficiency and comprehensive mineral resource supply chain monitoring and early warning method and system has important significance for improving the operation efficiency of the mineral resource supply chain and reducing risks.
Disclosure of Invention
The invention aims to provide a mineral resource safety monitoring and early warning method, a system, equipment and a medium, which can monitor the whole process and accurately early warn a mineral resource supply chain so as to improve the operation efficiency of the supply chain and reduce risks.
In order to achieve the above object, the present invention provides the following solutions:
the mineral resource safety monitoring and early warning method comprises the following steps:
constructing a multi-layer complex network model;
writing a crawler program to crawl a target website, and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data;
performing feature extraction on historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; the risk characteristics comprise time, position, level, influence space range, influence time period and economic loss caused by risk occurrence;
a naive Bayes classifier is built according to the characteristic vector of the marked historical data in the multi-element mineral resource data;
marking unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayes classifier to obtain marked multi-element mineral resource data;
and inputting the marked multi-element mineral resource data and risk characteristics of historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
Optionally, constructing a multi-layer complex network model, specifically including:
determining nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node comprises a plurality of regional nodes, and the country node is used as a node of a country layer;
determining the connection relation between mine nodes and regional nodes according to the geographic position, the yield and the mineral types;
determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation;
determining an association between each node; the associations include supply chain relationships, trade flows, and market demands;
based on the connection relationship and association between each node, a multi-layer complex network model is constructed using the network model.
Optionally, the network model includes a directed graph or undirected graph model in graph theory.
Optionally, the feature extraction of the historical data in the multi-element mineral resource data by adopting a natural language processing technology is further included before the risk feature and the feature vector of the historical data in the multi-element mineral resource data are obtained:
preprocessing multi-element mineral resource data; the preprocessing includes removing HTML tags, pruning spaces, converting case, deleting special characters, normalization processing, and formatting processing.
The mineral resource safety monitoring and early warning system is applied to the mineral resource safety monitoring and early warning method, and comprises the following steps:
the multi-layer complex network model building module is used for building a multi-layer complex network model;
the crawler module is used for compiling a crawler program to crawl the target website and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data;
the naive Bayes classifier construction module is used for constructing a naive Bayes classifier according to marked historical data in the multi-element mineral resource data;
the labeling module is used for labeling unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayesian classifier to obtain labeled multi-element mineral resource data;
the characteristic extraction module is used for extracting characteristics of historical data in the marked multi-element mineral resource data by adopting a natural language processing technology to obtain risk characteristics of the historical data in the marked multi-element mineral resource data; the risk characteristics comprise time, position, level, influence space range, influence time period and economic loss caused by risk occurrence;
and the simulation module is used for inputting the marked multi-element mineral resource data and the risk characteristics of historical data in the marked multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
Optionally, the multi-layer complex network model building module specifically includes:
a node determining unit configured to determine nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node comprises a plurality of regional nodes, and the country node is used as a node of a country layer;
the first connection relation determining unit is used for determining the connection relation between the mine nodes and the regional nodes according to the geographic position, the yield and the mineral categories;
the first connection relation determining unit is used for determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation;
an association determining unit configured to determine an association between each of the nodes; the associations include supply chain relationships, trade flows, and market demands;
and the construction unit is used for constructing a multi-layer complex network model by using the network model based on the connection relation and the association between each node.
An electronic device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the mineral resource safety monitoring and early warning method when executing the computer program.
A computer readable storage medium, wherein a computer program is stored on the storage medium, and the computer program is executed to realize the mineral resource safety monitoring and early warning method.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention discloses a mineral resource safety monitoring and early warning method, a system, equipment and a medium, wherein the method comprises the following steps: constructing a multi-layer complex network model; writing a crawler program to crawl a target website, and collecting multi-element mineral resource data; real-time acquisition of target website data can be realized by writing a crawler program, and the multi-element mineral resource data can be updated in real time; performing feature extraction on historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; a naive Bayes classifier is built according to the characteristic vector of the marked historical data in the multi-element mineral resource data; marking unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayes classifier to obtain marked multi-element mineral resource data; the naive Bayes classifier is adopted for labeling, so that the workload of manual labeling can be reduced. And inputting the marked multi-element mineral resource data and risk characteristics of historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result. The invention monitors the whole process of the mineral resource supply chain and accurately pre-warns the mineral resource supply chain so as to improve the operation efficiency of the supply chain and reduce the risk.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a mineral resource safety monitoring and early warning method according to an embodiment of the invention;
FIG. 2 is a block diagram of a multi-layer complex network model according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a mineral resource safety monitoring and early warning method, a system, equipment and a medium, which can monitor the whole process and accurately early warn a mineral resource supply chain so as to improve the operation efficiency of the supply chain and reduce risks.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
As shown in FIG. 1, the mineral resource safety monitoring and early warning method of the invention comprises the following steps:
step 101: and constructing a multi-layer complex network model. The multi-layer complex network model abstracts each link of a mineral resource supply chain of a production mine, a region where the production mine is located and a supply country into nodes, respectively establishes a single-layer complex network, and establishes connection among the nodes.
Step 102: writing a crawler program to crawl a target website, and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data.
Step 102 specifically includes:
determining websites and target pages to be crawled: the web sites and pages requiring data collection are determined, which can comprise related web sites of mineral resource companies, suppliers, logistics companies and the like, and a crawling pool is built by gathering url of target web sites.
Configuration crawler: a crawler is written and implemented using a suitable programming language and library. The program is configured to simulate browser behavior, send HTTP requests and parse responses.
Webpage analysis and data extraction: the required data is extracted by parsing the HTML or other web page source code. This can be done using XPath, regular expression, or HTML parsing library tools.
Storing data: the extracted multi-element mineral resource data is stored in a database or file for subsequent analysis and processing.
Text cleaning and preprocessing: the crawled multi-element mineral resource data is cleaned and preprocessed, including removing HTML tags, non-alphabetic characters, stop words, and the like. This helps to clean up the multi-element mineral resource data and to enhance the effectiveness of subsequent processing. The preprocessing includes removing HTML tags, pruning spaces, converting case, deleting special characters, normalization processing, and formatting processing.
A specific example of step 102 is provided:
step 1021: in order to obtain the original data of the attribute characteristics of the multi-layer complex network model node in the step 101, the network crawlers are used for crawling the data of the related information of mineral resources such as the world resource reserves, import and export trade, investigation investment, mining projects and the like which are published by the network. In order to obtain multi-element mineral resource data more efficiently, a crawler crawling pool is constructed for website URL sources. Crawling pools are a mechanism for managing and scheduling crawler tasks. It includes a task queue, a scheduler, a crawler process/thread pool, a deduplication machine, and a memory. The task queue is responsible for storing URLs or tasks to be crawled, and the scheduler takes tasks out of the queue and assigns them to available crawler processes/threads for processing. The crawler process/thread sends the request, parses the page, fetches the data, and stores the results. The deduplication machine is used to avoid repeatedly crawling the same page. The crawling pool of the crawler website can improve crawling efficiency and simplify task management. The target websites include economic trade websites, geological survey websites, business websites, outcoming websites, mineral sector websites, mineral resource industry association websites, and mineral company websites.
Step 1022: crawling target website real-time data
A website to be crawled and corresponding information is determined from the crawling pool. Verifying the legitimacy of the web site crawling and adhering to the rules of the anti-crawling mechanism which may exist. Using the Starpy framework of Python, it is determined where and how to access the data to be crawled by looking at the source code of the web page and analyzing its structure. The selector can effectively help the target information parsing according to the HTML structure and CSS of the web page. By sending an HTTP request to obtain the content of the target web page and setting a request header, processing cookies, sessions, etc., the web page content is extracted from the response, and the required data is extracted using a corresponding parsing library (e.g., beautfulso). The target element is located and extracted using a CSS selector or XPath expression or the like. The real-time information you extract is stored in a suitable data storage system, such as a database, CSV file or other format. And finally, setting a timing task, automatically running a crawler program, and updating data in a database and a file at regular intervals.
Step 1023: cleaning crawling result data
Preprocessing multi-element mineral resource data; the preprocessing comprises the steps of removing HTML labels, trimming spaces, converting cases, deleting special characters, normalizing and formatting, and specifically comprises the following steps:
after data crawling, the raw data acquired often needs to be cleaned and preprocessed for better use and analysis. The following is a general data cleaning procedure:
(1) And checking whether invalid data such as noise, missing values, repeated items, blank characters and the like exist in the multi-element mineral resource data. Such invalid data may be identified and deleted using a data cleansing tool or code. The data is converted into a unified format and unit to ensure consistency of the data. For example, date and time are converted into a standard format, and the values are unified into the same unit. Checking whether missing values exist in the multi-element mineral resource data, and then selecting a proper processing method according to the situation. Common methods include deleting rows/columns containing missing values, interpolating the missing values, using default values, etc. For text data in the multi-element mineral resource data, some additional processing may be required, such as removing HTML tags, trimming spaces, converting cases, deleting special characters, etc.
(2) The multi-element mineral resource data is converted into an appropriate data type according to the meaning and purpose of the multi-element mineral resource data. For example, converting a character string into a number, converting a date-time character string into a date-time object, and the like. Abnormal values, i.e. values that are significantly different from other data or outside a reasonable range, are detected and processed. The outliers may be selected for removal, replaced with reasonable estimates, or corrected using statistical methods. Repeated data lines are checked and deleted as needed to avoid repeated influence of the multiple mineral resource data on subsequent analysis. For the multi-element mineral resource data related to different dimensions, normalization or normalization processing is performed for comparison and analysis. Common methods include min-max normalization, Z-score normalization, and the like.
(3) And verifying the cleaned multi-element mineral resource data to ensure that the cleaned multi-element mineral resource data meets the expected rules and requirements. Statistical analysis, visualization, etc. may be used to check the distribution, relationship, and consistency of the cleaned multi-element mineral resource data. The cleaned multi-element mineral resource data is saved to a proper format, such as a database, a CSV file, and the like, for subsequent analysis and use.
Step 103: and extracting features of the historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data. The risk characteristics include time, location, level, impact space scope, impact time period, and economic loss caused by the risk occurrence.
The natural language processing technology is a Bag of words model (Bag of words) or TF-IDF (Term Frequency-Inverse Document Frequency) and the like. These techniques may convert text into a numerical feature representation for subsequent analysis. Firstly, the marked multi-element mineral resource data are converted into feature vectors which can be processed by a machine learning algorithm. The bag of words model and TF-IDF are used here to extract features.
Word bag model: the text is split into words and the frequency of occurrence of each word in the text is calculated. A vector representing text is obtained in which each element represents the frequency of a word. TF-IDF: the importance of the word is determined by calculating the word frequency (TF) and the Inverse Document Frequency (IDF). TF represents the frequency of occurrence of a word in the text and IDF represents the importance of the word in the whole corpus. A vector representation text is finally obtained in which each element represents the TF-IDF value of a word.
In an implementation, step 104 is preceded by: word segmentation: dividing the marked multi-element mineral resource data into sequences of words or phrases. This facilitates subsequent post-labeling multi-element mineral resource data analysis and feature extraction.
Step 104: and establishing a naive Bayesian classifier according to the feature vectors of the marked historical data in the multi-element mineral resource data.
A data set is prepared containing the marked text samples and their corresponding natural risk categories. The text samples in the data set are converted into feature vectors through preprocessing and feature extraction steps. A naive bayes classifier is trained using the training data set. The classifier will learn the association between text features and corresponding natural risk categories to enable classification of new unlabeled text. After training is completed, the new unlabeled text may be classified and identified using a trained naive bayes classifier.
Step 105: and marking unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayes classifier to obtain marked multi-element mineral resource data.
The same feature extraction step is performed on the new unlabeled text, converting it into feature vectors. And inputting the feature vectors into a trained naive Bayes classifier to obtain predicted natural risk categories.
From the processed data and the trained model, risk factors and risk levels for each piece of text may be derived. The results can be used in scenes such as risk assessment, early warning systems, public opinion monitoring and the like, and help people to better understand and manage risks.
In the specific implementation, the influence degree of the historical data in the marked multi-element mineral resource data can be analyzed to analyze: and analyzing emotion tendencies in historical data in the multi-element mineral resource data to judge whether the risk influence is positive, negative or neutral. The method of random forests and cyclic neural networks is used for training emotion classification models for historical data in multi-element mineral resource data, and in the training process, algorithms learn mapping relations from the historical data in the multi-element mineral resource data to emotion categories and can be used for emotion classification, namely risk degree classification, of new multi-element mineral resource data.
The key features are extracted using a machine learning visualization algorithm SHAP (SHapleyAdditive exPlanations), generating SHAP values for each feature. SHAP values can be used to explain why the prediction is biased in a certain direction. By plotting the SHAP values, it is possible to understand how each feature affects the predicted outcome and how they drive the predicted outcome to change. And then the extracted key influencing factors are monitored in real time.
Step 106: and inputting the marked multi-element mineral resource data and risk characteristics of historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
Step 106 specifically includes:
step 1061: and (3) inputting the risk characteristics of the historical data in the multi-element mineral resource data obtained in the step (103) and the marked multi-element mineral resource data obtained in the step (105) into a three-layer complex network. According to the requirements of network structure and data format, the risk characteristics of the historical data in the multi-element mineral resource data obtained in the step 103 and the marked multi-element mineral resource data obtained in the step 105 can be stored in the form of text files, databases and the like, and corresponding input interfaces are prepared. For example, country, region, mine information is passed into complex network nodes, and identified and categorized risks and their grades are passed into the border of complex networks. Data conversion and mapping: the risk factor data for natural language processing is converted into a format required by the network model. The data is mapped appropriately according to the requirements of the network model, and the text information is converted into nodes, edges or other network elements.
Step 1062: after the data is transmitted into the multi-layer complex network, the data and the structure of the model are basically completed, and then the parameter setting of the simulation is determined, including the simulation time period, the node state transition rule, the network topology structure and the like. According to the requirements and the reality situation, parameters are reasonably set to realize accurate simulation. And (3) performing simulation: and inputting the constructed network model and the set parameters into a simulation system by using the selected simulation tool or programming language to perform simulation. And simulating the processes of state change, information transmission, risk diffusion and the like among the nodes according to the set time period. In the simulation process, the simulation result is monitored in real time. The probability and severity of risk occurrence is determined by analyzing node status, network index changes, etc. And carrying out risk early warning according to the set early warning rules and threshold values, and timely finding and taking measures to cope with potential risks. Feedback and optimization: and feeding back and optimizing according to the simulation result and the monitoring and early warning condition. And (3) adjusting and improving a network model, simulation parameters, early warning rules and the like so as to improve the accuracy and effect of simulation.
Step 1063: and generating a visual report or chart according to the simulation result, and mainly displaying the risk influence grade influence and the group affecting the country from multiple dimensions. According to the requirements, the result is applied to the actual situation, and relevant works such as decision making, risk management and the like are supported.
As shown in fig. 2, as an embodiment, the construction of the multi-layer complex network model specifically includes:
determining nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node includes a plurality of regional nodes, and the country node serves as a node of the country layer.
And determining the relation and connection modes among mines, areas and country nodes. Depending on the actual situation, the following ways can be considered: connection between mine layer and zone layer: generally, a regional node should contain multiple mine nodes, meaning that the region has multiple mineral resource points. The connection between the mine and the area may be determined by geographical location, production, mineral type, etc. Connection between regional layer and country layer: a country may contain multiple regional nodes indicating the presence of multiple geographic or administrative regions within the country. The connection mode can consider the factors such as the traffic network, administrative administration relation and the like in the country.
And determining the connection relation between the mine nodes and the regional nodes according to the geographic position, the yield and the mineral types.
And determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation.
In addition to the direct hierarchical relationships, other relationships between nodes need to be determined, such as supply chain relationships, trade flows, market demands, etc. Relevant information can be obtained through data collection, market research and expertise, and correlation analysis and modeling can be performed using appropriate algorithms.
Determining an association between each node; the associations include supply chain relationships, trade flows, and market demands.
Based on the relation and association between the nodes, a three-layer complex network of various mineral resource international trade supply chains is constructed by using an appropriate network model. Common network models include directed graph or undirected graph models in graph theory, where nodes represent mines, areas and countries and edges represent connections and relationships between nodes.
Based on the connection relationship and association between each node, a multi-layer complex network model is constructed using the network model.
In particular implementations, mine nodes include mineral categories, yields, and reserves of resources; the regional nodes include geographic locations, traffic conditions, and policy environments; the national nodes include economic viability, trade policies, and legal systems.
As an embodiment, the network model comprises a directed graph or undirected graph model in graph theory.
Example 2
A mineral resource safety monitoring and early warning system applied to the mineral resource safety monitoring and early warning method of embodiment 1, the mineral resource safety monitoring and early warning system comprising:
and the multi-layer complex network model building module is used for building a multi-layer complex network model.
The crawler module is used for compiling a crawler program to crawl the target website and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data.
The feature extraction module is used for extracting features of historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; the risk characteristics include time, location, level, impact space scope, impact time period, and economic loss caused by the risk occurrence.
The naive Bayes classifier construction module is used for constructing a naive Bayes classifier according to the characteristic vectors of the marked historical data in the multi-element mineral resource data.
The labeling module is used for labeling unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayesian classifier, so as to obtain the labeled multi-element mineral resource data.
And the simulation module is used for inputting the marked multi-element mineral resource data and the risk characteristics of the historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
As an embodiment, the multi-layer complex network model building module specifically includes:
a node determining unit configured to determine nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node includes a plurality of regional nodes, and the country node serves as a node of the country layer.
And the first connection relation determining unit is used for determining the connection relation between the mine nodes and the regional nodes according to the geographic position, the yield and the mineral product types.
And the connection relation determining unit is used for determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation.
An association determining unit configured to determine an association between each of the nodes; the associations include supply chain relationships, trade flows, and market demands.
And the construction unit is used for constructing a multi-layer complex network model by using the network model based on the connection relation and the association between each node.
An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the mineral resource security monitoring and early warning method of embodiment 1 when the computer program is executed.
A computer-readable storage medium having stored thereon a computer program which, when executed, implements the mineral resource safety monitoring and warning method of embodiment 1.
The invention has the following beneficial effects:
(1) The informatization level of mineral resource supply chain management is improved, and the management cost is reduced.
(2) The main risk affecting the safety of the mineral resource supply chain is extracted, so that the subsequent targeted provision of precautionary measures is facilitated.
(3) And the risk factors of the mineral resource supply chain are monitored in real time, so that the safety problem is timely early-warned, and the risk damage cost is reduced. In summary, the present invention provides an innovative method and system that can effectively monitor and pre-warn the mineral resource supply chain, providing decision support and risk management means for relevant practitioners.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (8)
1. The mineral resource safety monitoring and early warning method is characterized by comprising the following steps of:
constructing a multi-layer complex network model;
writing a crawler program to crawl a target website, and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data;
performing feature extraction on historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; the risk characteristics comprise time, position, level, influence space range, influence time period and economic loss caused by risk occurrence;
a naive Bayes classifier is built according to the characteristic vector of the marked historical data in the multi-element mineral resource data;
marking unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayes classifier to obtain marked multi-element mineral resource data;
and inputting the marked multi-element mineral resource data and risk characteristics of historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
2. The mineral resource safety monitoring and early warning method according to claim 1, wherein the construction of the multilayer complex network model specifically comprises the following steps:
determining nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node comprises a plurality of regional nodes, and the country node is used as a node of a country layer;
determining the connection relation between mine nodes and regional nodes according to the geographic position, the yield and the mineral types;
determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation;
determining an association between each node; the associations include supply chain relationships, trade flows, and market demands;
based on the connection relationship and association between each node, a multi-layer complex network model is constructed using the network model.
3. The mineral resource safety monitoring and early warning method according to claim 2, wherein the network model comprises a directed graph or undirected graph model in graph theory.
4. The mineral resource safety monitoring and early warning method according to claim 1, wherein the method further comprises the steps of:
preprocessing multi-element mineral resource data; the preprocessing includes removing HTML tags, pruning spaces, converting case, deleting special characters, normalization processing, and formatting processing.
5. A mineral resource safety monitoring and early warning system, characterized in that the mineral resource safety monitoring and early warning system is applied to the mineral resource safety monitoring and early warning method according to any one of claims 1 to 4, and the mineral resource safety monitoring and early warning system comprises:
the multi-layer complex network model building module is used for building a multi-layer complex network model;
the crawler module is used for compiling a crawler program to crawl the target website and collecting multi-element mineral resource data; the multi-element mineral resource data comprise marked historical data and unmarked historical data; the noted historical data comprises historical data and natural risk categories corresponding to the historical data;
the feature extraction module is used for extracting features of historical data in the multi-element mineral resource data by adopting a natural language processing technology to obtain risk features and feature vectors of the historical data in the multi-element mineral resource data; the risk characteristics comprise time, position, level, influence space range, influence time period and economic loss caused by risk occurrence;
the naive Bayes classifier construction module is used for constructing a naive Bayes classifier according to the feature vectors of the marked historical data in the multi-element mineral resource data;
the labeling module is used for labeling unlabeled historical data in the multi-element mineral resource data by adopting a naive Bayesian classifier to obtain labeled multi-element mineral resource data;
and the simulation module is used for inputting the marked multi-element mineral resource data and the risk characteristics of the historical data in the multi-element mineral resource data into a multi-layer complex network model for simulation, and monitoring and early warning the simulation result.
6. The mineral resource safety monitoring and early warning system according to claim 5, wherein the multi-layer complex network model building module specifically comprises:
a node determining unit configured to determine nodes of a plurality of layers; the layers comprise a mine layer, an area layer and a country layer; each layer includes a node; the nodes are mines, areas or countries; the mine nodes are used as nodes of a mine layer; the regional nodes comprise a plurality of mine nodes, and the regional nodes serve as nodes of the regional layer; the country node comprises a plurality of regional nodes, and the country node is used as a node of a country layer;
the first connection relation determining unit is used for determining the connection relation between the mine nodes and the regional nodes according to the geographic position, the yield and the mineral categories;
the first connection relation determining unit is used for determining the connection relation between the regional node and the national node according to the traffic network and the administrative administration relation;
an association determining unit configured to determine an association between each of the nodes; the associations include supply chain relationships, trade flows, and market demands;
and the construction unit is used for constructing a multi-layer complex network model by using the network model based on the connection relation and the association between each node.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the mineral resource security monitoring and warning method of any one of claims 1 to 4 when the computer program is executed by the processor.
8. A computer readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when executed, implements the mineral resource safety monitoring and pre-warning method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311440675.3A CN117422581A (en) | 2023-11-01 | 2023-11-01 | Mineral resource safety monitoring and early warning method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311440675.3A CN117422581A (en) | 2023-11-01 | 2023-11-01 | Mineral resource safety monitoring and early warning method, system, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117422581A true CN117422581A (en) | 2024-01-19 |
Family
ID=89532350
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311440675.3A Pending CN117422581A (en) | 2023-11-01 | 2023-11-01 | Mineral resource safety monitoring and early warning method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117422581A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622537A (en) * | 2004-12-15 | 2005-06-01 | 中国科学院计算机网络信息中心 | Short circuit communication method of layered type switching network |
| CN111445110A (en) * | 2020-03-05 | 2020-07-24 | 深圳供电局有限公司 | Cable channel-based environmental risk decision method and device and computer equipment |
| CN114386429A (en) * | 2021-12-22 | 2022-04-22 | 应急管理部信息研究院 | Coal mine disaster risk prediction method and system based on semantic recognition |
| CN114969477A (en) * | 2022-05-25 | 2022-08-30 | 中国地质科学院矿产资源研究所 | Mineral resource database building method and system based on multi-data crawling and integration |
| CN115563376A (en) * | 2022-10-26 | 2023-01-03 | 中国地质调查局沈阳地质调查中心 | Mineral resource database building method based on multi-data crawling and integration |
| WO2023061039A1 (en) * | 2021-10-13 | 2023-04-20 | 中通服和信科技有限公司 | Tailing pond risk monitoring and early-warning system based on internet of things |
| CN116070385A (en) * | 2023-02-09 | 2023-05-05 | 中国地质科学院矿产资源研究所 | Automatic risk identification method and system for overseas mineral resource supply chain |
-
2023
- 2023-11-01 CN CN202311440675.3A patent/CN117422581A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1622537A (en) * | 2004-12-15 | 2005-06-01 | 中国科学院计算机网络信息中心 | Short circuit communication method of layered type switching network |
| CN111445110A (en) * | 2020-03-05 | 2020-07-24 | 深圳供电局有限公司 | Cable channel-based environmental risk decision method and device and computer equipment |
| WO2023061039A1 (en) * | 2021-10-13 | 2023-04-20 | 中通服和信科技有限公司 | Tailing pond risk monitoring and early-warning system based on internet of things |
| CN114386429A (en) * | 2021-12-22 | 2022-04-22 | 应急管理部信息研究院 | Coal mine disaster risk prediction method and system based on semantic recognition |
| CN114969477A (en) * | 2022-05-25 | 2022-08-30 | 中国地质科学院矿产资源研究所 | Mineral resource database building method and system based on multi-data crawling and integration |
| CN115563376A (en) * | 2022-10-26 | 2023-01-03 | 中国地质调查局沈阳地质调查中心 | Mineral resource database building method based on multi-data crawling and integration |
| CN116070385A (en) * | 2023-02-09 | 2023-05-05 | 中国地质科学院矿产资源研究所 | Automatic risk identification method and system for overseas mineral resource supply chain |
Non-Patent Citations (6)
| Title |
|---|
| 包丽霞;: "数字矿井监控、预测系统的研究及应用", 中国商界(上半月), no. 05, 15 May 2009 (2009-05-15) * |
| 李献峰;薛惠锋;苏锦旗;: "基于钻孔信息的二维剖面图的三维实体重建", 计算机应用研究, no. 07, 15 July 2008 (2008-07-15) * |
| 杨燕敏;: "突发水污染环境事件应急监测常见问题及对策", 资源节约与环保, no. 04, 25 April 2020 (2020-04-25) * |
| 林吉海, 谢文兵: "基于BP网络算法的矿产资源评价方法", 重庆工商大学学报(自然科学版), no. 05, 30 October 2004 (2004-10-30) * |
| 邹永斌;陈兴蜀;王文贤;: "基于贝叶斯分类器的主题爬虫研究", 计算机应用研究, no. 09, 15 September 2009 (2009-09-15) * |
| 陈其慎 等: "矿产资源供应基地评价与供应链调查理论技术方法", 地球学报, vol. 42, no. 2, 31 March 2021 (2021-03-31), pages 159 - 166 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10977293B2 (en) | Technology incident management platform | |
| EP4195112A1 (en) | Systems and methods for enriching modeling tools and infrastructure with semantics | |
| Ajayi et al. | Big data platform for health and safety accident prediction | |
| Li et al. | Fault diagnosis expert system of semiconductor manufacturing equipment using a Bayesian network | |
| Ansari | Cost-based text understanding to improve maintenance knowledge intelligence in manufacturing enterprises | |
| KR102421904B1 (en) | the method to advance the analysis of the causes of disasters | |
| CN113656805B (en) | Event map automatic construction method and system for multi-source vulnerability information | |
| Xu et al. | Analysis of agricultural exports based on deep learning and text mining | |
| CN107526833B (en) | URL management method and system | |
| CN116993306A (en) | Knowledge graph-based construction method and device of network collaborative manufacturing system | |
| CN117422581A (en) | Mineral resource safety monitoring and early warning method, system, equipment and medium | |
| CN116225848A (en) | Log monitoring method, device, equipment and medium | |
| US11494416B2 (en) | Automated event processing system | |
| Hu et al. | A classification model of power operation inspection defect texts based on graph convolutional network | |
| Zhang et al. | DGWC: Distributed and generic web crawler for online information extraction | |
| CN114529383A (en) | Method and system for realizing tax payment tracking and tax loss early warning | |
| CN114625725A (en) | Visualization method, system, equipment and medium for air monitoring data | |
| Fathiya et al. | University Admissions Predictor Using Logistic Regression | |
| Kopaygorodsky | Methods and technologies of building an intelligent service for energy technology forecasting | |
| Jiang et al. | A knowledge graph–based requirement identification model for products remanufacturing design | |
| Sajjadi Ghaemmaghami | User Behavioral Modeling for Web-based Systems | |
| Ruidong et al. | The Research of Big Data Analysis Platform of Oil & Gas Production | |
| Regla et al. | An Enhanced Research Productivity Monitoring System for Higher Education Institutions (HEI's) with Natural Language Processing (NLP) | |
| Ozyurt et al. | Using Web Crawling for Automated Country Risk Assessment | |
| Demirjian et al. | Artificial Intelligence and the Future of Environmental Health and Safety Management. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |