CN115563376A - Database construction method of mineral resource data based on multivariate data crawling and integration - Google Patents

Abstract

The invention provides a mineral resource database building method based on multi-data crawling and integration, which relates to the technical field of mineral resource data management and aims to solve the problems of large data volume of mineral resources and inconvenient information extraction, fusion and analysis; the data crawled by the web crawler can be screened according to requirements through the preprocessing module and the processing module, and part of irrelevant information is screened and cleaned to obtain the required data; through the cooperation of the classification module and the storage module, the data is classified and stored, the data can be received and stored, and the follow-up calling of the data in the storage module is facilitated.

Description

Database construction method of mineral resource data based on multivariate data crawling and integration

technical field

The invention relates to the technical field of mining resource data management, in particular to a method for building a database of mineral resource data based on multivariate data crawling and integration.

Background technique

Mineral resources, also known as mineral resources, refer to those formed through geological mineralization, naturally occurring in the crust or buried in the ground or exposed on the surface, in solid, liquid or gaseous state, and have development and utilization value A collection of minerals or useful elements.

Mineral resources are important natural resources, which are formed after millions or even hundreds of millions of years of geological changes. They are an important material basis for social production and development. People's production and life in modern society cannot do without mineral resources.

The mineral resources database (mineral resources database) is a collection of data managed by computers, including the geographical location, geological conditions, quantity, quality, and economic value of mineral resources in the country or the world. The mineral resources database can be used as the basis for analyzing the supply and demand of mineral resources, the basis for formulating land use and mineral development policies, and can also be used for the study of regional metallogenic laws and metallogenic predictions. The widespread use of electronic computers has improved the collection, management, and analysis of mineral resource data, prompting rapid progress in this work.

In 1973, the U.S. Geological Survey established a computer-managed resource database, including information on geology, reserves, and resources of about 4,000 domestic and 6,000 foreign deposits and deposits. Most countries in the world have established their own mineral resource databases. For the World Mineral Resources Database, the comparability of classification standards and estimation methods of mineral resources among various countries is very important, and the application of the United Nations Solid Mineral Reserves/Resources Classification Framework is a feasible way to solve this problem.

Diversification: Refers to the development of things, reaching a very rich level, with various classifications and industries. Combined with historical facts, the diversification of modern civilization reflects the process of the spread of Western learning to the east. Including differences in beliefs, cultures, customs, and thinking, also called differences in transformation. The reason for this is that people have different feelings about changes in their surroundings, differences in comprehension ability, thinking habits, and levels of understanding.

Crawling data means to obtain the content information on the required website through programs, such as text, video, pictures and other data.

A web crawler (also known as a web spider, a web robot, and more often referred to as a web chaser in the FOAF community) is a program or script that automatically grabs information on the World Wide Web according to certain rules. Other less commonly used names include ant, autoindex, emulator, or worm.

A web crawler is a program that automatically extracts web pages. It downloads web pages from the World Wide Web for search engines and is an important component of search engines. Traditional crawlers start from the URL of one or several initial webpages, obtain the URLs on the initial webpage, and continuously extract new URLs from the current page and put them into the queue during the process of crawling webpages until a certain stop condition of the system is met.

The work flow of the focused crawler is relatively complicated. It needs to filter links that have nothing to do with the topic according to a certain webpage analysis algorithm, keep useful links and put them into the URL queue waiting to be crawled. Then, it will select the URL of the web page to be crawled next from the queue according to a certain search strategy, and repeat the above process until it stops when a certain condition of the system is reached. In addition, all webpages crawled by crawlers will be stored by the system, analyzed, filtered, and indexed for later query and retrieval; for focused crawlers, the analysis results obtained in this process may also be Give feedback and guidance on future crawling processes.

With the rapid development of network information, mineral resource data volume is large and professional, and there are differences in data storage among different mineral types. Not only are there various types, but also intricate and poor versatility. How to quickly extract and integrate the required multi-mineral resources Data, to achieve data fusion and analysis, is of great significance for improving the level of mineral data informatization.

Contents of the invention

In view of the problems existing in the prior art, the present invention discloses a mineral resource data database building system based on multivariate data crawling and integration, and the adopted technical solution includes

The data server is used to establish the database of mineral resource data;

The central server is used to obtain mineral resource information from the diversified mineral resource information database;

The data crawling module is used to crawl the mineral resource information database using a web crawler for data crawling;

The preprocessing module is used for preliminary screening of the data obtained by the web crawler;

The processing module is used to further screen the preprocessed data to ensure that the data obtained by the web crawler is the data required by the user;

The classification module is used to classify the processed data so as to store the data;

The storage module stores the processed and classified mineral resource data in the mineral resource data database building system.

As a preferred technical solution of the present invention, the processing of the transmitted data by the preprocessing module includes the following steps:

Contrast the webpage confidence of data crawling with the keywords entered for retrieval;

If the number of keywords in the webpage information is less than the preset threshold, it will be discarded, and if the keyword is greater than the preset threshold, it will be transmitted to the processing module. Through the preprocessing module and the processing module, the data crawled by the web crawler can be screened according to the requirements. , to screen and clean some irrelevant information to obtain the required data.

As a preferred technical solution of the present invention, the processing module includes the following steps for processing the preprocessed data information:

Obtain web page information that matches keywords;

Review the obtained web page information;

The correct web page information is sent to the classification module, and the unsuitable web page information is deleted.

As a preferred technical solution of the present invention, the classification module classifies the processed data, which can be classified according to text, picture, parameter and video.

The method for building a database of mineral resources data based on multivariate data crawling and integration adopts a technical solution that includes the following steps:

Establish a database of mineral resource data;

Obtain diversified information about mineral resources through the central server;

Use web crawlers to crawl data from information base;

Preprocess the data crawled;

Perform secondary processing on the preprocessed data;

Store the processed data in the mineral resource database.

As a preferred technical solution of the present invention, the establishment of the mineral resource data database is enough specifically: according to the mineral resource data type, according to the type of mineral or project, the general field and the extended field are distinguished, and the mineral resource database structure is established. Resource data database construction, by using web crawlers to crawl relevant content in various mineral resource data around the world, and integrate them into the required mineral resource database. The built database covers a large amount of information data to meet the needs.

As a preferred technical solution of the present invention, the use of web crawlers to crawl data on information base information includes the input of keywords, specifically the keywords of the information to be crawled are first input, and the keywords can be one or more, And the keywords are separated by spaces, commas or semicolons.

As a preferred technical solution of the present invention, the mineral resource database also includes a query terminal, which is used for users at all levels to obtain the results of query, statistics, analysis and early warning from the central server, and the results are displayed in tables or graphics. Through the classification module In cooperation with the storage module, the data is classified and stored, which can improve the reception and storage of data, and facilitate the subsequent call of the data in the storage module, which is conducive to improving the efficiency of extracting mineral resource-related data from various files and storing them in the warehouse. It is of great significance to improve the information level of geological and mineral resources.

Beneficial effects of the present invention: the present invention builds a database of mineral resources data, and uses web crawlers to crawl relevant content in various mineral resource data around the world, and integrates them into required mineral resource databases. The built database covers a large number of Information data to meet the needs; through the preprocessing module and processing module, the data crawled by the web crawler can be screened according to the demand, and some irrelevant information can be screened and cleaned to obtain the required data; through the classification module and storage With the cooperation of the modules, the data is classified and stored, which can improve the reception and storage of data, and facilitate the subsequent call of the data in the storage module, which is conducive to improving the efficiency of extracting mineral resource-related data from various files and storing them in the warehouse. The level of geological and mineral information is of great significance.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention;

Fig. 2 is a schematic structural diagram of the system of the present invention;

Fig. 3 is a schematic diagram of the pretreatment module of the present invention;

Fig. 4 is a schematic diagram of the processing module of the present invention;

Fig. 5 is a schematic diagram of the classification module of the present invention.

detailed description

Example 1

As shown in Fig. 2 to Fig. 5, the present invention discloses a mineral resource data database building system based on multivariate data crawling and integration, and the adopted technical solution is, including

The data server is used to establish the database of mineral resource data;

The central server is used to obtain mineral resource information from the diversified mineral resource information database;

The data crawling module is used to crawl the mineral resource information database using a web crawler for data crawling;

The preprocessing module is used for preliminary screening of the data obtained by the web crawler;

The processing module is used to further screen the preprocessed data to ensure that the data obtained by the web crawler is the data required by the user;

The classification module is used to classify the processed data so as to store the data;

The storage module stores the processed and classified mineral resource data in the mineral resource data database building system.

Data server, the types of mineral resource data stored in the data server include: spatial data, time data, attribute data and other data, etc., spatial data such as shp data and MapGIS data, etc.; time data such as date and year; attribute data such as tables data, etc.; other data includes pdf files, text documents, and videos.

The central server, the central server is connected with the external multiple mineral resource data sources, and data can be exchanged between them. The multiple mineral resource data sources mainly include the websites of various geological survey bureaus (such as the US Geological Survey USGS, British Geological Survey BGS, Japan Petroleum Natural gas and metal mineral resources organization JOGMEC, etc.), the websites of the Ministry of Commerce of various countries, the websites of the Ministry of Foreign Affairs of various countries, the websites of mineral departments of various countries (such as the Ministry of Mines and Mineral Development of Zambia, etc.), the websites of mineral resource industry associations (Nonferrous Metals Association, World Iron and Steel Association, etc.) , the websites of large mining companies (such as Minmetals, Chinalco, Glencore, etc.), and other websites (such as the website of Control Risk, a risk consulting company, and the website of OneGology, a global geological data platform, etc.).

As a preferred technical solution of the present invention, the processing of the transmitted data by the preprocessing module includes the following steps:

Contrast the webpage confidence of data crawling with the keywords entered for retrieval;

If the number of keywords in the webpage information is less than the preset threshold, it will be discarded, and if the keyword is greater than the preset threshold, it will be transmitted to the processing module. Through the preprocessing module and the processing module, the data crawled by the web crawler can be screened according to the requirements. , to screen and clean some irrelevant information to obtain the required data.

As a preferred technical solution of the present invention, the processing module includes the following steps for processing the preprocessed data information:

Obtain web page information that matches keywords;

Review the obtained web page information;

The correct web page information is sent to the classification module, and the unsuitable web page information is deleted.

As a preferred technical solution of the present invention, the classification module classifies the processed data, which can be distinguished and classified according to text, pictures, parameters, and videos, and the data is classified and stored through the cooperation of the classification module and the storage module, which can Improving the receiving and storage of data and facilitating the subsequent call of data in the storage module is conducive to improving the efficiency of extracting mineral resource-related data from various files and storing them in storage, which is of great significance for improving the level of geological and mineral information.

As shown in Figure 1, the present invention discloses a method for building a database of mineral resource data based on multivariate data crawling and integration, including the following steps:

Establish a database of mineral resource data;

Obtain diversified information about mineral resources through the central server;

Use web crawlers to crawl data from information base;

Preprocess the data crawled;

Perform secondary processing on the preprocessed data;

Store the processed data in the mineral resources database.

The purpose of preprocessing the data crawled is to filter the corresponding data information according to the input keywords, and conduct a preliminary screening of various data that does not meet the requirements, so as to extract the data that meets the requirements.

The data information that does not meet the requirements is mainly based on the comparison of the retrieved content, and the percentage of the content of the keywords contained in the content is determined; if the percentage of the content is less than the preset threshold, the content retrieved in this article does not meet, and the content is searched. Delete; if the percentage of the content is greater than or equal to the preset threshold, the content will compound the requirements and enter the next module.

Perform secondary processing on the preprocessed data; for erroneous data, use statistical analysis methods to identify possible error values or outliers, such as deviation analysis, identify values that do not obey the distribution or regression equation, or use a simple rule base ( Common-sense rules, business-specific rules, etc.) check data values, or use constraints between different attributes, external data to detect and clean data.

For duplicate data, check whether the records are equal by judging whether the attribute values between the records are equal, and the equal records are merged into one record (that is, merged/cleared). Merge/clear is the basic method of deduplication.

As a preferred technical solution of the present invention, it is enough to set up the database of mineral resource data as follows:

According to the data type of mineral resources, general fields and extended fields are distinguished according to mineral types or projects, and the database structure of mineral resources is established.

As a preferred technical solution of the present invention, the use of web crawlers to crawl data on information base information includes the input of keywords, specifically the keywords of the information to be crawled are first input, and the keywords can be one or more, And the keywords are separated by spaces, commas or semicolons.

After the data classification is completed through the classification module, that is, when the data is integrated into the warehouse, a memory buffer for temporarily storing the warehoused data is opened in the free memory, and multiple batches of warehoused data received from time to time are stored in the warehouse. The memory buffer, and then send these data to the storage module of the database in a centralized manner.

As a preferred technical solution of the present invention, the mineral resources database also includes a query terminal for users at all levels to obtain query, statistics, analysis and early warning results from the central server, and the results are displayed in tables or graphics.

Although the above-mentioned specific embodiments of the present invention have been described in detail, the present invention is not limited to the above-mentioned embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art, various The modification or deformation without creative work is still within the protection scope of the present invention.

Claims (8)

1. Mineral resources data base system of establishing based on multivariate data crawl and integration its characterized in that: comprises that

The data server is used for establishing a mineral resource data database;

the central server is used for acquiring mineral resource information from the diversified mineral resource information base;

the data crawling module is used for crawling data of the mineral resource information base by using a web crawler;

the preprocessing module is used for primarily screening the data acquired by the web crawler;

the processing module is used for further screening the preprocessed data to ensure that the data acquired by the web crawler is the data required by the user;

the classification module is used for classifying the processed data so as to store the data;

and the storage module is used for storing the processed and classified mineral resource data into a mineral resource data database building system.

2. The multivariate data crawling and integration based mineral resource database construction system as claimed in claim 1, wherein: the preprocessing module processes the transmitted data and comprises the following steps:

comparing the web page confidence of data crawling with the input searched keywords;

abandoning the keywords in the webpage information, wherein the number of the keywords is smaller than a preset threshold value, and transmitting the keywords larger than the preset threshold value to a processing module.

3. The mineral resource database building system based on multivariate data crawling and integration according to claim 1, wherein: the processing module for processing the preprocessed data information comprises the following steps:

acquiring webpage information conforming to the keywords;

rechecking the acquired webpage information;

and sending the correct webpage information to the classification module, and deleting the inconsistent webpage information.

4. The mineral resource database building system based on multivariate data crawling and integration according to claim 1, wherein: the classification module classifies the processed data and can distinguish and classify the data according to characters, pictures, parameters and videos.

5. The mineral resource database building method based on the multi-data crawling and integration is characterized by comprising the following steps of:

establishing a mineral resource database;

acquiring diversified information about mineral resources through a central server;

using a web crawler to perform data crawling on information of the information base;

preprocessing data obtained by data crawling;

carrying out secondary processing on the preprocessed data;

and storing the processed data into a mineral resource database.

6. The mineral resource database building method based on multivariate data crawling and integration according to claim 5, wherein the building of the mineral resource database is specifically as follows: and according to the data type of the mineral resources, distinguishing the general field and the extension field according to the mineral types or projects, and establishing a mineral resource database structure.

7. The mineral resources data base building method based on multivariate data crawling and integration according to claim 5, characterized in that: the data crawling for the information base information by using the web crawler comprises the step of inputting keywords, specifically, the keywords of the information to be crawled are firstly input, the keywords can be one or more, and the keywords are separated by spaces, commas or semicolons.

8. The mineral resources data base building method based on multivariate data crawling and integration according to claim 5, characterized in that: the mineral resource database also comprises a query end which is used for all levels of users to obtain results of query, statistics, analysis and early warning from the central server, and the results are displayed in a table or graphic mode.

Images