CN113254508A - Data warehouse for natural gas pipeline pigging operation and data mining method - Google Patents
Data warehouse for natural gas pipeline pigging operation and data mining method Download PDFInfo
- Publication number
- CN113254508A CN113254508A CN202110694989.0A CN202110694989A CN113254508A CN 113254508 A CN113254508 A CN 113254508A CN 202110694989 A CN202110694989 A CN 202110694989A CN 113254508 A CN113254508 A CN 113254508A
- Authority
- CN
- China
- Prior art keywords
- data
- gas pipeline
- natural gas
- pipeline
- pigging operation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/10—Geometric CAD
- G06F30/18—Network design, e.g. design based on topological or interconnect aspects of utility systems, piping, heating ventilation air conditioning [HVAC] or cabling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2113/00—Details relating to the application field
- G06F2113/14—Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2119/00—Details relating to the type or aim of the analysis or the optimisation
- G06F2119/12—Timing analysis or timing optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2119/00—Details relating to the type or aim of the analysis or the optimisation
- G06F2119/14—Force analysis or force optimisation, e.g. static or dynamic forces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Geometry (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computational Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a data warehouse for natural gas pipeline pigging operation and a data mining method. The method comprises the following steps: firstly, determining a data source of natural gas pipeline pigging operation; secondly, completing data integration by utilizing an ETL tool; then establishing a natural gas pipeline pigging operation data warehouse according to the Epicentre data model; then, extracting the data from the warehouse and loading the data into a data mart; and finally, carrying out data mining of the natural gas pipeline pigging operation based on a defined algorithm. The method fully considers the value and the discreteness of the actual pigging operation data, realizes the consistency and the integrality of the pigging operation data, provides a management basis for evaluating the safety of the pigging operation of the natural gas pipeline, and provides technical support for developing data mining and intelligent pigging of the pigging operation of the natural gas pipeline.
Description
Technical Field
The invention belongs to the field of safety guarantee of natural gas conveying pipelines, and particularly relates to a data warehouse for natural gas pipeline pigging operation and a data mining method.
Background
Natural gas is a clean high-quality energy and chemical raw material, and pipeline transportation is the main transportation mode of the natural gas. According to the middle-long-term pipe network development planning released by the national development and improvement commission and the national energy agency, the total mileage of natural gas pipelines in China is estimated to reach 16.3 ten thousand kilometers in 2025. In the operation process of production and daily production of the natural gas pipeline, dirt, impurities, accumulated liquid and the like left in the process of construction or transportation need to be removed through pipe cleaning operation, so that the purposes of providing pipeline conveying efficiency, prolonging the service life of the pipeline and guaranteeing the safe operation of the pipeline are achieved.
At present, the natural gas pipeline pigging operation is mainly developed by taking a station as a unit and taking a pipe section as an interval, and the generated pigging operation data is visual embodiment of a pigging scheme and is an important basis for reflecting the running condition of the natural gas pipeline. However, the natural gas pipeline pigging operation data has a large amount of information and little useful information, and is stored in a relatively dispersed form in the work logs of different stations or databases of different systems, so that inconsistency and incompleteness of the data are easy to occur. The data warehouse is an efficient data storage system, and can store, extract and maintain data from different places, heterogeneous data sources or other databases in the data warehouse after processing to form a series of theme-oriented, integrated, stable and different time-series sets, so that a comprehensive analysis-oriented and decision-making-oriented application environment is provided for users. Data mining is the mining of useful information from large amounts of data as decision support. At present, the existing natural gas pipeline pigging operation data have the characteristics of discreteness, immediacy, authenticity, visibility, relevance, value, specialty and the like, and the problems of great difficulty in developing data mining and intelligent research exist. Therefore, the existing data storage mode for the natural gas pipeline pigging operation cannot specially classify, efficiently store and effectively utilize data, cannot provide technical support for data mining, and cannot probe the pigging rule of a certain natural gas pipeline. Therefore, a method is needed to establish a data warehouse for gas pipeline pigging operation, and analyze and calculate historical pigging operation data of a certain gas pipeline, so as to solve inconsistency and incompleteness of the gas pipeline pigging operation data, meet the requirement of mining useful information from the historical pigging operation data, and provide technical support for guiding future intelligent pigging construction and application.
To sum up, the existing natural gas pipeline pigging operation data has the characteristics of discreteness, immediacy, authenticity, visibility, relevance, value, specialty and the like, and has the problems of inconsistency, incompleteness and the like, no mature and efficient pigging operation data warehouse is available for specially classifying and efficiently storing the historical pigging operation data, and the historical pigging operation data cannot be conveniently maintained and effectively utilized. The method can provide a data warehouse for the natural gas pipeline pigging operation and a data mining method, so that a data warehouse system for the natural gas pipeline pigging operation is constructed, historical pigging operation data is analyzed and processed, and data mining is carried out, so that the problems of inconsistency and incompleteness of pigging operation data of the natural gas pipeline are solved, the uniformity and the completeness of the pigging operation data are realized, technical support is provided for carrying out information mining and effective utilization of the pigging operation data of the natural gas pipeline, a decision basis is provided for formulating a pigging scheme, and reference is provided for evaluating the safety of a pigging process of the natural gas pipeline.
Disclosure of Invention
The invention provides a data warehouse for natural gas pipeline pigging operation and a data mining method, aiming at solving various defects and shortcomings in the prior art, and adopting the following technical scheme:
the data warehouse and the data mining method for the natural gas pipeline pigging operation are provided, and the method comprises the following steps:
step one, determining a data source of the natural gas pipeline pigging operation. The data source is the most original data stored in each business department, and is information record and calculation result generated by natural gas pipeline pigging operation, including work logs of each business department, databases of other software or systems, pigging operation record, design data of natural gas pipeline, and the like;
and step two, integrating data of the natural gas pipeline pigging operation. The data integration is to extract the data of the pigging operation from a data source by utilizing an ETL tool, load the data into a data warehouse according to a defined mapping relation, check the legality of the data in the loading process, and record a loading result into a working log of the data integration for query;
and step three, establishing a data warehouse for natural gas pipeline pigging operation. And establishing a data warehouse for the natural gas pipeline pigging operation based on the common data model and the software development standard. Adopting an Epicentre data model of an oil exploration and development integrated platform of an international POSC organization to construct a data warehouse of natural gas pipeline pigging operation;
and step four, extracting data of the natural gas pipeline pigging operation. The data extraction is to automatically extract data information in a data warehouse of natural gas pipeline pigging operation according to a defined data warehouse model and load the data information into data marts with different application themes according to a defined mapping relation;
and fifthly, performing data mart of the natural gas pipeline pigging operation. Data marts are collections of data built on the basis of the subject of the research or application in question, and are direct vehicles for data mining. The method comprises the steps of pressure data mart, flow data mart, pipe cleaner operation speed data mart, pigging time data mart, terrain change data mart and the like of natural gas pipeline pigging operation;
and sixthly, data mining of the natural gas pipeline pigging operation. Data mining is to extract relevant data from a data mart for analysis according to different research or application topics, and mine the knowledge hidden behind the data based on appropriate algorithms.
Step one, determining a data source of the natural gas pipeline pigging operation. The data source is the most original data stored in each business department, and is information records and calculation results generated by the gas pipeline pigging operation. The method comprises the following concrete steps:
(1) the data source is determined as a data source, and the data source of the natural gas pipeline pigging operation is mainly a business department or software related to the pigging operation and comprises a ball serving station, a ball receiving station, a valve chamber or station yard along the natural gas pipeline, an SCADA system, a dispatching center and the like;
(2) and determining metadata, wherein the metadata is data describing the whole natural gas pipeline pigging operation process and comprises work logs of all service departments, databases of other software or systems, pigging operation records, design data of the natural gas pipeline and the like.
And step two, integrating data of the natural gas pipeline pigging operation. The data integration is to extract the data of the pigging operation from a data source by utilizing an ETL tool, load the data into a data warehouse according to a defined mapping relation, check the legality of the data in the loading process, and record the loading result into a working log of the data integration for query. The method comprises the following concrete steps:
(1) the ETL tool is used for extracting data, so that the original distributed pigging operation data in the application system is not influenced, the originally existing heterogeneous data source still operates independently, data service is provided for the respective application system, and the original application architecture of an enterprise is not damaged.
(2) A mapping relationship is defined. Defining a mapping relation according to the serve time of the pigging operation and the region where the pipeline or the serve station is located by using the line name of the natural gas pipe as a folder; wherein, the name of metadata is defined by the area where the pipeline or the service station is located, and the arrangement mode of the metadata is defined by the service time of the pigging operation;
(3) and selecting the metadata according to a definition relation, checking the legality of the metadata such as format, definition and the like, loading the legal metadata into a data warehouse, and isolating the illegal metadata.
And step three, establishing a data warehouse for natural gas pipeline pigging operation. And adopting an Epicentre data model of an oil exploration and development integrated platform of the international POSC organization to construct a data warehouse of the gas pipeline pigging operation. The method comprises the following concrete steps:
(1) establishing a natural gas pipeline basic parameter data warehouse:
extracting the natural gas pipeline basic parameters from a data source, and loading the natural gas pipeline basic parameters into a natural gas pipeline basic parameter data warehouse in a data integration mode, so as to establish a natural gas pipeline basic parameter data warehouse;
the natural gas pipeline basic parameter data warehouse comprises pipeline names, pipeline lengths, wall thicknesses, inner diameters, outer diameters, roughness, pipeline along-line elevation change conditions, design pressure, design gas transmission quantity, crossing structures, environment temperatures, mileage and elevations of pipeline along-line station yards or valve chambers and the like.
(2) Establishing a pigging operation running state parameter data warehouse:
extracting the operation state parameters of the pigging operation from a data source, and loading the operation state parameters into a pigging operation state parameter data warehouse in a data integration mode so as to establish a pigging operation state parameter data warehouse;
the warehouse for the operational state parameter data of the pipe cleaning operation comprises the pressure and the gas transmission amount of a ball sending station, the pressure and the gas transmission amount of a ball receiving station, the change situation of the pressure and the flow along the pipeline along the time, the gas inlet and distribution situation along the pipeline, the operation speed of a pipe cleaner, the time when the pipe cleaner reaches each station or valve chamber, the pipe cleaning time and the like.
(3) Establishing a pig parameter data warehouse:
extracting the pipe cleaner parameters from the data source, and loading the pipe cleaner parameters into a pipe cleaner parameter data warehouse in a data integration mode, so as to establish a pipe cleaner parameter data warehouse;
the pig parameter data repository includes: pig type, material, length, diameter, interference, mass, modulus of elasticity, coefficient of sliding friction, and the like.
And step four, extracting data of the natural gas pipeline pigging operation. The data extraction is to automatically extract data information in a data warehouse of natural gas pipeline pigging operation according to a defined data warehouse model and load the data information into data marts with different application themes according to a defined mapping relation. The method comprises the following concrete steps:
(1) defining an automatic extraction data warehouse model:
according to the application theme of the data mart, different automatic extraction data warehouse models are defined, so that metadata in a data warehouse can be efficiently extracted and maintained;
the automatic extraction data warehouse model comprises a pipe cleaner running speed extraction model, a pipe cleaning time extraction model, a pipeline line pressure extraction model, a pipeline line flow extraction model, a pipeline line terrain change extraction model and the like.
(2) Extraction model for running speed of pipe cleaner
The data extracted from the natural gas pipeline basic parameter data warehouse by the pipeline cleaner running speed extraction model comprises the pipeline name, the pipeline length, the wall thickness, the inner diameter, the outer diameter, the roughness, the elevation change condition along the pipeline, the environment temperature, the mileage and the elevation of a station yard or a valve chamber along the pipeline and the like;
the data extracted by the pipeline cleaner operation speed extraction model from the pipeline cleaning operation state parameter data warehouse comprises the change situation of pressure and flow along the pipeline along the time, the gas inlet and distribution situation along the pipeline, the pipeline cleaner operation speed, the time when the pipeline cleaner reaches each station or valve chamber, the pipeline cleaning time and the like;
the data extracted by the pig run speed extraction model from the pig parameter data repository includes: pig type, material, length, diameter, interference, mass, modulus of elasticity, coefficient of sliding friction, and the like.
(3) Pigging time extraction model
The data extracted from the data warehouse by the pigging time extraction model comprises the following data: pipeline basic parameter data such as pipeline name, pipeline length, elevation change condition of the pipeline along the pipeline, a crossing structure, mileage and elevation of a station yard or a valve chamber along the pipeline and the like, and pipeline cleaning operation state parameter data such as pipeline cleaning operation speed, time of the pipeline cleaning to reach each station yard or valve chamber, pipeline cleaning time and the like.
(4) Pipeline pressure extraction model
The data extracted from the data warehouse by the pipeline pressure extraction model comprises the following data: the pressure of the ball serving station, the pressure of the ball receiving station, the change situation of the pressure along the pipeline along with time, the pipeline name, the pipeline length and the like.
(5) Pipeline flow extraction model
The data extracted from the data warehouse by the pipeline flow extraction model comprises the following data: the flow of the ball serving station, the flow of the ball receiving station, the change situation of the flow along the pipeline along with time, the air inlet and distribution situation along the pipeline, the pipeline name, the pipeline length and other data.
(6) Pipeline terrain change extraction model
The data extracted from the data warehouse by the pipeline terrain change extraction model comprises the following data: pipeline name, pipeline length, elevation change condition of the pipeline along the pipeline, crossing structure, mileage and elevation of the pipeline along a station yard or a valve chamber, and the like.
And step five, establishing a data mart of the natural gas pipeline pigging operation. The method comprises a pressure data market, a flow data market, a tube cleaner running speed data market, a tube cleaning time data market, a terrain data market and the like of the natural gas pipeline cleaning operation.
The method comprises the following concrete steps:
(1) pig operating speed data mart
And according to the pipeline pig running speed extraction model, relevant data related to the pipeline pig running speed is extracted from the data warehouse and loaded into a pipeline pig running speed data mart.
(2) Inventory time data mart
And according to the pigging time extraction model, extracting relevant data related to the pigging time from the data warehouse and loading the relevant data into the pigging time data mart.
(3) Pressure data mart
And according to the pipeline pressure extraction model, extracting relevant books related to the pressure from a data warehouse and loading the books into a pressure data mart.
(4) Traffic data mart
Extracting die according to flow along pipeline
Type, relevant data related to the traffic is extracted from the data warehouse and loaded into the traffic data mart.
(5) Terrain data marts
And according to the terrain change extraction model along the pipeline, relevant data related to the terrain are extracted from the data warehouse and loaded into the terrain data mart.
And sixthly, data mining of the natural gas pipeline pigging operation. According to different research or application subjects, respectively five application subjects or research objects including pig running speed, pigging time, flow, pressure and terrain are extracted from the data mart for analysis, and the knowledge hidden behind the data is mined based on a proper algorithm. The method comprises the following concrete steps:
(1) determining application topics or research objects for data mining
For the natural gas pipeline cleaning operation, an application theme or a research object for developing data mining comprises 5 aspects, namely the running speed of a pipe cleaner, the pipe cleaning time, the flow change of the pipeline along the pipeline, the pressure change of the pipeline along the pipeline and the terrain change of the pipeline along the pipeline.
(2) Basis of data mining
Based on different application themes, the data mining is carried out according to data marts, including 5 data marts such as pipe cleaner running speed, pipe cleaning time, pressure, flow and terrain. And according to the user requirements, carrying out corresponding data mining work according to the corresponding data marts, thereby extracting useful information as a decision basis.
(3) Data mining algorithm
When data mining of natural gas pipeline pigging operation is carried out, a statistical method and a induction method are introduced to analyze the existing information. A mathematical statistical model is first established, and then the relevant knowledge is extracted by the model.
(4) Algorithm of pig running speed
Aiming at a folder with a pipeline name as a file name, algorithms of historical data of the running speed of the existing pipeline cleaner can be divided into two types, wherein one type is used for analyzing a specific one-time cleaning operation, and the other type is used for analyzing all cleaning operations of the pipeline;
when a specific pipe cleaning operation is analyzed, a mathematical calculation method is adopted to respectively calculate the maximum pipe cleaner running speed, the minimum pipe cleaner running speed and the average pipe cleaner running speed, so that the maximum pipe cleaner running speed, the minimum pipe cleaner running speed and the average pipe cleaner running speed are used as reference basis for judging whether the pipe cleaner running speed of the specific pipe cleaning operation is reasonable or not;
in the formula (I), the compound is shown in the specification,
-average pig run speed, km/h;
l-the pig travel distance, km;
t is pigging time, h;
when all the pigging operations of a certain natural gas pipeline are analyzed, a statistical method and a regression method are adopted to calculate the distribution interval of the running speed of the pipe cleaner, and the proportion of each distribution interval of the running speed of the pipe cleaner is obtained, so that the overall situation of the pipeline pigging operation is judged.
(5) Time-to-pigging algorithm
Aiming at a folder with a pipeline name as a file name, analyzing historical data of the existing pipeline cleaner time by an algorithm of the historical data, wherein the algorithm is mainly aiming at all pipeline cleaning operations of the pipeline;
counting all the pigging time of the pipeline by adopting a statistical method, and carrying out comparative analysis to obtain the maximum value, the minimum value and the average value of historical data of all the pigging time, the distribution interval of the pigging time and the proportion of the distribution interval to the pigging time, thereby obtaining a statistical result of judging the pigging operation time;
(6) flow and pressure algorithm
The change conditions of the flow and the pressure along the pipeline can be respectively obtained from a flow data mart and a pressure data mart according to flow and pressure algorithms, and the flow and the pressure algorithms are mainly used for obtaining useful information such as maximum flow, minimum flow, average flow, maximum pressure drop, minimum pressure drop, average pressure drop, maximum pressure drop rate and the like through analysis.
(7) Terrain algorithm
The change condition of the terrain along the pipeline can be obtained from the terrain data mart by a terrain algorithm, and the maximum elevation and the minimum elevation are mainly obtained through statistics, and data such as the maximum elevation difference and the minimum elevation difference are calculated.
The invention provides a data warehouse for natural gas pipeline cleaning operation and a data mining method, which are novel methods for developing related researches or applications of pipe cleaner operation speed, pipe cleaning time, pipeline pressure along a pipeline, flow and terrain change conditions through data integration, data extraction and data fitting on the basis of pipeline basic parameters, pipe cleaning operation state parameters and pipe cleaner parameters of a natural gas long-distance pipeline and in combination with related theories of the data warehouse and data mining. The method is easy to realize the consistency and the completeness of the data of the pigging operation, can be convenient for maintaining and managing the pigging operation data and providing technical service for developing data mining, and solves the problem that no mature technology and a calculation method are available at present for carrying out unified maintenance management and efficient utilization on the pigging operation data. The method has clear working flow, convenient maintenance and management and wide application prospect, can extract useful data information from massive pigging operation data, provides technical support for developing data mining and intelligent pigging, and provides decision basis for evaluating the safety of pigging operation of the natural gas pipeline.
Drawings
FIG. 1 is a flow chart of a data warehouse and data mining method for gas pipeline pigging;
FIG. 2 is a data warehouse model diagram of a natural gas pipeline pigging operation;
FIG. 3 is a data mart model diagram of a natural gas pipeline pigging operation.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following examples. It should be understood that the specific examples described herein are intended to be illustrative only and are not intended to be limiting.
The invention provides a data warehouse for natural gas pipeline pigging operation and a data mining method, wherein the method mainly comprises the following calculation steps:
step one, determining a data source of the natural gas pipeline pigging operation. The data source is the most original data stored in each business department, and is information record and calculation result generated by the natural gas pipeline pigging operation, including work logs of each business department, databases of other software or systems, pigging operation record, design data of the natural gas pipeline, and the like.
And step two, integrating data of the natural gas pipeline pigging operation. The data integration is to extract the data of the pigging operation from a data source by utilizing an ETL tool, load the data into a data warehouse according to a defined mapping relation, check the legality of the data in the loading process, and record the loading result into a working log of the data integration for query.
And step three, establishing a data warehouse for natural gas pipeline pigging operation. And establishing a data warehouse for the natural gas pipeline pigging operation based on the common data model and the software development standard. And adopting an Epicentre data model of an oil exploration and development integrated platform of the international POSC organization to construct a data warehouse of the gas pipeline pigging operation.
And step four, extracting data of the natural gas pipeline pigging operation. The data extraction is to automatically extract data information in a data warehouse of natural gas pipeline pigging operation according to a defined data warehouse model and load the data information into data marts with different application themes according to a defined mapping relation.
And fifthly, performing data mart of the natural gas pipeline pigging operation. Data marts are collections of data built on the basis of the subject of the research or application in question, and are direct vehicles for data mining. The method comprises a pressure data market, a flow data market, a tube cleaner running speed data market, a tube cleaning time data market, a terrain change data market and the like of natural gas pipeline cleaning operation.
And sixthly, data mining of the natural gas pipeline pigging operation. Data mining is to extract relevant data from a data mart for analysis according to different research or application topics, and mine the knowledge hidden behind the data based on appropriate algorithms.
In step one, data sources are determined, including work logs of each business department, databases of other software or systems, pigging operation records, design data of natural gas pipelines, and the like. The method comprises data information such as pipeline name, diameter, wall thickness, length, elevation along the pipeline, mileage and elevation of a station yard or a valve chamber, pressure, gas transmission amount, pigging time, pig parameters and the like.
In the second step, the data of the pigging operation needs to be extracted from the data source by using an ETL tool according to a defined mapping relation, the legality of the data is checked, the legal data is loaded into a data warehouse, the inconsistency and the incompleteness of the pigging operation data in different data sources are solved, and the uniformity of the pigging operation data is realized.
In the third step, a data warehouse of the gas pipeline pigging operation needs to be constructed based on an Epicentre data model of an oil exploration and development integrated platform of the international POSC organization, and the data warehouse is respectively 3 data warehouses of gas pipeline basic parameters, pigging operation state parameters, pipe cleaner parameters and the like:
(1) natural gas pipeline basic parameter data warehouse
The metadata in the natural gas pipeline basic parameter data warehouse is natural gas pipeline basic parameters which are extracted from a data source and accord with a mapping relation, and comprises pipeline names, pipeline lengths, wall thicknesses, inner diameters, outer diameters, roughness, elevation change conditions along pipelines, design pressure, design gas transmission quantity, crossing structures, environment temperatures, mileage and elevation of a pipeline station yard or a valve chamber along the pipelines and the like;
(2) pigging operation running state parameter data warehouse
The metadata in the pigging operation running state parameter data warehouse is pigging operation running state parameters which are extracted from a data source and accord with a mapping relation, and comprises pressure and gas transmission of a ball sending station, pressure and gas transmission of a ball receiving station, pressure and flow along a pipeline along the time, gas inlet and distribution conditions along the pipeline, running speed of a pipe cleaner, time when the pipe cleaner reaches each station or valve chamber, pigging time and the like;
(3) pig parameter data warehouse
Metadata in the pig parameter data repository, which are pig parameters extracted from the data sources that conform to the mapping relationship, includes: pig type, material, length, diameter, interference, mass, modulus of elasticity, coefficient of sliding friction, and the like.
In the fourth step, an automatic extraction data warehouse model is defined, and then the data information in the data warehouse is loaded into the data marts of different application topics or research objects according to the defined mapping relationship:
(1) extraction model for running speed of pipe cleaner
The pipeline cleaner operation speed extraction model needs more data information extracted from the data warehouse, and the data in 3 data warehouses such as natural gas pipeline basic parameters, pipeline cleaner operation state parameters, pipeline cleaner parameters and the like are all related;
the data extracted from the natural gas pipeline basic parameter data warehouse comprises the pipeline name, the pipeline length, the wall thickness, the inner diameter, the outer diameter, the roughness, the elevation change condition along the pipeline, the environmental temperature, the mileage and the elevation of a station yard or a valve chamber along the pipeline and the like;
the data extracted from the pipe cleaning operation state parameter data warehouse comprises the change situation of pressure and flow along the pipeline along the time, the gas inlet and distribution situation along the pipeline, the running speed of a pipe cleaner, the time when the pipe cleaner reaches each station or valve chamber, the pipe cleaning time and the like;
data extracted from the pig parameter data warehouse includes: pig type, material, length, diameter, interference, mass, modulus of elasticity, coefficient of sliding friction, and the like;
(2) pigging time extraction model
The data extracted from the data warehouse by the pigging time extraction model mainly relates to two data warehouses of natural gas pipeline basic parameters and pigging operation running state parameters, and comprises the following steps: pipeline basic parameter data such as pipeline name, pipeline length, elevation change condition of the pipeline along the pipeline, a crossing structure, mileage and elevation of a station yard or a valve chamber along the pipeline and the like, and pipeline cleaning operation state parameter data such as pipeline cleaning operation speed, the time when the pipeline cleaner reaches each station yard or valve chamber, pipeline cleaning time and the like;
(3) pipeline pressure extraction model
The data extracted from the data warehouse by the pipeline pressure extraction model comprises the following data: the ball serving station pressure, the ball receiving station pressure, the change situation of the pressure along the pipeline along with time, the pipeline name, the pipeline length and other data;
(4) pipeline flow extraction model
The data extracted from the data warehouse by the pipeline flow extraction model comprises the following data: the data of the flow of the ball serving station, the flow of the ball receiving station, the change situation of the flow along the pipeline along with time, the air inlet and distribution situation along the pipeline, the pipeline name, the pipeline length and the like;
(5) pipeline terrain change extraction model
The data extracted from the data warehouse by the pipeline terrain change extraction model comprises the following data: pipeline name, pipeline length, elevation change condition of the pipeline along the pipeline, crossing structure, mileage and elevation of the pipeline along a station yard or a valve chamber, and the like.
In the fifth step, according to different application themes, different data extraction models are adopted to establish corresponding data marts, including 5 data marts such as pressure, flow, pipe cleaner running speed, pipe cleaning time and terrain change:
(1) pig operating speed data mart
Extracting relevant data related to the running speed of the pipe cleaner from a data warehouse according to the pipe cleaner running speed extraction model, and loading the relevant data into a pipe cleaner running speed data mart for analyzing the running speed of the pipe cleaner;
(2) inventory time data mart
Extracting relevant data related to the pigging time from a data warehouse according to a pigging time extraction model, and loading the data into a pigging time data mart for analyzing the pigging time;
(3) pressure data mart
Extracting relevant books related to the pressure from a data warehouse according to the pipeline pressure extraction model, and loading the books into a pressure data mart for analyzing the change condition of the pipeline pressure along the pipeline;
(4) traffic data mart
According to the pipeline flow extraction model along the pipeline, relevant data related to the flow is extracted from a data warehouse and loaded into a flow data mart for analyzing the change condition of the pipeline flow along the pipeline;
(5) terrain data marts
And according to the terrain change extraction model along the pipeline, extracting relevant data related to the terrain from a data warehouse, and loading the relevant data into a terrain data mart for analyzing the change condition of the terrain along the pipeline.
In the sixth step, data mining of the gas pipeline pigging operation can be further performed. And respectively starting from five application themes of pipe cleaner running speed, pipe cleaning time, flow, pressure and terrain, mining the hidden knowledge behind the data based on a proper algorithm:
(1) basis of data mining
The data mining is carried out according to data marts, and the data marts comprise 5 data marts such as pipe cleaner running speed, pipe cleaning time, pressure, flow and terrain. And according to the user requirements, carrying out corresponding data mining work according to the corresponding data marts, thereby extracting useful information as a decision basis.
(2) Algorithm of pig running speed
Aiming at a folder with a pipeline name as a file name, algorithms of historical data of the running speed of the existing pipeline cleaner can be divided into two types, wherein one type is used for analyzing a specific one-time cleaning operation, and the other type is used for analyzing all cleaning operations of the pipeline;
when a specific pipe cleaning operation is analyzed, a mathematical calculation method is adopted to respectively calculate the maximum pipe cleaner running speed, the minimum pipe cleaner running speed and the average pipe cleaner running speed, so that the maximum pipe cleaner running speed, the minimum pipe cleaner running speed and the average pipe cleaner running speed are used as reference basis for judging whether the pipe cleaner running speed of the specific pipe cleaning operation is reasonable or not;
when all the pigging operations of a certain natural gas pipeline are analyzed, a statistical method and a regression method are adopted to calculate the distribution interval of the running speed of the pipe cleaner, and the proportion of each distribution interval of the running speed of the pipe cleaner is obtained, so that the overall situation of the pipeline pigging operation is judged.
(3) Time-to-pigging algorithm
Aiming at a folder with a pipeline name as a file name, analyzing historical data of the existing pipeline cleaner time by an algorithm of the historical data, wherein the algorithm is mainly aiming at all pipeline cleaning operations of the pipeline;
counting all the pigging time of the pipeline by adopting a statistical method, and carrying out comparative analysis to obtain the maximum value, the minimum value and the average value of historical data of all the pigging time, the distribution interval of the pigging time and the proportion of the distribution interval to the pigging time, thereby obtaining a statistical result of judging the pigging operation time;
(4) flow and pressure algorithm
The change conditions of the flow and the pressure along the pipeline can be respectively obtained from a flow data mart and a pressure data mart according to flow and pressure algorithms, and the flow and the pressure algorithms are mainly used for obtaining useful information such as maximum flow, minimum flow, average flow, maximum pressure drop, minimum pressure drop, average pressure drop, maximum pressure drop rate and the like through analysis.
(5) Terrain algorithm
The change condition of the terrain along the pipeline can be obtained from the terrain data mart by a terrain algorithm, and the maximum elevation and the minimum elevation are mainly obtained through statistics, and data such as the maximum elevation difference and the minimum elevation difference are calculated.
The application of the principles of the present invention will now be further described with reference to specific examples.
Example (c): the length of a natural gas long-distance pipeline is 27.5km, the diameter of the pipeline is 813mm, the wall thickness is 8.8mm, the starting point height range is 282m, the end point height is 297m, the inlet pressure of the pipeline in the pipeline cleaning process is 5.15MPa, the outlet pressure is 5.08MPa, and the gas transmission capacity is 620 multiplied by 104m3And d, the ball serving time is 9:30, and the ball receiving time is 12: 15. In addition, historical data of the gas pipeline pigging operation are stored in a dispatching center in an Excel table mode, and data analysis of pigging time is tried to be developed.
The first step is as follows: determining a data source, wherein the data source is historical data of the pigging operation stored in a dispatching center of the pipeline; the pipeline length L is 27.5km, the pipeline diameter d is 813mm, the wall thickness delta is 8.8mm, and the starting point height H is long0282m, end point elevation Hi297m, ball serving time 9:30, ball receiving time, pipe inlet pressure Q05.15MPa, pipeline outlet pressure Qi5.08MPa, gas transmission quantity Q at pipeline inletTransfusion system＝620×104m3/d。
The second step is that: data integration, namely extracting historical data of the pigging operation from a data center by using an ETL tool, and checking the type of data storage to enable the data storage to conform to a defined mapping relation;
thirdly, a data warehouse sets the pipeline length L as 27.5km, the pipeline diameter d as 813mm, the wall thickness delta as 8.8mm and the starting point height H0282m, end point elevation HiStoring data of 297m and the like into a natural gas pipeline basic parameter data warehouse;
the ball serving time, the ball receiving time and the pipeline inlet pressure Q05.15MPa, pipeline outlet pressure Qi5.08MPa, gas transmission quantity Q at pipeline inletTransfusion system＝620×104m3The data such as the data/d and the like are stored in a data warehouse of the operation state parameters of the pigging operation;
storing the pipe cleaner parameters in the historical data of the pipeline into a pipe cleaner parameter data warehouse;
fourthly, data extraction, namely defining a pigging time extraction model according to the application theme as the pigging time, then extracting related data information from a data warehouse,the method comprises the following steps: pipeline length L equal to 27.5km and starting point height H0282m, end point elevation HiPipeline basic parameter data such as 297m and pigging operation running state parameter data such as serve time, receive time and pigging time;
fifthly, performing data mart, namely forming a pigging time data mart according to related data obtained by a pigging time extraction model so as to lay a foundation for data mining;
sixthly, data mining, namely analyzing the historical data of similar or similar pigging operation by taking parameters such as ball sending time, gas transmission amount, gas transmission pressure and the like as references according to pigging time data mart and adopting a statistical method aiming at the pipeline to obtain the maximum value, the minimum value and the average value of all pigging time historical data, and the distribution interval and the occupied proportion of the pigging time;
and finally, comparing the ball sending time 9:30 and the ball receiving time 12:15 of the cleaning operation with various data obtained by analysis in historical data, finding that the time consumed by the cleaning operation is similar to the average cleaning time and belongs to the working condition of normal cleaning operation, and accordingly, judging that the cleaning operation of the long-distance natural gas pipeline is normal and the natural gas pipeline runs basically normally.
And comparing the data analysis conclusion of the pigging time with the actual working condition, and finding that the analysis conclusion basically accords with the actual working condition, so that the analysis result of the method is in accordance with the actual pigging operation data and the operation rule of the natural gas pipeline cleaner, and the method can be used as a basis for making a pigging scheme and can be used for guiding the field actual production. Therefore, the present invention has its necessity and utility.
The above description is only a preferred embodiment of the present invention, and should not be taken as limiting the invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (4)
1. A data warehouse for gas pipeline pigging operation and a data mining method are mainly used for extracting pigging operation data from other storage positions and loading the pigging operation data into the data warehouse, so that the integrality and consistency of the pigging operation data are realized, technical support is provided for developing data mining, and decision basis is provided for evaluating the safety of the gas pipeline pigging operation. The method mainly comprises the following steps:
step one, determining a data source of the natural gas pipeline pigging operation. The data source is the most original data stored in each business department, and is information record and calculation result generated by natural gas pipeline pigging operation, including work logs of each business department, databases of other software or systems, pigging operation record, design data of natural gas pipeline, and the like;
and step two, integrating data of the natural gas pipeline pigging operation. The data integration is to extract the data of the pigging operation from a data source by utilizing an ETL tool, load the data into a data warehouse according to a defined mapping relation, check the legality of the data in the loading process, and record a loading result into a working log of the data integration for query;
and step three, establishing a data warehouse for natural gas pipeline pigging operation. And establishing a data warehouse for the natural gas pipeline pigging operation based on the common data model and the software development standard. Adopting an Epicentre data model of an oil exploration and development integrated platform of an international POSC organization to construct a data warehouse of natural gas pipeline pigging operation;
and step four, extracting data of the natural gas pipeline pigging operation. The data extraction is to automatically extract data information in a data warehouse of natural gas pipeline pigging operation according to a defined data warehouse model and load the data information into data marts with different application themes according to a defined mapping relation;
and fifthly, performing data mart of the natural gas pipeline pigging operation. Data marts are collections of data built on the basis of the subject of the research or application in question, and are direct vehicles for data mining. The method comprises the steps of pressure data mart, flow data mart, pipe cleaner operation speed data mart, pigging time data mart, terrain change data mart and the like of natural gas pipeline pigging operation;
and sixthly, data mining of the natural gas pipeline pigging operation. Data mining is to extract relevant data from a data mart for analysis according to different research or application topics, and mine the knowledge hidden behind the data based on appropriate algorithms.
2. The data warehouse and the data mining method for gas pipeline pigging operation according to claim 1, characterized in that the ETL tool is used to extract the gas pipeline pigging operation data from the data source and load the data into the data warehouse, including the work logs of each business department, the database of other software or systems, the pigging operation records, the design data of the gas pipeline, etc.
3. The data warehouse and the data mining method for the gas pipeline pigging operation according to claim 1, wherein the data warehouse for the gas pipeline pigging operation is constructed based on an Epicentre data model and comprises a gas pipeline basic parameter data warehouse, a pigging operation state parameter data warehouse and a pipe cleaner parameter data warehouse.
4. The data warehouse and the data mining method for gas pipeline pigging according to claim 1, characterized in that based on different application themes and defining a mapping relation, pigging history data in the data warehouse are extracted and loaded into data marts, including 5 data marts of pressure, flow, terrain, pigging time, pig running speed, etc.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110694989.0A CN113254508B (en) | 2021-06-22 | 2021-06-22 | Data warehouse for natural gas pipeline pigging operation and data mining method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110694989.0A CN113254508B (en) | 2021-06-22 | 2021-06-22 | Data warehouse for natural gas pipeline pigging operation and data mining method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113254508A true CN113254508A (en) | 2021-08-13 |
| CN113254508B CN113254508B (en) | 2022-10-11 |
Family
ID=77189237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110694989.0A Active CN113254508B (en) | 2021-06-22 | 2021-06-22 | Data warehouse for natural gas pipeline pigging operation and data mining method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113254508B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115213179A (en) * | 2022-06-06 | 2022-10-21 | 东营汇聚丰石油科技有限公司 | High-speed cyclonic pipeline cleaning device and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105843880A (en) * | 2016-03-21 | 2016-08-10 | 中国矿业大学 | Coal mine multi-dimensional data warehousing system based on multiple data marts |
| CN106339509A (en) * | 2016-10-26 | 2017-01-18 | 国网山东省电力公司临沂供电公司 | Power grid operation data sharing system based on large data technology |
| US20170308796A1 (en) * | 2016-04-21 | 2017-10-26 | Utopus Insights, Inc. | System and method for forecasting leaks in a fluid-delivery pipeline network |
| CN107480263A (en) * | 2016-08-19 | 2017-12-15 | 北京市燃气集团有限责任公司 | The management method and management system of a kind of data resource |
| CN108418849A (en) * | 2018-01-09 | 2018-08-17 | 北京科东电力控制系统有限责任公司 | A kind of oil-gas pipeline SCADA system |
| CN108520006A (en) * | 2018-03-14 | 2018-09-11 | 广东能龙教育股份有限公司 | Data mining method based on pipeline screening |
| CN112765862A (en) * | 2021-01-31 | 2021-05-07 | 西南石油大学 | Method for calculating running parameters of pipe cleaner in pipe cleaning process of natural gas pipeline |
-
2021
- 2021-06-22 CN CN202110694989.0A patent/CN113254508B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105843880A (en) * | 2016-03-21 | 2016-08-10 | 中国矿业大学 | Coal mine multi-dimensional data warehousing system based on multiple data marts |
| US20170308796A1 (en) * | 2016-04-21 | 2017-10-26 | Utopus Insights, Inc. | System and method for forecasting leaks in a fluid-delivery pipeline network |
| CN107480263A (en) * | 2016-08-19 | 2017-12-15 | 北京市燃气集团有限责任公司 | The management method and management system of a kind of data resource |
| CN106339509A (en) * | 2016-10-26 | 2017-01-18 | 国网山东省电力公司临沂供电公司 | Power grid operation data sharing system based on large data technology |
| CN108418849A (en) * | 2018-01-09 | 2018-08-17 | 北京科东电力控制系统有限责任公司 | A kind of oil-gas pipeline SCADA system |
| CN108520006A (en) * | 2018-03-14 | 2018-09-11 | 广东能龙教育股份有限公司 | Data mining method based on pipeline screening |
| CN112765862A (en) * | 2021-01-31 | 2021-05-07 | 西南石油大学 | Method for calculating running parameters of pipe cleaner in pipe cleaning process of natural gas pipeline |
Non-Patent Citations (2)
| Title |
|---|
| KAMRAN GHANE: "Big Data Pipeline with ML-Based and Crowd Sourced Dynamically Created and Maintained Columnar Data Warehouse for Structured and Unstructured Big Data", 《2020 3RD INTERNATIONAL CONFERENCE ON INFORMATION AND COMPUTER TECHNOLOGIES (ICICT)》 * |
| 梁晓龙: "数字天然气管道的建设与管理初探", 《石油工业技术监督》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115213179A (en) * | 2022-06-06 | 2022-10-21 | 东营汇聚丰石油科技有限公司 | High-speed cyclonic pipeline cleaning device and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113254508B (en) | 2022-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104142984B (en) | It is a kind of to be based on thick fine-grained video fingerprint retrieval method | |
| CN101071098A (en) | Underground steel gas pipe network pipeline corrosion prediction system | |
| CN113254508B (en) | Data warehouse for natural gas pipeline pigging operation and data mining method | |
| CN109636092B (en) | Warehouse goods space allocation method based on double-factor optimization | |
| CN111814289B (en) | Water supply pipe network pipe burst analysis method and analysis system based on schema theory | |
| CN109754177A (en) | Pollution sources portrait label system, the building method of pollution sources portrait and its application | |
| CN106056252A (en) | Shale gas field well group division three-level optimization method | |
| CN102169491A (en) | Dynamic detection method for multi-data concentrated and repeated records | |
| CN102521347B (en) | Pattern matching intermediate result management method based on priority | |
| CN112418624A (en) | Method for establishing intelligent municipal pipe network operation evaluation index system and application system | |
| CN111046253A (en) | Internet-based interference-free geothermal heat supply monitoring system and method | |
| Li et al. | Set predicates in sql: Enabling set-level comparisons for dynamically formed groups | |
| CN106651040A (en) | Optimal layout method of urban waterlogging monitoring points | |
| CN110209742A (en) | One kind is based on block chain according to data importance classification storage system and method | |
| CN110245118A (en) | BIM data information three-dimensional gridding retrieval filing method and filing system thereof | |
| CN114385766A (en) | Natural gas pipeline data processing method based on HIVE and data warehouse system | |
| CN104318501A (en) | Pipeline network topological relation establishment method, device and system | |
| Phan-Luong | Searching data cube for submerging and emerging cuboids | |
| CN114428814A (en) | Self-service analysis modeling method for different-scale aggregation of large data set | |
| CN112765232B (en) | Data management method and system for major equipment of nuclear power plant | |
| CN104615773A (en) | Method of dynamically pushing product design documents on basis of parallel activation model | |
| CN114663000B (en) | Storage, transportation and intelligent terminal linked LNG management method and Internet of things system | |
| CN101937468A (en) | Hierarchical quantitative measurement service process-to-process distance system | |
| Li et al. | Research on key technologies of data warehouse construction in unified data center in electric power enterprises | |
| CN112966027B (en) | Entity association mining method based on dynamic probe |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |