CN112328667A - Shale gas field ground engineering digital handover method based on data blooding margin - Google Patents
Shale gas field ground engineering digital handover method based on data blooding margin Download PDFInfo
- Publication number
- CN112328667A CN112328667A CN202010693603.XA CN202010693603A CN112328667A CN 112328667 A CN112328667 A CN 112328667A CN 202010693603 A CN202010693603 A CN 202010693603A CN 112328667 A CN112328667 A CN 112328667A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- source
- structured
- consanguinity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000003860 storage Methods 0.000 claims abstract description 49
- 238000005516 engineering process Methods 0.000 claims abstract description 19
- 238000010276 construction Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 13
- 230000000007 visual effect Effects 0.000 claims abstract description 7
- 239000008280 blood Substances 0.000 claims description 41
- 210000004369 blood Anatomy 0.000 claims description 41
- 238000012384 transportation and delivery Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 238000013499 data model Methods 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 7
- 230000002688 persistence Effects 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 5
- 238000011161 development Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 abstract 1
- 238000011084 recovery Methods 0.000 abstract 1
- 238000012546 transfer Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
- G06F16/287—Visualization; Browsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/288—Entity relationship models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/02—Agriculture; Fishing; Forestry; Mining
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Marine Sciences & Fisheries (AREA)
- General Health & Medical Sciences (AREA)
- Agronomy & Crop Science (AREA)
- Mining & Mineral Resources (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Animal Husbandry (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a shale gas field ground engineering digital handover method based on a data consanguinity technology, which comprises a whole set of technical scheme, data consanguinity relation configuration, an interface packaging method, visual data consanguinity chart construction and the like in the processes of data acquisition, storage, service and result recovery, so that the whole data link is completely processed and controlled, the full life cycle of data is opened, a data closed loop is formed, and the quality and the service capacity of the data in the whole data link are continuously improved.
Description
Technical Field
The invention belongs to the field of data handover, and particularly relates to a shale gas field ground engineering digital handover method based on data blooding margin.
Background
In the traditional digital transfer of engineering construction, data related to a project are manually integrated, stage transfer is carried out according to project milestones, and transfer contents comprise various documents and drawings generated in the design/purchase/construction process and data/models distributed in various professional software. In the transfer process, the data provider and the data receiver need to check the integrity, consistency, correctness and timeliness of the transfer data together, and the transfer workload is huge.
When data problems occur, data sources need to be searched from massive documents through manual comparison, and the data problems are found, so that the efficiency is low, mistakes are easy to occur, and a large amount of manpower and material resources are wasted. If the data cannot be traced effectively, errors are accumulated gradually, and the normal production operation of the engineering object is influenced finally.
In practice of numerous projects, the data relationship tracing can often stay at a system or table level, the relationship is managed by a data flow mode, and the definition and the viewing performance of the relationship at a field level are not ideal.
The shale gas field ground engineering construction process involves a lot of specialties, and a large amount of data in different formats are generated in the design, construction and debugging processes and distributed on different professional software. In the period of centralized construction and data transfer, the project data transfer is complex and has large workload, and the collection, processing and storage of the project data need to be standardized and normalized so as to ensure the orderly execution of the transfer work and improve the production preparation efficiency. A unified platform is needed to collect data required by each specialty, unified standard management is carried out on bottom layer data, data specifications and management specifications are formulated, unified data outlets are provided for each upper layer application system in the future, the time cost of data retrieval is reduced, and data sharing and intercommunication among the specialties are achieved.
Disclosure of Invention
The invention provides a shale gas field ground engineering digital handover method based on data consanguinity aiming at the problems of complex data handover and large engineering quantity in the existing shale gas field ground engineering construction.
The specific implementation content of the invention is as follows:
the invention discloses a shale gas field ground engineering digital handover method based on data consanguinity, which is characterized in that before data acquisition, a React front-end technology is combined with a Spring Cloud micro-service technology to define the data consanguinity, the defined data consanguinity is stored in a database according to a Json protocol for persistence, and the defined relation is applied to a data acquisition layer to perform data analysis and is stored in a data warehouse; and then, acquiring data, registering and registering metadata of the acquired data, taking the metadata of the acquired data as input data of data relationship, and caching the acquired data.
In order to better implement the invention, further, the collected data comprises structured data, model data and unstructured data;
the metadata of the model data and the unstructured data comprise source information and structure information of the collected data; defining and configuring the blood relationship of the data flow direction through source information and structure information on a configuration interface for configuring the blood relationship of the data, and caching the received collected data according to an original structure;
the structural data definition data blood relationship is realized by two modes, and the specific method comprises the following steps:
the method comprises the following steps: making a data consanguinity relationship definition Excel template, and then carrying out batch data import on the data consanguinity relationship definition Excel template through uploading a data consanguinity relationship definition Excel template on a configuration interface for configuring data consanguinity relationship;
the second method comprises the following steps: defining and configuring the blood relationship of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field on a configuration interface for configuring the blood relationship of the data flow direction; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data.
In order to better implement the present invention, further, for the caching of the collected data, different modes of caching are performed according to different data classifications:
the structured data comprises structured data in a relational database, content data identified by tables and fields are acquired by relying on an open-source Kafka message queue, the data are packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4j database;
the model data comprises 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D general diagram three-dimensional model; model data is imported through a Kafka system, and actual model data is pulled out for storage through task scheduling according to Uniform Resource Locators (URLs) corresponding to the models in the metadata; storing the model data into a FastDFS distributed file system;
the unstructured data comprises engineering drawings, SP PID intelligent PID data drawings and data, Office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured files are stored into a FastDFS distributed file system.
In order to better implement the present invention, further, for the unstructured data, in the parsing and extracting process of the packet, the engineering object identification information associated with the unstructured data is also extracted, where the engineering object identification information is structured data information, and the engineering object identification information and the corresponding unstructured data are associated with each other in the structured database, so as to implement data linkage between the structured and unstructured data.
In order to better realize the invention, further, the acquired data is converted according to international standards, disordered dispersed data is formed into standard and universal engineering data, and the converted data is stored in a standard storage area to form a stable data structure; storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4Jj database; model data and unstructured files are stored into the FastDFS distributed file system.
The standard and universal engineering data are normalized through a data model, a service entity list, a service entity data structure table and a service entity document list which are established in advance; the data model, the service entity list, the service entity data structure table and the service entity document list need to meet the following requirements:
the service structure is combed in a tree directory hierarchical mode, and service entities are classified;
defining and configuring the logic category and classification of the asset service;
and defining and designing a data structure table for the physical storage layer.
In order to better implement the present invention, further, for the data stored in the standard storage area, the data is not directly pushed to the upper layer application for providing the data service, the data needs to be subjected to service conversion processing according to the service requirement in a service delivery manner, and the data is stored in the subject storage area for the upper layer application to use after the data requirement is subjected to service conversion; for some personalized applications, the personalized application needs to be subjected to personalized customization processing and then stored in a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization.
In order to better implement the invention, further, when the service delivery is carried out, a standard RESTful API is registered and an interface is opened in a service directory through a high-allocable and light-development mode based on an API gateway, so that the construction of a data service interface is realized; the constructed data service interfaces are combined and arranged to realize data services under different scenes; and the transmission object of the data service is used as one input of the data consanguinity relation.
In order to better realize the invention, further, after the upper layer application finishes using the data provided by the theme storage area or the customized storage area, the output result data is restored to the result restoring area, then the data stored in the result restoring area is subjected to standardized processing in a data rule base mode, so that the restored result data conforms to the specification definition of the international standard ISO 15926 and the enterprise data storage standard, and then is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
and recording data conversion of each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoration by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j database.
In order to better implement the invention, further, a structural data blood relationship map is generated by an Echarts technology for visual display; the structured data blood relationship graph uses metadata as a distinguishing basis of field level data blood relationship and uses a data table as a distinguishing basis of table level data blood relationship;
carrying out data consanguinity relation display on data acquired, stored, subjected to service delivery application, subjected to personalized customization application and subjected to result restoration by using the structured data consanguinity graph; and viewing the blood relationship contained in the structured data blood relationship graph according to the sequence of a source system, a source owner, a source field, a source table, a target field, a target owner and a target system.
In order to better realize the invention, further, the label setting is carried out on the unstructured data, the source and the stage of the unstructured data are refined, and the data label is set according to the format and the keywords of the deliverable, so that the data source can be rapidly searched and inquired.
Compared with the prior art, the invention has the following advantages and beneficial effects:
(1) the data blood relationship of field levels is checked through a visual data blood relationship map, so that the engineering data can be traced quickly and simply;
(2) unified data acquisition, unified standard management of bottom data, data specification and management specification formulation, unified data export for each upper application system in the future, reduction of data retrieval time cost, realization of professional data sharing intercommunication.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is an exemplary illustration of a configuration interface for downloading a data relationship definition Excel template in practice of the present invention;
FIG. 3 is an exemplary diagram of a configuration interface for data relationship definition metadata in accordance with the present invention;
FIG. 4 is a diagram illustrating an example of data relationship check in the practical operation of the present invention.
Detailed Description
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments, and therefore should not be considered as a limitation to the scope of protection. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
Example 1:
the invention discloses a shale gas field ground engineering digital handover method based on a blood-related technology, as shown in figure 1, firstly, data is required to be collected, the collected data comprises structured data, unstructured data, an industrial drawing, a model and the like, then, the collected data is cached and stored according to rules of data storage and rules of shale gas field business, and the different classified data are stored into different databases according to different classifications of the data in the storage, such as the structured data, the unstructured data and the like; for the collected data after being distinguished and stored, conversion processing is carried out according to international standards, external unordered scattered data form standardized and generalized engineering data, and the converted data enter a standard storage area to form a stable data structure; storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4j database; model data and unstructured files are stored into the FastDFS distributed file system. Then storing general, standard and steady data into a subject storage area or a customized storage area according to requirements for use by upper-layer application; after the upper-layer application uses the provided data, the result data can be output, data backtracking is carried out on the output result data through the blood relationship, data management such as quick retrieval and analysis is realized, and result backstorage or history isolation is carried out on the result data subjected to big data analysis.
The working principle is as follows: the invention overcomes the problems of large workload, low efficiency, easy error and difficult data tracing in the traditional data delivery by artificial integration, introduces a data consanguinity technology, realizes the communication with databases in a construction period and an operation period, receives data from design, purchase, construction, inspection and maintenance of cloud and infrastructure, performs asset storage and management, and flexibly responds to the data requirements of upper-layer intelligent application.
Example 2:
on the basis of the above embodiment 1, in order to better implement the present invention, further, in the data collection process, the receiving adapter receiving the data registers and registers the metadata of the collected data, and uses the metadata of the collected data as the input data of the data consanguinity; the metadata comprises source information and structure information of the acquired data; and caching the received collected data according to an original structure.
For the cache of the collected data, caching in different modes is carried out according to classification, and the concrete classification comprises the following steps: structured data, model data, unstructured data, real-time data;
the structured data comprises structured data in a relational database, content data identified by tables and fields are acquired by relying on an open-source Kafka message queue, the data are packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4j database;
processing the real-time data, converting the real-time data into structured data, and then storing the structured data in a MySQL database;
the model data comprises 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D general diagram three-dimensional model; model data is imported through a Kafka system, and actual model data is pulled out for storage through task scheduling according to Uniform Resource Locators (URLs) corresponding to the models in the metadata; storing the model data into a FastDFS distributed file system;
the unstructured data comprises engineering drawings, SP PID intelligent PID data drawings and data, Office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured files are stored into a FastDFS distributed file system.
The working principle is as follows: in the aspect of acquisition, aiming at data acquisition of heterogeneous sources, the method unifies the acquisition devices, encapsulates technical components and scheduling operation and provides the technical components and the scheduling operation for users; in the aspect of storage, aiming at the situations that functions of each layer of data storage are shared and database types are more, the method integrates data storage and unified calling in an object-oriented mode and provides an integrated storage and calling interface.
Other parts of this embodiment are the same as those of embodiment 1, and thus are not described again.
Example 3:
on the basis of any of the foregoing embodiments 1-2, in order to better implement the present invention, further, for the unstructured data, in the parsing and extracting process of the packet, the engineering object identification information associated with the unstructured data is also extracted at the same time, where the engineering object identification information is structured data information, and a correlation is established between the engineering object identification information and corresponding unstructured data in a structured database, so as to implement data linkage between structured and unstructured data.
Meanwhile, converting the acquired data according to international standards, forming standard and universal engineering data from unordered scattered data, storing the converted data in a standard storage area, forming a stable data structure, storing the structured data in a MySQL database, and storing the incidence relation of the structured data in a Neo4j database; model data and unstructured files are stored into the FastDFS distributed file system.
The standard and universal engineering data are normalized through a data model, a service entity list, a service entity data structure table and a service entity document list which are established in advance; the data model, the service entity list, the service entity data structure table and the service entity document list need to meet the following requirements:
(1) the service structure is combed in a tree directory hierarchical mode, and service entities are classified;
(2) defining and configuring the logic category and classification of the asset service;
(3) and defining and designing a data structure table for the physical storage layer.
Other parts of this embodiment are the same as any of embodiments 1-2 described above, and thus are not described again.
Example 4:
on the basis of any one of the embodiments 1 to 3, in order to better implement the present invention, further, for the data stored in the standard storage area, the data is not directly pushed to the upper application for providing the data service, the data needs to be subjected to service conversion processing in a service delivery manner according to the service requirement, and the data requirement is subjected to service conversion and then stored in the subject storage area for the upper application to use; for some personalized applications, the personalized application needs to be subjected to personalized customization processing and then stored in a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization; and the data of the standard storage area, the theme storage area and the customized storage area are structured data and are also stored in the MySQL database.
When the service is delivered, a standard RESTful API is registered and an interface is opened in a service directory on the basis of an API gateway in a high-availability and light-development mode, so that the construction of a data service interface is realized; the constructed data service interfaces are combined and arranged to realize data services under different scenes; and the transmission object of the data service is used as one input of the data consanguinity relation.
The working principle is as follows: in the aspect of service output, a micro service technology framework is utilized to encapsulate and modify a data interface into a micro service, and the micro service is exposed to an upper layer application by an API gateway to be called. At the service delivery level, the data service is provided by not performing customized development for each application, but constructing a data service interface in a highly-configurable and light-development mode. And registering and opening an interface in a service directory through a standard RESTful API in a mode based on an API gateway on the technical level. All standardized, generalized interfaces in the service directory can be combined and orchestrated to meet data services in different scenarios. The transmission object of the data service is constructed and integrated based on the logic entity in the data model and also serves as one of the important inputs of the data consanguinity relation
Other parts of this embodiment are the same as any of embodiments 1 to 3, and thus are not described again.
Example 5:
on the basis of any one of the above embodiments 1 to 4, in order to better implement the present invention, as shown in fig. 2, 3, and 4, further, after the upper layer application has used up the data provided by the subject storage area or the customized storage area, the output result data is restored to the result restoring area, and then the data stored in the result restoring area is standardized in the manner of the data rule base, so that the restored result data conforms to the specification definitions of the international standard ISO 15926 and the enterprise data storage standard, and then is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
and recording data conversion of each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoration by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j database.
Defining the data blooding margin relation by using a read front-end technology and combining a Spring Cloud micro-service technology, storing the defined data blooding margin relation into a database according to a Json protocol for persistence, applying the defined relation in a data acquisition layer for data analysis and storing the data analysis into a data warehouse;
the definition of the data blood relationship is realized by two modes, and the specific method is as follows:
the method comprises the following steps: making a data consanguinity relationship definition Excel template, and then conducting batch data import on a configuration interface for configuring data consanguinity relationship by uploading the data consanguinity relationship definition Excel template, wherein an interface schematic diagram in actual operation is shown in figure 2;
the second method comprises the following steps: defining and configuring the blood relationship of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field on a configuration interface for configuring the blood relationship of the data flow direction; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data; the schematic diagram of the interface in actual operation is shown in fig. 3.
Generating a structural data blood relationship graph by an Echarts technology for visual display; the structured data blood relationship graph uses metadata as a distinguishing basis of field level data blood relationship and uses a data table as a distinguishing basis of table level data blood relationship;
carrying out data consanguinity relation display on data acquired, stored, subjected to service delivery application, subjected to personalized customization application and subjected to result restoration by using the structured data consanguinity graph; viewing the blood relationship contained in the structural data blood relationship graph in sequence according to the sequence of a source system, a source owner, a source field, a source table, a target field, a target owner and a target system; the specific actual operation is shown in fig. 4, for example, in fig. 4, due to color limitation, a red color ball in the original image is shown as black in fig. 4, and when the mouse is hovered on a node of the circular graph in actual operation, the blood relationship between the corresponding field and the data table can be independently checked; and clicking a drop-down box for selecting the target field to check the blood relationship corresponding to the selected target field.
In fig. 4, the small red color balls represent data fields of the engineering entity, the large red color balls represent the engineering entity data table corresponding to the fields, and the arrows represent the data flow direction from the source fields to the source table, the target table and the target fields in sequence; the data blood margin graph provides a zooming function, and the relationship of the blood margin can be zoomed in or zoomed out according to the overall perception or local focusing positioning requirement.
And performing labeling setting on the unstructured data, refining the source and the stage of the unstructured data, and setting a data label according to the format and the keywords of the deliverable so as to quickly retrieve and inquire the data source.
The working principle is as follows: the method comprises the steps of defining the data flow direction relation between external data and internal data and between the internal data in a visual mode, using a read front-end technology in combination with a Spring Cloud micro-service technology, storing the blood relationship defined by the front end into a database according to a Json protocol for persistence, applying the defined relation in a data acquisition layer to analyze the data and store the data into a data warehouse, and displaying the mapping of the data flow direction through a visual data blood relationship graph. In the process of using the platform to perform data handover, if data tracing needs to be performed, the data consanguinity relation of field levels can be checked through the visualized data consanguinity map, and a simple method is provided for data tracing of engineering construction. After the upper application uses the data provided by the platform, the result data of the output comprehensive research is restored to the result restoring area of the platform, and then the data is standardized according to the mode of the data rule base, so that the data conforms to the ISO 15926 and the standard definition of the enterprise data storage standard and finally enters the standard storage area; the method is characterized in that the change of data is recorded based on an information processing link of a data blood margin technology, whether the processing relation among the data is reasonable or not can be analyzed, and the influence of the change of upstream data on the downstream is analyzed; the source of the upstream problem is tracked as downstream data changes, etc. The data consanguinity relation analysis of the table level and the field level clearly positions the incidence relations of mapping, calculation, circulation and the like among different engineering entities, the analysis precision is high, and the method has a wider application prospect.
Other parts of this embodiment are the same as any of embodiments 1 to 4, and thus are not described again.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and all simple modifications and equivalent variations of the above embodiments according to the technical spirit of the present invention are included in the scope of the present invention.
Claims (10)
1. A shale gas field ground engineering digital handover method based on data consanguinity is characterized in that before data acquisition, a React front-end technology is used for defining data consanguinity relations in combination with a Spring Cloud micro-service technology, the defined data consanguinity relations are stored in a database according to a Json protocol for persistence, and data analysis is performed on the defined relations in a data acquisition layer and stored in a data warehouse; and then, acquiring data, registering and registering metadata of the acquired data, taking the metadata of the acquired data as input data of data relationship, and caching the acquired data.
2. The method for the digital handover of the shale gas field ground engineering based on the data blooding margin as claimed in claim 1, wherein the collected data comprises structured data, model data, unstructured data;
the metadata of the model data and the unstructured data comprise source information and structure information of the collected data; defining and configuring the blood relationship of the data flow direction through source information and structure information on a configuration interface for configuring the blood relationship of the data, and caching the received collected data according to an original structure;
the structural data definition data blood relationship is realized by two modes, and the specific method comprises the following steps:
the method comprises the following steps: making a data consanguinity relationship definition Excel template, and then carrying out batch data import on the data consanguinity relationship definition Excel template through uploading a data consanguinity relationship definition Excel template on a configuration interface for configuring data consanguinity relationship;
the second method comprises the following steps: defining and configuring the blood relationship of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field on a configuration interface for configuring the blood relationship of the data flow direction; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data.
3. The shale gas field ground engineering digital handover method based on the data blooding margin as claimed in claim 2, wherein for the caching of the collected data, different modes of caching are performed according to different data classifications:
the structured data comprises structured data in a relational database, content data identified by tables and fields are acquired by relying on an open-source Kafka message queue, the data are packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4j database;
the model data comprises 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D general diagram three-dimensional model; model data is imported through a Kafka system, and actual model data is pulled out for storage through task scheduling according to Uniform Resource Locators (URLs) corresponding to the models in the metadata; storing the model data into a FastDFS distributed file system;
the unstructured data comprises engineering drawings, SP PID intelligent PID data drawings and data, Office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured files are stored into a FastDFS distributed file system.
4. The shale gas field ground engineering digital handover method based on the data bloodletting of claim 3, wherein for the unstructured data, in the parsing and extracting process of the message, engineering object identification information associated with the unstructured data is also extracted, wherein the engineering object identification information is structured data information, and the association between the engineering object identification information and the corresponding unstructured data is established in a structured database, so that data connection between the structured data and the unstructured data is realized.
5. The shale gas field ground engineering digital handover method based on the data blooding margin as claimed in claim 1, wherein the collected data is converted according to international standards, unordered scattered data is formed into standard and universal engineering data, and the converted data is stored in a standard storage area to form a stable data structure; storing the structured data into a MySQL database, and storing the incidence relation of the structured data into a Neo4j database; model data and unstructured files are stored into the FastDFS distributed file system.
The standard and universal engineering data are normalized through a data model, a service entity list, a service entity data structure table and a service entity document list which are established in advance; the data model, the service entity list, the service entity data structure table and the service entity document list need to meet the following requirements:
(1) the service structure is combed in a tree directory hierarchical mode, and service entities are classified;
(2) defining and configuring the logic category and classification of the asset service;
(3) and defining and designing a data structure table for the physical storage layer.
6. The shale gas field ground engineering digital handover method based on data consanguinity as claimed in claim 5, wherein for the data stored in the standard storage area, the data is not directly pushed to the upper application for providing data service, the data needs to be subjected to service conversion processing in a service delivery mode according to service requirements, and the data requirements are subjected to service conversion and then stored in the subject storage area for the upper application to use; for some personalized applications, the personalized application needs to be subjected to personalized customization processing and then stored in a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization.
7. The digital shale gas field ground engineering handover method based on the data blooding margin as claimed in claim 6, wherein, when the service delivery is carried out, a standard RESTful API is registered and an interface is opened in a service directory through a high-availability and light-development mode based on an API gateway, so as to realize the construction of a data service interface; the constructed data service interfaces are combined and arranged to realize data services under different scenes; and the transmission object of the data service is used as one input of the data consanguinity relation.
8. The shale gas field ground engineering digital handover method based on data blooding margin as claimed in claim 7, wherein after the upper layer application uses up the data provided by the theme storage area or the customized storage area, the output result data is restored to the result restoration area, and then the data stored in the result restoration area is standardized in the form of a data rule base, so that the restored result data conforms to the specification definition of international standard ISO 15926 and the enterprise data storage standard, and then is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
and recording data conversion of each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoration by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j database.
9. The method for the digital handover of the shale gas field ground engineering based on the data blooding margin of claim 8, wherein a structured data blooding margin map is generated by an Echarts technology for visual display; the structured data blood relationship graph uses metadata as a distinguishing basis of field level data blood relationship and uses a data table as a distinguishing basis of table level data blood relationship;
carrying out data consanguinity relation display on data acquired, stored, subjected to service delivery application, subjected to personalized customization application and subjected to result restoration by using the structured data consanguinity graph; and viewing the blood relationship contained in the structured data blood relationship graph according to the sequence of a source system, a source owner, a source field, a source table, a target field, a target owner and a target system.
10. The data-based bloody margin shale gas field ground engineering digital handover method as claimed in claim 9, wherein, the label setting is performed on the unstructured data, the source and the stage of the unstructured data are refined, and the data label is set according to the format and the keywords of the deliverable, so as to facilitate the rapid retrieval and query of the data source.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010693603.XA CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010693603.XA CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112328667A true CN112328667A (en) | 2021-02-05 |
| CN112328667B CN112328667B (en) | 2023-09-08 |
Family
ID=74303748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010693603.XA Active CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112328667B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145919A (en) * | 2022-06-30 | 2022-10-04 | 中冶赛迪信息技术(重庆)有限公司 | Method, device, equipment and medium for generating data blood relationship between service systems |
| CN115687309A (en) * | 2022-12-30 | 2023-02-03 | 浙江大学 | Non-invasive cigarette warehouse-in and warehouse-out full-process data blood margin construction method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN109286653A (en) * | 2017-07-21 | 2019-01-29 | 埃森哲环球解决方案有限公司 | Intelligent cloud engineering platform |
| CN109446279A (en) * | 2018-10-15 | 2019-03-08 | 顺丰科技有限公司 | Based on neo4j big data genetic connection management method, system, equipment and storage medium |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
| CN111400305A (en) * | 2020-02-20 | 2020-07-10 | 深圳市魔数智擎人工智能有限公司 | Characteristic engineering blood relationship based backtracking and visualization method |
-
2020
- 2020-07-17 CN CN202010693603.XA patent/CN112328667B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN109286653A (en) * | 2017-07-21 | 2019-01-29 | 埃森哲环球解决方案有限公司 | Intelligent cloud engineering platform |
| CN109446279A (en) * | 2018-10-15 | 2019-03-08 | 顺丰科技有限公司 | Based on neo4j big data genetic connection management method, system, equipment and storage medium |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
| CN111400305A (en) * | 2020-02-20 | 2020-07-10 | 深圳市魔数智擎人工智能有限公司 | Characteristic engineering blood relationship based backtracking and visualization method |
Non-Patent Citations (2)
| Title |
|---|
| KERAMAT HASSANI 等: "An Approach to Tracking Data Lineage in Mediator Based Information Integration Systems" * |
| 张新阳 等: "基于电力行业的大数据时代下元数据管理方法" * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145919A (en) * | 2022-06-30 | 2022-10-04 | 中冶赛迪信息技术(重庆)有限公司 | Method, device, equipment and medium for generating data blood relationship between service systems |
| CN115687309A (en) * | 2022-12-30 | 2023-02-03 | 浙江大学 | Non-invasive cigarette warehouse-in and warehouse-out full-process data blood margin construction method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112328667B (en) | 2023-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111435344B (en) | Big data-based drilling acceleration influence factor analysis model | |
| CN110781236A (en) | Method for constructing government affair big data management system | |
| CN111611458B (en) | Method for realizing system data architecture carding based on metadata and data analysis technology in big data processing | |
| US8484255B2 (en) | Automatic conversion of multidimentional schema entities | |
| CN109344223A (en) | A kind of Building Information Model management system and method based on cloud computing technology | |
| CN112651126B (en) | BIM-based construction engineering construction optimization system, method, terminal and storage medium | |
| CN111176867B (en) | Data sharing exchange and open application platform | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN111400393B (en) | Data processing method and device based on multi-application platform and storage medium | |
| CN112633822B (en) | Asset management method based on digital twin technology, storage medium and mobile terminal | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN112559351B (en) | CFD software verification and confirmation database platform | |
| CN111708774B (en) | Industry analytic system based on big data | |
| CN112328667B (en) | Shale gas field ground engineering digital handover method based on data blood margin | |
| US10397326B2 (en) | IRC-Infoid data standardization for use in a plurality of mobile applications | |
| CN112347071A (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN114254033A (en) | Data processing method and system based on BS architecture | |
| WO2021056854A1 (en) | Internet novel virtual data center system and construction method therefor | |
| CN114817256A (en) | Quick unified storage system of thing networking | |
| CN113342874A (en) | Wind power big data analysis system and process based on cloud computing | |
| CN111813555B (en) | Super-fusion infrastructure layered resource management system based on internet technology | |
| Shi et al. | Automated geoprocessing mechanism, processes and workflow for seamless online integration of geodata services and creating geoprocessing services | |
| CN111104558A (en) | Distributed multi-source data processing method and system | |
| US11983187B2 (en) | Global status monitoring for open data platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |