CN112328667B - Shale gas field ground engineering digital handover method based on data blood margin - Google Patents
Shale gas field ground engineering digital handover method based on data blood margin Download PDFInfo
- Publication number
- CN112328667B CN112328667B CN202010693603.XA CN202010693603A CN112328667B CN 112328667 B CN112328667 B CN 112328667B CN 202010693603 A CN202010693603 A CN 202010693603A CN 112328667 B CN112328667 B CN 112328667B
- Authority
- CN
- China
- Prior art keywords
- data
- blood
- service
- source
- unstructured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000008280 blood Substances 0.000 title claims abstract description 48
- 210000004369 blood Anatomy 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000003860 storage Methods 0.000 claims abstract description 50
- 238000005516 engineering process Methods 0.000 claims abstract description 20
- 238000010276 construction Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 13
- 238000010586 diagram Methods 0.000 claims abstract description 9
- 230000000007 visual effect Effects 0.000 claims abstract description 8
- 238000012384 transportation and delivery Methods 0.000 claims description 16
- 238000006243 chemical reaction Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 9
- 238000013499 data model Methods 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 7
- 230000002688 persistence Effects 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000004519 manufacturing process Methods 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000011161 development Methods 0.000 claims description 2
- 238000004806 packaging method and process Methods 0.000 abstract 1
- 238000007726 management method Methods 0.000 description 6
- 238000012546 transfer Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
- G06F16/287—Visualization; Browsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/288—Entity relationship models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/02—Agriculture; Fishing; Forestry; Mining
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Marine Sciences & Fisheries (AREA)
- General Health & Medical Sciences (AREA)
- Agronomy & Crop Science (AREA)
- Mining & Mineral Resources (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Animal Husbandry (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data blood edge technology-based shale gas field ground engineering digital handover method, which comprises a whole set of technical scheme, data blood edge relation configuration, interface packaging method, visual data blood edge diagram construction and the like in the processes of data acquisition, storage, service and achievement restoring, so that the whole data link is completely processed and managed, the whole life cycle of the data is opened, and a data closed loop is formed, thereby continuously improving the quality and service capability of the data in the whole data link.
Description
Technical Field
The invention belongs to the field of data handover, and particularly relates to a shale gas field ground engineering digital handover method based on data blood edges.
Background
The traditional engineering construction digital handover integrates the data related to the project artificially, carries out phase handover according to project milestones, and the handover content comprises various documents, drawings and data/models distributed in various professional software generated in the design/purchase/construction process. The data provider and the data receiver need to check the integrity, consistency, correctness and timeliness of the handover data together in the handover process, and the handover workload is huge.
When the data problem occurs, the data source is required to be searched from the mass documents through manual comparison, the data problem is found, the efficiency is low, errors are easy to occur, and a large amount of manpower and material resources are wasted. If the data cannot be effectively traced, errors are gradually accumulated, and normal production and operation of the engineering object are finally affected.
In many project practices, tracing back data relationships often resides at the system or table level, managing relationships by way of data flow, and defining and viewing relationships at the field level is not ideal.
The shale gas field ground engineering construction process involves a lot of professions, generates a large amount of data with different formats in the design, construction and debugging processes, and is distributed on different professional software. In the period of centralized construction and data transfer, engineering data transfer is complex and has large workload, and standardized and normalized engineering data acquisition, processing and storage are required, so that orderly transfer work is ensured, and production preparation efficiency is improved. A unified platform is needed to collect data required by each specialty, unified standard management is carried out on the bottom data, data specifications and management specifications are formulated, unified data outlets are provided for each upper application system in the future, the data retrieval time cost is reduced, and data sharing intercommunication among the specialty is realized.
Disclosure of Invention
Aiming at the problems that the data transfer is complex and the engineering quantity is large in the conventional shale gas field ground engineering construction, the invention provides a shale gas field ground engineering digital transfer method based on data blood margin, and the data blood margin technology is used for making blood margin relations to realize unified management of data through metadata setting blood margin relations.
The invention has the following specific implementation contents:
the invention discloses a data blood margin-based shale gas field ground engineering digital handover method, which comprises the steps of defining a data blood margin relation by using a reaction front-end technology and a Spring Cloud micro-service technology before data acquisition, storing the defined data blood margin relation into a database according to a Json protocol for persistence, applying the defined relation to a data acquisition layer for data analysis and storing the data analysis into a data warehouse; and then collecting data, registering and registering metadata of the collected data, taking the metadata of the collected data as input data of a data blood relationship, and caching the collected data.
In order to better implement the invention, further, the collected data comprises structured data, model data and unstructured data;
the metadata of the model data and the unstructured data comprise source information and structure information of the collected data; defining and configuring the blood-edge relation of the data flow direction through source information and structure information on a configuration interface for configuring the blood-edge relation of the data, and caching the received acquired data according to an original structure;
the structural data definition data blood-edge relation is realized in two ways, and the specific method is as follows:
the method comprises the following steps: manufacturing a data blood relationship definition Excel template, and then carrying out batch data import on a configuration interface for configuring the data blood relationship through uploading the data blood relationship definition Excel template;
the second method is as follows: defining and configuring the blood-edge relation of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field in a configuration interface for configuring the blood-edge relation of the data; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data.
In order to better realize the invention, further, for the caching of the acquired data, different modes of caching are carried out according to different data classifications:
the structured data comprises structured data in a relational database, content data identified through tables and fields is collected by means of an open-source Kafka message queue, the data is packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database;
the model data comprise 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D total image three-dimensional model; importing model data through a Kafka system, and pulling out actual model data by task scheduling according to a Uniform Resource Locator (URL) corresponding to the model in the metadata for storage; storing the model data in a FastDFS distributed file system;
the unstructured data comprise engineering drawings, SP PID intelligent PID data drawings and data, office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured file is stored into the FastDFS distributed file system.
In order to better realize the invention, further, for the unstructured data, in the process of analyzing and extracting the message, engineering object identification information associated with the unstructured data is extracted, the engineering object identification information is structured data information, and the association between the engineering object identification information and the corresponding unstructured data is established in a structured database, so that the data link between the structured data and the unstructured data is realized.
In order to better realize the invention, further, the collected data is converted according to the international standard, unordered scattered data is formed into standard and general engineering data, and the converted data is stored in a standard storage area to form a steady-state data structure; storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4Jj graph database; model data and unstructured files are stored into the FastDFS distributed file system.
The standard and general engineering data are normalized through a data model, a service entity list, a service entity data structure list and a service entity document list which are established in advance; the data model, the business entity list, the business entity data structure list and the business entity document list are required to meet the following requirements:
the service structure is carded in a hierarchical mode of the tree directory, and the service entities are classified;
defining and configuring logical categories and classifications of asset services;
and defining and designing a data structure table for the physical storage layer.
In order to better realize the invention, further, for the data stored in the standard storage area, the data is not directly pushed to the upper layer application to provide data service, the data is required to be subjected to service conversion processing according to service requirements in a service delivery mode, and the data requirements are stored into the upper layer application of the theme storage area for use after service conversion; the personalized application is required to be personalized and customized and then stored into a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization.
In order to better realize the invention, further, when the service delivery is carried out, the construction of a data service interface is realized by registering and opening an interface in a service catalog through a standard RESTful API based on an API gateway in a highly configurable and lightly developed mode; realizing the data service under different scenes by combining and arranging the constructed data service interfaces; and takes the transmission object of the data service as one input of the data blood relationship.
In order to better realize the invention, further, after the upper layer application uses the data provided by the theme storage area or the customized storage area, the output result data is restored to the result restoring area, and then the data stored in the result restoring area is standardized in a data rule base mode, so that the restored result data accords with the standard definition of the international standard ISO 159226 and the enterprise data storage standard and is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
the method comprises the steps of recording data conversion in each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoring by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j graph database.
In order to better realize the invention, further, the structured data blood-lineage diagram is generated by the Echarts technology for visual display; the structural data blood-edge map uses metadata as a distinguishing basis of field-level data blood-edge relations and uses a data table as a distinguishing basis of table-level data blood-edge relations;
carrying out data blood relationship display on the collected, stored, service delivery application, personalized customization application and result restoring data by using the structured data blood relationship graph; and the blood-edge relation contained in the structured data blood-edge graph is checked sequentially according to the sequence of the source system, the source owner, the source field, the source table, the target field, the target owner and the target system.
In order to better realize the invention, the unstructured data is further subjected to labeling, sources and phases of the unstructured data are refined, and data labels are set according to the format of the deliveries and keywords, so that the sources of the data can be quickly searched and queried.
Compared with the prior art, the invention has the following advantages:
(1) The data blood-source relation of the field level is checked through the visual data blood-source map, so that quick and simple tracing of engineering data is realized;
(2) Unified data acquisition, unified standard management is carried out on the bottom data, data specifications and management specifications are formulated, unified data outlets are provided for various upper application systems in the future, the data retrieval time cost is reduced, and data sharing and intercommunication among professions are realized.
Drawings
FIG. 1 is a complete flow chart of the data processing of the present invention;
FIG. 2 is a diagram illustrating an exemplary configuration interface for downloading a data blood relationship definition Excel template in the actual operation of the present invention;
FIG. 3 is a diagram illustrating an exemplary configuration interface for defining metadata for a data blood relationship in an actual operation of the present invention;
fig. 4 is a diagram showing an example of data blood-edge relationship viewing in the actual operation of the present invention.
Detailed Description
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it should be understood that the described embodiments are only some embodiments of the present invention, but not all embodiments, and therefore should not be considered as limiting the scope of protection. All other embodiments, which are obtained by a worker of ordinary skill in the art without creative efforts, are within the protection scope of the present invention based on the embodiments of the present invention.
Example 1:
the invention discloses a shale gas field ground engineering digital handover method based on a blood-margin technology, as shown in fig. 1, firstly, data are required to be collected, the collected data comprise structured data, unstructured data, an industrial drawing, a model and the like, then the collected data are cached and stored according to a rule of storing the data and a rule of shale gas field business, and in the storage, the data of different classifications are stored into different databases according to different classifications of the data, such as the structured data, the unstructured data and the like; for the collected data after the storage is distinguished, conversion processing is carried out according to international standards, the external unordered scattered data is formed into standardized and generalized engineering data, and the converted data enters a standard storage area to form a steady-state data structure; storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database; model data and unstructured files are stored into the FastDFS distributed file system. Then, the general, standard and steady data are stored in a theme storage area or a custom storage area for upper application according to the requirements; after the provided data are used by the upper layer application, the result data are output, the output result data are traced back through blood relationship, data management such as quick search analysis is realized, and the result data after big data analysis are restored or history isolation is carried out.
Working principle: the invention solves the problems of large workload, low efficiency, easy error and difficult data tracing when the traditional data delivery is carried out by artificial integration, introduces a data blood source technology, realizes the communication with databases in a construction period and an operation period, receives data from the cloud and infrastructure such as design, purchase, construction, maintenance and the like, carries out asset storage and management, and flexibly responds to the data requirements of upper intelligent application.
Example 2:
in order to better realize the invention, on the basis of the embodiment 1, further, in the process of collecting data, the receiving adapter for receiving the data registers and registers metadata of the collected data, and takes the metadata of the collected data as input data of data blood edges; the metadata comprises source information and structure information of the acquired data; and caching the received acquisition data according to an original structure.
For the caching of the collected data, caching in different modes is carried out according to classification, wherein the specific classification comprises: structured data, model data, unstructured data, real-time data;
the structured data comprises structured data in a relational database, content data identified through tables and fields is collected by means of an open-source Kafka message queue, the data is packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database;
processing the real-time data, converting the real-time data into structured data, and storing the structured data in a MySQL database;
the model data comprise 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D total image three-dimensional model; importing model data through a Kafka system, and pulling out actual model data by task scheduling according to a Uniform Resource Locator (URL) corresponding to the model in the metadata for storage; storing the model data in a FastDFS distributed file system;
the unstructured data comprise engineering drawings, SP PID intelligent PID data drawings and data, office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured file is stored into the FastDFS distributed file system.
Working principle: in the aspect of collection, aiming at heterogeneous data collection, the method unifies a collector, encapsulates technical components and scheduling operation and provides the technical components and scheduling operation for users; in the aspect of storage, aiming at the situation that each layer of functions of data storage are shared respectively and the database types are more, the method integrates the data storage and unified call in an object-oriented manner and provides an integrated storage and call interface.
Other portions of this embodiment are the same as those of embodiment 1 described above, and thus will not be described again.
Example 3:
in order to better implement the present invention on the basis of any one of the above embodiments 1-2, further, in the parsing and extracting process of the message, the engineering object identification information associated with the unstructured data is also extracted, the engineering object identification information is structured data information, and an association is established between the engineering object identification information and the corresponding unstructured data in the structured database, so as to implement data connection between the structured data and the unstructured data.
Meanwhile, converting the collected data according to international standards, forming unordered scattered data into standard and general engineering data, storing the converted data into a standard storage area, forming a steady-state data structure, storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database; model data and unstructured files are stored into the FastDFS distributed file system.
The standard and general engineering data are normalized through a data model, a service entity list, a service entity data structure list and a service entity document list which are established in advance; the data model, the business entity list, the business entity data structure list and the business entity document list are required to meet the following requirements:
(1) The service structure is carded in a hierarchical mode of the tree directory, and the service entities are classified;
(2) Defining and configuring logical categories and classifications of asset services;
(3) And defining and designing a data structure table for the physical storage layer.
Other portions of this embodiment are the same as any of embodiments 1-2 described above, and thus will not be described again.
Example 4:
on the basis of any one of the above embodiments 1 to 3, in order to better implement the present invention, further, for the data stored in the standard storage area, the data is not directly pushed to the upper layer application to provide data service, and the data needs are subjected to service conversion processing according to service requirements in a service delivery mode, and then are stored into the upper layer application of the subject storage area for use after service conversion; the personalized application is required to be personalized and customized and then stored into a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization; the data of the standard storage area, the theme storage area and the custom storage area are structured data and are also stored in the MySQL database.
When the service delivery is carried out, the construction of a data service interface is realized by registering and opening an interface in a service catalog through a standard RESTful API based on an API gateway in a highly configurable and lightly developed mode; realizing the data service under different scenes by combining and arranging the constructed data service interfaces; and takes the transmission object of the data service as one input of the data blood relationship.
Working principle: in terms of service output, the data interface is packaged and modified into micro service by using a micro service technology framework, and the micro service is exposed to an upper layer application to be called in a mode of an API gateway. At the service delivery level, rather than custom developing for each application to provide data services, the data service interface is built through a highly configurable, lightweight development approach. The technical level is based on the manner of an API gateway, and the interfaces are registered and opened in a service directory through a standard RESTful API. All standardized, generalized interfaces in the service directory can be combined and orchestrated to meet the data services in different scenarios. The transmission object of the data service is constructed and integrated based on the logic entity in the data model and is also used as one of the important inputs of the data blood relationship
Other portions of this embodiment are the same as any of embodiments 1 to 3 described above, and thus will not be described again.
Example 5:
in order to better implement the present invention on the basis of any one of the above embodiments 1 to 4, as shown in fig. 2, 3 and 4, further, after the upper layer application uses the data provided by the theme storage area or the customized storage area, the output result data is restored to the result restoring area, and then the data stored in the result restoring area is standardized in a data rule base manner, so that the restored result data accords with the specification definition of international standard ISO 15926 and enterprise data storage standard, and then is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
the method comprises the steps of recording data conversion in each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoring by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j graph database.
Defining a data blood-edge relationship by using a reaction front-end technology and a Spring Cloud micro-service technology, storing the defined data blood-edge relationship into a database according to a Json protocol for persistence, and carrying out data analysis by applying the defined relationship to a data acquisition layer and storing the data analysis into a data warehouse;
the definition of the data blood-edge relationship is realized in two ways, and the specific method is as follows:
the method comprises the following steps: preparing a data blood edge relationship definition Excel template, and then carrying out batch data import on a configuration interface for configuring the data blood edge relationship through uploading the data blood edge relationship definition Excel template, wherein an interface schematic diagram in actual operation is shown in fig. 2;
the second method is as follows: defining and configuring the blood-edge relation of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field in a configuration interface for configuring the blood-edge relation of the data; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data; in actual operation, the interface is schematically shown in fig. 3.
Generating a structured data blood-lineage diagram through Echarts technology for visual display; the structural data blood-edge map uses metadata as a distinguishing basis of field-level data blood-edge relations and uses a data table as a distinguishing basis of table-level data blood-edge relations;
carrying out data blood relationship display on the collected, stored, service delivery application, personalized customization application and result restoring data by using the structured data blood relationship graph; the blood-edge relation contained in the structured data blood-edge graph is checked sequentially according to the sequence of a source system, a source owner, a source field, a source table, a target field, a target owner and a target system; specific practical operation examples are shown in fig. 4, and in fig. 4, due to color limitation, a red color ball in an original image is shown as black in fig. 4, and a mouse is hovered above a circular graph node in practical operation, so that the blood relationship between corresponding fields and a data table can be independently checked; clicking the drop-down box of the selected target field can check the blood-edge relation corresponding to the selected target field.
The small red color balls in fig. 4 represent the data fields of the engineering entities, the large red color balls represent the engineering entity data tables corresponding to the fields, and the arrows represent the data flow directions from the source field to the source table, the target table and the target field in sequence; the data blood-edge map provides a scaling function, and the blood-edge relation is enlarged or reduced to be checked according to the whole perception or local focusing positioning requirement.
And (3) carrying out labeling setting on unstructured data, refining sources and stages of the unstructured data, and setting data labels according to formats and keywords of deliveries so as to facilitate quick retrieval and query of the sources of the data.
Working principle: defining the relation of the data flow direction between external data and internal data and between the internal data in a visual mode, storing the blood-edge relation defined by the front end into a database for persistence according to a Json protocol by combining a real front end technology with a Spring Cloud micro-service technology, analyzing the data by applying the defined relation to a data acquisition layer and storing the data in a data warehouse, and displaying the mapping of the data flow direction through a visual data blood-edge diagram. In the process of data transfer by using the platform, if data tracing is needed, the field-level data blood-edge relationship can be checked through the visual data blood-edge map, so that a simple method is provided for engineering construction data tracing. After the upper layer application uses the data provided by the platform, restoring the result data of the output comprehensive study to a result restoring area of the platform, and carrying out standardized processing on the result data according to a data rule base mode to ensure that the result data accords with the standard definition of the ISO 15926 and the enterprise data storage standard and finally enters a standard storage area; the information processing link based on the data blood-edge technology records the change of the data, so that whether the processing relationship between the data is reasonable or not can be analyzed, and the influence of the change of the upstream data on the downstream can be analyzed; tracking the source of upstream problems as downstream data changes, etc. The table level and field level data blood relationship analysis clearly locates the association relationship of mapping, calculation, circulation and the like among different engineering entities, has high analysis precision and wider application prospect.
Other portions of this embodiment are the same as any of embodiments 1 to 4 described above, and thus will not be described again.
The foregoing description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and any simple modification, equivalent variation, etc. of the above embodiment according to the technical matter of the present invention fall within the scope of the present invention.
Claims (7)
1. The data blood margin-based shale gas field ground engineering digital handover method is characterized in that before data acquisition, a reaction front-end technology is combined with a Spring Cloud micro-service technology to define a data blood margin relation, the defined data blood margin relation is stored into a database according to a Json protocol for persistence, and a defined relation is applied to a data acquisition layer for data analysis and storage in a data warehouse; then, data acquisition is carried out, metadata of acquired data are registered and registered, the metadata of the acquired data are used as input data of a data blood-edge relationship, and the acquired data are cached;
the collected data comprises structured data, model data and unstructured data;
the metadata of the model data and the unstructured data comprise source information and structure information of the collected data; defining and configuring the blood-edge relation of the data flow direction through source information and structure information on a configuration interface for configuring the blood-edge relation of the data, and caching the received acquired data according to an original structure;
the structural data definition data blood-edge relation is realized in two ways, and the specific method is as follows:
the method comprises the following steps: manufacturing a data blood relationship definition Excel template, and then carrying out batch data import on a configuration interface for configuring the data blood relationship through uploading the data blood relationship definition Excel template;
the second method is as follows: defining and configuring the blood-edge relation of the data flow direction through a source system, a source owner, a source table, a source field, a target system, a target owner, a target table and a target field in a configuration interface for configuring the blood-edge relation of the data; the source system, the source owner, the source table, the source field, the target system, the target owner, the target table and the target field are metadata of the structured data;
converting the collected data according to international standard, forming unordered scattered data into standard and general engineering data, and storing the converted data into a standard storage area to form a steady-state data structure; storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database; storing the model data and unstructured files in a FastDFS distributed file system;
the standard and general engineering data are normalized through a data model, a service entity list, a service entity data structure list and a service entity document list which are established in advance; the data model, the business entity list, the business entity data structure list and the business entity document list are required to meet the following requirements:
(1) The service structure is carded in a hierarchical mode of the tree directory, and the service entities are classified;
(2) Defining and configuring logical categories and classifications of asset services;
(3) Defining and designing a data structure table for a physical storage layer;
generating a structured data blood-lineage diagram through Echarts technology for visual display; the structural data blood-edge map uses metadata as a distinguishing basis of field-level data blood-edge relations and uses a data table as a distinguishing basis of table-level data blood-edge relations;
carrying out data blood relationship display on the collected, stored, service delivery application, personalized customization application and result restoring data by using the structured data blood relationship graph; and the blood-edge relation contained in the structured data blood-edge graph is checked sequentially according to the sequence of the source system, the source owner, the source field, the source table, the target field, the target owner and the target system.
2. The data blood-based shale gas field surface engineering digital handover method as claimed in claim 1, wherein for the data acquisition caching, caching in different modes is performed according to data classification:
the structured data comprises structured data in a relational database, content data identified through tables and fields is collected by means of an open-source Kafka message queue, the data is packaged into Json messages and pushed to a Kafka platform, and then the messages are analyzed and stored through a data processing program; the specific operation of the storage is as follows: storing the structured data into a MySQL database, and storing the association relation of the structured data into a Neo4j graph database;
the model data comprise 3D model file data, an AVEVA three-dimensional model, an SP3D three-dimensional model, a Revit software building three-dimensional model, a line GIS three-dimensional model and an AutoCAD Civil 3D total image three-dimensional model; importing model data through a Kafka system, and pulling out actual model data by task scheduling according to a Uniform Resource Locator (URL) corresponding to the model in the metadata for storage; storing the model data in a FastDFS distributed file system;
the unstructured data comprise engineering drawings, SP PID intelligent PID data drawings and data, office files and PDF files; for unstructured data, pushing metadata of the unstructured data to a platform through a Kafka message queue, and extracting actual file data for storage according to a file URL in the metadata by task scheduling; the unstructured file is stored into the FastDFS distributed file system.
3. The method for digitally handing over the shale gas field ground engineering based on the data blood margin according to claim 2, wherein for the unstructured data, in the process of analyzing and extracting the message, engineering object identification information associated with the unstructured data is extracted, the engineering object identification information is structured data information, and an association is established between the engineering object identification information and the corresponding unstructured data in a structured database, so that data connection between the structured data and the unstructured data is realized.
4. The method for digitally handing over the shale gas field ground engineering based on the data blood margin according to claim 1, wherein the data stored in the standard storage area is not directly pushed to an upper layer application to provide data service, the data is required to be subjected to service conversion processing according to service requirements in a service delivery mode, and the data requirements are stored into the upper layer application of the theme storage area for use after service conversion; the personalized application is required to be personalized and customized and then stored into a customized storage area for upper-layer application; the personalized application comprises interface personalization, protocol personalization and query personalization.
5. The data blood-based shale gas field ground engineering digital handover method according to claim 4, wherein the construction of the data service interface is realized by registering and opening an interface in a service catalog through a standard RESTful API based on an API gateway in a highly configurable and light development mode when the service delivery is carried out; realizing the data service under different scenes by combining and arranging the constructed data service interfaces; and takes the transmission object of the data service as one input of the data blood relationship.
6. The method for digitally handing over shale gas field ground engineering based on data blood margin according to claim 5, wherein after the data provided by the subject storage area or the customized storage area is used by the upper layer application, the output result data is restored to the result restoring area, and then the data stored in the result restoring area is standardized in a data rule base mode, so that the restored result data accords with the specification definition of international standard ISO 1596 and enterprise data storage standard, and then is stored in the standard storage area; the achievement restoring area is based on a MongoDB database;
the method comprises the steps of recording data conversion in each step in the processes of data acquisition, storage, service delivery application, personalized customization application and achievement restoring by adopting a message queue and data persistence technology, and storing the records in a MySQL database and a Neo4j graph database.
7. The data blood-margin-based shale gas field ground engineering digital handover method according to claim 1, wherein the unstructured data is subjected to labeling, sources and the stages of the unstructured data are refined, and data labels are set according to the formats and keywords of deliverables so as to facilitate rapid searching and querying of the sources of the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010693603.XA CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010693603.XA CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112328667A CN112328667A (en) | 2021-02-05 |
| CN112328667B true CN112328667B (en) | 2023-09-08 |
Family
ID=74303748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010693603.XA Active CN112328667B (en) | 2020-07-17 | 2020-07-17 | Shale gas field ground engineering digital handover method based on data blood margin |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112328667B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145919A (en) * | 2022-06-30 | 2022-10-04 | 中冶赛迪信息技术(重庆)有限公司 | Method, device, equipment and medium for generating data blood relationship between service systems |
| CN115687309B (en) * | 2022-12-30 | 2023-04-18 | 浙江大学 | Non-invasive cigarette warehouse-in and warehouse-out full-process data blood margin construction method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN109286653A (en) * | 2017-07-21 | 2019-01-29 | 埃森哲环球解决方案有限公司 | Intelligent cloud engineering platform |
| CN109446279A (en) * | 2018-10-15 | 2019-03-08 | 顺丰科技有限公司 | Based on neo4j big data genetic connection management method, system, equipment and storage medium |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
| CN111400305A (en) * | 2020-02-20 | 2020-07-10 | 深圳市魔数智擎人工智能有限公司 | Characteristic engineering blood relationship based backtracking and visualization method |
-
2020
- 2020-07-17 CN CN202010693603.XA patent/CN112328667B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN109286653A (en) * | 2017-07-21 | 2019-01-29 | 埃森哲环球解决方案有限公司 | Intelligent cloud engineering platform |
| CN109446279A (en) * | 2018-10-15 | 2019-03-08 | 顺丰科技有限公司 | Based on neo4j big data genetic connection management method, system, equipment and storage medium |
| CN110232098A (en) * | 2019-04-22 | 2019-09-13 | 汇通达网络股份有限公司 | A kind of data warehouse administered based on data and genetic connection designs |
| CN110555032A (en) * | 2019-09-09 | 2019-12-10 | 北京搜狐新媒体信息技术有限公司 | Data blood relationship analysis method and system based on metadata |
| CN111400305A (en) * | 2020-02-20 | 2020-07-10 | 深圳市魔数智擎人工智能有限公司 | Characteristic engineering blood relationship based backtracking and visualization method |
Non-Patent Citations (1)
| Title |
|---|
| 张新阳 等.基于电力行业的大数据时代下元数据管理方法.基于电力行业的大数据时代下元数据管理方法.2018,第14卷(第31期),第258-259页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112328667A (en) | 2021-02-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109213754B (en) | Data processing system and data processing method | |
| CN110781236A (en) | Method for constructing government affair big data management system | |
| CN111435344B (en) | Big data-based drilling acceleration influence factor analysis model | |
| CN105045820B (en) | Method for processing video image information of high-level data and database system | |
| US8484255B2 (en) | Automatic conversion of multidimentional schema entities | |
| CN109344223A (en) | A kind of Building Information Model management system and method based on cloud computing technology | |
| WO2018051096A1 (en) | System for importing data into a data repository | |
| CN111176867B (en) | Data sharing exchange and open application platform | |
| CN109542967A (en) | Smart city data-sharing systems and method based on XBRL standard | |
| CN114379608A (en) | Multi-source heterogeneous data integration processing method for urban rail transit engineering | |
| CN112328667B (en) | Shale gas field ground engineering digital handover method based on data blood margin | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN111078781B (en) | Multi-source flow type big data fusion convergence processing frame model implementation method | |
| CN114925045A (en) | PaaS platform for large data integration and management | |
| CN108170855B (en) | CityGML and IFC data unified query and data extraction method | |
| Mesiti et al. | Towards a user-friendly loading system for the analysis of big data in the internet of things | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN112699100A (en) | Management and analysis system based on metadata | |
| CN115934856A (en) | Method and system for constructing comprehensive energy data assets | |
| CN114254033A (en) | Data processing method and system based on BS architecture | |
| US20140143248A1 (en) | Integration to central analytics systems | |
| CN106657282B (en) | Method and device for integrating running state information of converter station equipment | |
| CN116523328A (en) | Intelligent decision-making method for cooperation of aviation equipment and manufacturing industry chain | |
| CN101968747A (en) | Cluster application management system and application management method thereof | |
| CN113342874A (en) | Wind power big data analysis system and process based on cloud computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |