CN112100181A - A sand table-based data resource management method - Google Patents

Abstract

A data resource management method based on a sand table belongs to the technical field of data resource management, and particularly relates to a data resource management method based on a sand table. Aiming at the problems of the existing data resources such as lack of standards, serious redundancy, poor data quality and the like, the requirements of people on the development and utilization of the data resources generally have the characteristics of dynamism, diversity, specialty and the like. By the sand table-based data resource management method, data resource management standards are unified, data resource hierarchies are defined, a data resource catalog is constructed, inquireable, understandable and easily-obtained data resources are provided for each specialty and each hierarchy, and work of cross-specialty and cross-hierarchy data acquisition, data sharing, data tracing and the like is supported.

Description

A sand table-based data resource management method

technical field

The invention belongs to the technical field of data resource management, and in particular relates to a sand table-based data resource management method.

Background technique

With the continuous deepening of the construction and application of my country's information system, a large number of basic data resources and rich data resource needs have been accumulated, showing the characteristics of scattered resources, diverse types, different standards, and low sharing, lacking enterprise-level unified management and Results are shared. Due to the problems of high application threshold, data difficult to understand, and difficult to obtain services, data resource services cannot meet the needs of enterprise-level data assets to be visible, traceable, traceable, and desirable. A data resource management method that can be queried, understood and easily obtained.

SUMMARY OF THE INVENTION

Aiming at the problems of lack of standards, serious redundancy, and poor data quality in existing data resources, people's needs for the development and utilization of data resources are generally dynamic, diverse, and professional. Through the sand table-based data resource management method, unify data resource management standards, clarify data resource levels, and build data resource catalogs to provide queried, understandable, and easy-to-obtain data resources for all majors and levels, supporting cross-professional and cross-disciplinary Hierarchical data acquisition, data sharing, and data traceability.

The present invention includes the following steps:

The binary tree method recursively traverses the database metadata to obtain the basic information of data resources, and establishes a data synchronization task. Considering the large amount of log information and the strong timeliness, the method of full + incremental is adopted, and other metadata (low update frequency) are used every Weekly use the full amount method. Add the dt field to distinguish and identify different systems. When the data synchronization task is established, write the system abbreviation into the dt field.

Recursive formula: a _n+k ×m ₁ a _n+k-1 +m ₂ a _n+k-2 +…m _k a _n

(1) Classification of data resource catalogs

Aiming at the problems of data resources such as high application threshold, difficult data to understand, and difficult to obtain services, the present invention builds a classification system of source layer, sharing layer and analysis layer of data resources, and provides queried, viewable and viewable data for all majors and all levels. Understand the data resource sandbox. Based on the inventory system function menu, according to the principle of completeness, independence and irreplaceability, the classification of data resource catalogues at all levels is formed.

The source layer: positioned close to the source business system, the table structure is basically the same as the source business system table structure, mainly stores the stock and incremental data of the source business system, the data scope is a subset of the source business system, and the data resources are classified according to specialty, system , first-level business functions, and second-level business functions are classified and constructed, which are closer to the business system.

Shared Layer: Stores SG-CIM model tables and standard tables not covered by the model. Provide support for upper-layer applications and services in accordance with unified naming specifications and calibers for source-end business systems. The data resource classification includes two parts: SG-CIM model table data and standard table data. The SG-CIM model resource classification is constructed according to the first-level subject domain and the second-level subject domain of the SG-CIM model; the standard table resource classification is based on the system , first-level classification, and second-level classification are constructed, and the system to which the data table belongs is labeled as a label.

Analysis layer: Stores the result table after processing the shared layer data according to business logic to form analytical services such as datasets, tags, and indicators. The resource classification is constructed according to the analysis theme, primary theme, and secondary theme. The source system, department, and application department of the data table are labeled as tags.

(2) Data resource table sorting

The description of the data sheet business information mainly includes, but is not limited to, as follows:

Business description: It cannot be simply the same as the Chinese name, and must include the business meaning of the table. If the table stores business entity information, a brief description of the business entity and related terms should be given; if document information is stored, a brief description of the document process links should be given; if report information is stored, a brief description of the purpose of the report should be given, which needs to be maintained by business personnel. For example, in the C_CONS table, the business description is: Organizations or individuals that establish a power supply and consumption relationship with a power supply company in accordance with the law are called electricity customers, or users for short, and different electricity addresses are regarded as different users. Recorded information is generated through services such as new installations, capacity additions, and changes in electricity consumption archives, including user ID, user name, electricity consumption address, electricity consumption category, power supply voltage, load nature, contract capacity and other electricity consumption attributes.

(3) Sorting out data resource fields

The description of the data table field business information mainly includes, but is not limited to, as follows:

Field Description: Describes the business meaning of the field. Describe in detail the business meaning, purpose, and business activity of the data item, which is maintained by business personnel. For example: the field NOTE_T is PE_CODE, and the business description is "the bill type printed by default for the monthly electricity bill of electricity customers, which complies with the State Grid enterprise-level marketing management code class set, mainly ordinary invoices, value-added tax invoices, receipts, etc.".

(4) Data resource label

In order to facilitate users to have a better understanding of data resources, based on the data resource management function, complete the data label labeling work of the data table, the specific labels include but are not limited to, as follows:

Major: Fill in the name of the major to which the data sheet belongs;

Business Department: Fill in the name of the business department to which the data sheet belongs, for example: Business Department: Finance Department;

Source business system: fill in the name of the data source business system in this data sheet, and the abbreviation of the business system. If multiple source systems are involved, all enumerations need to be separated by semicolons, such as "source system: [ECP]";

Source data table name: fill in the Chinese name and English name of the data table of the data table data source, involving multiple source tables, all enumerations need to be separated by semicolons, for example: source data table name: DATA_ID_MAP_INFO/data identification mapping table;

Negative list data: If the table or this field is included in the negative list data, the label of "negative list data" needs to be added;

Data table type label: According to the data table type, add labels such as detailed business data, coding data, indicator data, report data, etc., for example: data table type label: indicator data;

Support application scene name label: For the source layer, shared layer data table such as supporting application construction, add application scene label, the scene label supports multiple, the scene name must be consistent with the scene name in the analysis layer data resource classification, for example: enterprise resources Real-time grasp, industry sector unit distribution by industry;

Custom labels: Each unit can customize labels according to specific needs, business scenarios, etc.

(5) Data resource classification and table mapping relationship

Based on the sorted data table information results, complete the mapping relationship between information systems, subject areas and application topics and data tables.

Sorting out the correspondence between data categories and business tables: Based on the last-level data catalog, sort out the correspondence between data categories and data tables, and link business tables to the last-level category of data categories, and ensure the sorted data The correspondence between categories and data tables is complete and accurate. details as follows:

Source layer data resource classification and table mapping relationship

Shared layer data resource classification and table mapping relationship

Analysis layer data resource sorting standard specification

The present invention has beneficial effects.

The invention effectively manages data resources and avoids the problems of scattered data resources, diverse types, different standards, low sharing degree, high threshold for data resource service application, difficult data to understand, and difficult to obtain services.

The invention establishes data resources that can be inquired, understandable and easy to obtain through the data resource management method based on the sand table, and realizes the visibility, inquiry, traceability and accessibility of data resources.

(1) Establish data resource catalogues, classify them according to different business types of data resources, build them in layers, improve the ability of data resources to be visible, searchable, and understandable, and satisfy the quick query, positioning, and application of data resources.

(2) The unified data resource management standard is conducive to users' understanding and use of data resources, and avoids differences between business personnel and developers due to the incomprehension of terminology.

Description of drawings

The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The protection scope of the present invention is not limited to the following descriptions.

1, 2, and 3 are flow charts of data resource information collection according to the present invention.

Detailed ways

(1) Data resource information collection

Static analysis method, the advantage of this method is that it avoids the influence of human factors, and the accuracy is not affected by the detail of the document description, test cases and sampling data. This method is based on the principle of compilation. And statically analyze and list the paths involved in the program logic to achieve an objective reflection of the data flow.

Through the static analysis method, the data resource information is collected in the form of a binary tree. Traversal is one of the most basic operations on a tree. The so-called traversal of a binary tree is to traverse all the nodes of the binary tree according to certain rules and sequences, so that each node is visited once and only once. Since the binary tree is a nonlinear structure, the traversal of the tree essentially converts each node of the binary tree into a linear sequence to represent it.

Preorder traversal: Access the binary tree in the order of root node -> left subtree -> right subtree, as shown in Figure 1,

Preorder traversal: visit the root node; traverse the left subtree with preorder recursion; traverse the right subtree with preorder recursion.

Preorder traversal result: ABDFECGHI

Thought process:

First visit the root node A, A is divided into left and right subtrees, because it is a recursive call, so the left subtree also follows the order of "first root node - then left - then right", so visit B node, then visit D node, visit When there is a branch at the F node, it also follows the order of "root node-then left--right" to visit the E node. At this time, the large subtree on the left has been visited, and then follow the order of the last visit to the right subtree, Visit the big subtree on the right. The big subtree on the right also visits the root node C first, and then visits the left subtree G. Because the left subtree of G does not exist, the next two visit the right subtree H of G, and finally visit the right subtree of C. tree I

In-order traversal: visit in the order of left subtree -> root node -> right subtree, as shown in Figure 2,

Inorder traversal: traverse the left subtree in inorder; visit the root node; traverse the right subtree in inorder.

Inorder traversal result: DBEFAGHCI

3. Post-order traversal: visit in the order of left subtree -> right subtree -> root node, as shown in Figure 3,

Post-order traversal: use post-order recursion to traverse the left subtree; use post-order recursion to traverse the right subtree; visit the root node.

The result of the post-order traversal: DEFBHGICA

(2) Data resource information sorting and inventory

Based on the collected metadata of the data resource table, supplement and improve the business information and management information of the data, including Chinese name, business description, responsible department, negative data information, etc., to form a complete and effective data resource system.

(3) Basic information maintenance of data resources

Based on the results of data resource information sorting and inventory, carry out basic information maintenance of data resources, including table description, field description, table Chinese name, field Chinese name, primary key or not, negative list and other information maintenance work.

(4) Data resource directory construction

Based on the inventory system function menu, in accordance with the principle of completeness, independence and irreplaceability. Data resource catalog construction According to the classification dimension of the data resource catalog, a source layer data resource catalog based on system module classification, a shared layer data resource catalog based on business domain classification, and an analysis layer data resource catalog based on business application classification are established respectively.

(5) Construction, release and application of data resource services

Through further business processing of data resources, the visualized display, unified management and sharing of data resource services can be realized, and inconvenient problems such as cumbersome operation procedures for business personnel to access data resources, difficult to understand related terms, and data application and use are solved, and data resource services can be better utilized. Value, for business personnel to quickly query and subscribe to data resource services, and combine with enterprise-level negative list sorting and construction results to quickly respond to business personnel needs, provide data resource service sharing and application, and conduct business-based full-life process management and control of data resources.

It can be understood that the above specific description of the present invention is only used to illustrate the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those of ordinary skill in the art should understand that the present invention can still be modified or It is equivalent to replacement to achieve the same technical effect; as long as the needs of use are met, they are all within the protection scope of the present invention.

Claims (1)

1. a data resource management method based on sand table, is characterized in that comprising the following steps: The binary tree method recursively traverses the database metadata to obtain the basic information of data resources, and establishes a data synchronization task. Considering the large amount of log information and the strong timeliness, the method of full + incremental is adopted, and other metadata (low update frequency) are used every Weekly use the full amount method. Add the dt field to distinguish and identify different systems. When the data synchronization task is established, write the system abbreviation into the dt field. Recursive formula: a _n+k ×m ₁ a _n+k-1 +m ₂ a _n+k-2 +…m _k a _n (1) Classification of data resource catalogs Aiming at the problems of data resources such as high application threshold, difficult data to understand, and difficult to obtain services, the present invention builds a classification system of source layer, sharing layer and analysis layer of data resources, and provides queried, viewable and viewable data for all majors and all levels. Understand the data resource sandbox. Based on the inventory system function menu, according to the principle of completeness, independence and irreplaceability, the classification of data resource catalogues at all levels is formed. The source layer: positioned close to the source business system, the table structure is basically the same as the source business system table structure, mainly stores the stock and incremental data of the source business system, the data scope is a subset of the source business system, and the data resources are classified according to specialty, system , first-level business functions, and second-level business functions are classified and constructed, which are closer to the business system. Shared Layer: Stores SG-CIM model tables and standard tables not covered by the model. Provide support for upper-layer applications and services in accordance with unified naming specifications and calibers for source-end business systems. The data resource classification includes two parts: SG-CIM model table data and standard table data. The SG-CIM model resource classification is constructed according to the first-level subject domain and the second-level subject domain of the SG-CIM model; the standard table resource classification is based on the system , first-level classification, and second-level classification are constructed, and the system to which the data table belongs is labeled as a label. Analysis layer: Stores the result table after processing the shared layer data according to business logic to form analytical services such as datasets, tags, and indicators. The resource classification is constructed according to the analysis theme, primary theme, and secondary theme. The source system, department, and application department of the data table are labeled as tags. (2) Data resource table sorting The description of the data sheet business information mainly includes, but is not limited to, as follows: Business description: It cannot be simply the same as the Chinese name, and must include the business meaning of the table. If the table stores business entity information, a brief description of the business entity and related terms should be given; if document information is stored, a brief description of the document process links should be given; if report information is stored, a brief description of the purpose of the report should be given, which needs to be maintained by business personnel. For example, in the C_CONS table, the business description is: Organizations or individuals that establish a power supply and consumption relationship with a power supply company in accordance with the law are called electricity customers, or users for short, and different electricity addresses are regarded as different users. Recorded information is generated through services such as new installations, capacity additions, and changes in electricity consumption archives, including user ID, user name, electricity consumption address, electricity consumption category, power supply voltage, load nature, contract capacity and other electricity consumption attributes.

(3) Sorting out data resource fields The description of the data table field business information mainly includes, but is not limited to, as follows: Field Description: Describes the business meaning of the field. Describe in detail the business meaning, purpose, and business activity of the data item, which is maintained by business personnel. For example: the field NOTE_T is PE_CODE, and the business description is "the bill type printed by default for the monthly electricity bill of electricity customers, which complies with the State Grid enterprise-level marketing management code class set, mainly ordinary invoices, value-added tax invoices, receipts, etc.". (4) Data resource label In order to facilitate users to have a better understanding of data resources, based on the data resource management function, complete the data label labeling work of the data table, the specific labels include but are not limited to, as follows: Major: Fill in the name of the major to which the data sheet belongs; Business Department: Fill in the name of the business department to which the data sheet belongs, for example: Business Department: Finance Department; Source business system: fill in the name of the data source business system in this data sheet, and the abbreviation of the business system. If multiple source systems are involved, all enumerations need to be separated by semicolons, such as "source system: [ECP]"; Source data table name: fill in the Chinese name and English name of the data table of the data table data source, involving multiple source tables, all enumerations need to be separated by semicolons, for example: source data table name: DATA_ID_MAP_INFO/data identification mapping table; Negative list data: If the table or this field is included in the negative list data, the label of "negative list data" needs to be added; Data table type label: According to the data table type, add labels such as detailed business data, coding data, indicator data, report data, etc., for example: data table type label: indicator data; Support application scene name label: For the source layer, shared layer data table such as supporting application construction, add application scene label, the scene label supports multiple, the scene name must be consistent with the scene name in the analysis layer data resource classification, for example: enterprise resources Real-time grasp, industry sector unit distribution by industry; Custom labels: Each unit can customize labels according to specific needs, business scenarios, etc. (5) Data resource classification and table mapping relationship Based on the sorted data table information results, complete the mapping relationship between information systems, subject areas and application topics and data tables. Sorting out the correspondence between data categories and business tables: Based on the last-level data catalog, sort out the correspondence between data categories and data tables, and link business tables to the last-level category of data categories, and ensure the sorted data The correspondence between categories and data tables is complete and accurate. details as follows: Source layer data resource classification and table mapping relationship

Shared layer data resource classification and table mapping relationship

Analysis layer data resource sorting standard specification

Images