CN110019176B - Data management control system for improving success rate of data management service - Google Patents
Data management control system for improving success rate of data management service Download PDFInfo
- Publication number
- CN110019176B CN110019176B CN201910288687.6A CN201910288687A CN110019176B CN 110019176 B CN110019176 B CN 110019176B CN 201910288687 A CN201910288687 A CN 201910288687A CN 110019176 B CN110019176 B CN 110019176B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- management
- subsystem
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Automation & Control Theory (AREA)
- Storage Device Security (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a data management control system for improving success rate of data management service, which comprises a resource hierarchical management subsystem, a data management organization establishment subsystem, a user identity verification subsystem, a right management subsystem and a resource allocation subsystem, wherein the resource hierarchical management subsystem is used for constructing data management organization, verifying user identities and rights management and allocating resources; the metadata management subsystem is used for collecting metadata of technology, service, operation and management and performing association analysis on the metadata; the model construction subsystem is used for identifying business concepts and constructing an enterprise data model; the data standardization subsystem is used for formulating unified data service language; and the data management operation subsystem is used for establishing a data management long-acting operation system. The data management control system for improving the success rate of the data management service realizes easy operation and can greatly improve the service efficiency and the implementation success rate of the data management; meanwhile, a closed-loop steady system can be formed, and a data management long-acting operation system is established.
Description
Technical Field
The invention relates to the field of computer software, in particular to the field of big data, and specifically relates to a data management control system for improving the success rate of data management service.
Background
IT construction starts from 60 years, the technology and application of software and hardware technology are continuously deepened in the aspect of data when the technology is changed over the day, and the data is not accurate and standard, so that the data can cause the loss of resource waste, decision errors and the like from the earliest data application, data storage to the current data analysis, data management, data statistics, data integration, data mining and the like, so that many large data analysis applications are declared to be failed due to low data quality. In general
The data is now referred to as "new petroleum", also known as "new currency", and is considered to be valuable, most organizations have realized that the big data age of unordered development is about to end. In the process of data construction, data management is receiving more and more attention, and dirty data is processed by adopting a data management method to obtain standard clean data, and data management is attempted for this reason. Although the basic concept of data management is primarily understood, the construction method of data management does not have clear knowledge, and a proper data management operation method is not available, and the method is basically in contradiction, goes forward and is explored in chaos.
Data governance refers to a process from using sporadic data to using unified data, from governance with little or no organization and flow, to enterprise-wide comprehensive data governance, from attempting to handle data chaotic conditions to data well-logging. Data governance is a data management concept that involves the ability of an organization to ensure that there is a high data quality throughout the life of the data. In the scope, the data management covers the data analysis from the front-end transaction processing system and the back-end business database to the terminal, and a closed-loop negative feedback system is formed from the source to the terminal and back to the source. For the purpose, data management is to supervise the acquisition, processing and use of data, and the supervision functions need to have the executive forces of discovery, supervision, control, communication and integration. Data governance is the strategy of handling data, i.e., how data is stored, accessed, verified, protected, and used.
Big data management can be divided into different stages from construction content and implementation targets, and each stage is used for completing different tasks, so that the construction content is deepened gradually along with the progressive stage, and different enterprise access points and requirements are different. Various challenges and challenges are encountered in the process from data asset management to improving data quality, which is implemented on-site. The data governance definitions given in the industry are not tens of times different from the cut-in view angle and the emphasis point of the data governance implementation, and a unified data governance method is not formed so far. And the prior art does not provide a widely applicable data management method which is easy to succeed, and a solution for effective operability is not provided for the method.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide the data management control system which has high accuracy, high efficiency and high safety and can improve the success rate of data management service.
In order to achieve the above object, the data management control system for improving the success rate of data management service according to the present invention is as follows:
the data management control system for improving the success rate of the data management service is mainly characterized by comprising the following components:
the resource hierarchical management subsystem is used for constructing data management organization, verifying user identity, managing authority and distributing resources;
the metadata management subsystem is connected with the resource hierarchical management subsystem and is used for collecting metadata of technology, service, operation and management, establishing a metadata model and associating and analyzing the metadata;
the model construction subsystem is connected with the metadata management subsystem and is used for identifying business concepts, constructing theme and entity concept models and constructing enterprise data models;
the data standardization subsystem is connected with the model construction subsystem and is used for formulating unified data service language and recording standard contents such as format, conversion, coding and the like.
The data management operation subsystem is connected with the data standardization subsystem and is used for establishing a data management long-acting operation system.
Preferably, the resource hierarchical management subsystem comprises:
the personnel organization management module is used for managing user organization;
the identity verification module is connected with the personnel organization management module and is used for verifying the identity information of the user;
the authority management module is connected with the identity verification module and is used for setting resource access authority and system operation authority for a user;
and the resource allocation module is connected with the authority management module and is used for allocating resources according to the user authority.
Preferably, the metadata management subsystem includes:
the technical metadata module is connected with the resource hierarchical management subsystem and is used for managing data describing physical and chemical data;
the business metadata module is connected with the technical metadata module and is used for managing the data defined by the description data;
the operation metadata module is connected with the business metadata module and is used for managing data describing a data processing process;
the management metadata module is connected with the operation metadata module and is used for managing data describing data attributes;
the meta model module is connected with the management meta data module and is used for managing the model conforming to the current situation of the stored enterprise data;
and the metadata analysis module is connected with the metadata model module and is used for carrying out multidimensional analysis on metadata according to the multi-granularity and hierarchical metadata model.
Preferably, the model building subsystem comprises:
the business concept identification module is connected with the metadata management subsystem and is used for carrying out continuity analysis on key business and identifying each key business unit;
the theme conceptual model construction module is connected with the business conceptual identification module and used for constructing a general business conceptual model, and the theme conceptual model is a general business concept;
the entity concept model construction module is connected with the topic concept model construction module and is used for constructing an entity concept model which is a unique identifier of a business term;
and the data model construction module is connected with the entity concept model construction module and is used for constructing a data model generated and consumed in an organization.
Preferably, the data normalization subsystem comprises:
the data attribute definition module is connected with the model construction subsystem and is used for carding data attributes;
the data entity definition module is connected with the data attribute definition module and is used for carrying out service definition of various data objects;
and the data theme definition module is connected with the data entity definition module and is used for defining the data theme.
Preferably, the data management operation subsystem includes:
the data standard full-flow management module is connected with the data standardization subsystem and is used for providing data dictionary service and metadata query service for all business personnel and application systems;
the data quality module is connected with the data standard full-flow management module and is used for managing data quality category, quality standard and quality check and displaying a data quality report;
the data security module is connected with the data quality module and is used for realizing data security according to the data type and the security level;
the message pushing module is connected with the data security module and is used for pushing the message to a user and providing user processing;
the data treatment evaluation module is connected with the message pushing module and is used for evaluating the treatment process and generating an evaluation report.
Preferably, the personnel organization includes data governance committee members, data asset administrators, information system administrators, business data administrators, and general users.
Preferably, the metadata includes technical attributes, management attributes and business attributes.
The data management control system for improving the success rate of the data management service is provided to solve the problem that the existing data management does not have an effective data management method with unified standard. Thereby realizing easy operation and greatly improving the service efficiency and the implementation success rate of data management; meanwhile, a closed-loop steady system can be formed, and a data management long-acting operation system is established.
Drawings
FIG. 1 is a schematic diagram of a data governance control system for improving success rate of data governance services according to the present invention.
FIG. 2 is a schematic diagram of a resource hierarchy management subsystem of a data governance control system for improving success rate of data governance services according to the present invention.
FIG. 3 is a schematic diagram of a metadata management subsystem of a data governance control system for improving success rate of data governance services according to the present invention.
FIG. 4 is a schematic diagram of a model building subsystem of the data governance control system of the present invention for improving the success rate of data governance services.
FIG. 5 is a schematic diagram of a data normalization subsystem of a data governance control system for improving the success rate of data governance services according to the present invention.
Fig. 6 is a schematic diagram of a data governance operation subsystem of the data governance control system for improving success rate of data governance services according to the present invention.
Detailed Description
In order to more clearly describe the technical contents of the present invention, a further description will be made below in connection with specific embodiments.
The data governance control system for improving the success rate of the data governance service comprises:
the resource hierarchical management subsystem is used for constructing data management organization, verifying user identity, managing authority and distributing resources;
the metadata management subsystem is connected with the resource hierarchical management subsystem and is used for collecting metadata of technology, service, operation and management, establishing a metadata model and associating and analyzing the metadata;
the model construction subsystem is connected with the metadata management subsystem and is used for identifying business concepts, constructing theme and entity concept models and constructing enterprise data models;
the data standardization subsystem is connected with the model construction subsystem and is used for formulating unified data service language and recording standard contents such as format, conversion, coding and the like.
The data management operation subsystem is connected with the data standardization subsystem and is used for establishing a data management long-acting operation system.
As a preferred embodiment of the present invention, the resource hierarchy management subsystem includes:
the personnel organization management module is used for managing user organization;
the identity verification module is connected with the personnel organization management module and is used for verifying the identity information of the user;
the authority management module is connected with the identity verification module and is used for setting resource access authority and system operation authority for a user;
and the resource allocation module is connected with the authority management module and is used for allocating resources according to the user authority.
As a preferred embodiment of the present invention, the metadata management subsystem includes:
the technical metadata module is connected with the resource hierarchical management subsystem and is used for managing data describing physical and chemical data;
the business metadata module is connected with the technical metadata module and is used for managing the data defined by the description data;
the operation metadata module is connected with the business metadata module and is used for managing data describing a data processing process;
the management metadata module is connected with the operation metadata module and is used for managing data describing data attributes;
the meta model module is connected with the management meta data module and is used for managing the model conforming to the current situation of the stored enterprise data;
and the metadata analysis module is connected with the metadata model module and is used for carrying out multidimensional analysis on metadata according to the multi-granularity and hierarchical metadata model.
As a preferred embodiment of the present invention, the model building subsystem includes:
the business concept identification module is connected with the metadata management subsystem and is used for carrying out continuity analysis on key business and identifying each key business unit;
the theme conceptual model construction module is connected with the business conceptual identification module and used for constructing a general business conceptual model, and the theme conceptual model is a general business concept;
the entity concept model construction module is connected with the topic concept model construction module and is used for constructing an entity concept model which is a unique identifier of a business term;
and the data model construction module is connected with the entity concept model construction module and is used for constructing a data model generated and consumed in an organization.
As a preferred embodiment of the present invention, the data normalization subsystem includes:
the data attribute definition module is connected with the model construction subsystem and is used for carding data attributes;
the data entity definition module is connected with the data attribute definition module and is used for carrying out service definition of various data objects;
and the data theme definition module is connected with the data entity definition module and is used for defining the data theme.
As a preferred embodiment of the present invention, the data management operation subsystem includes:
the data standard full-flow management module is connected with the data standardization subsystem and is used for providing data dictionary service and metadata query service for all business personnel and application systems;
the data quality module is connected with the data standard full-flow management module and is used for managing data quality category, quality standard and quality check and displaying a data quality report;
the data security module is connected with the data quality module and is used for realizing data security according to the data type and the security level;
the message pushing module is connected with the data security module and is used for pushing the message to a user and providing user processing;
the data treatment evaluation module is connected with the message pushing module and is used for evaluating the treatment process and generating an evaluation report.
As a preferred embodiment of the present invention, the personnel organization includes data administration committee members, data asset administrators, information system administrators, business data administrators, and general users.
As a preferred embodiment of the present invention, the metadata includes technical attributes, management attributes, and business attributes.
In a specific embodiment of the present invention, there is provided a data management method for improving a success rate of a data management service, the data management method comprising: the system comprises a resource hierarchical management subsystem, a metadata management subsystem, a model construction subsystem, a data standardization subsystem and a data management operation subsystem. The resource hierarchical management subsystem is used for constructing data management organizations and distributing authorities; the metadata management subsystem is used for collecting, managing and relating analysis of technical and business metadata; the model construction subsystem is used for constructing a data model; the data standardization subsystem is a process of standardization of definition, organization, supervision and protection of data by an organization and is used for formulating unified data service language, recording format, conversion, coding and other standard contents; the data management operation subsystem is used for establishing a data management long-acting operation system. The method combines the functions of an information technology part, a business part and a data management part, and performs closed-loop supervision to gradually form a steady data management system, has the characteristics of easy operation and easy success, and can greatly improve the success rate and the implementation efficiency of data management service.
The invention aims to provide a data management method for improving the success rate of data management service in a big data environment, so as to solve the problem that the existing data management does not have an effective data management method with unified standard. Thereby realizing easy operation and greatly improving the data management service efficiency and the implementation success rate; meanwhile, a closed-loop steady system can be formed, and a data management long-acting operation system is established.
In order to achieve the above object, a data management method for improving success rate of data management service according to the present invention is as follows: the data management method for improving the success rate of the data management service is mainly characterized in that the data management method is shown in figure 1 and comprises the following steps:
the resource hierarchical management subsystem is used for constructing a data management organization, verifying user identity, managing authority and distributing resources;
the metadata management subsystem is connected with the resource hierarchical management subsystem and is used for collecting technology, business, operation and management metadata, building a metadata model and analyzing metadata association.
The model construction subsystem is connected with the metadata management subsystem and is used for identifying business concepts, constructing a theme and entity concept model and constructing an enterprise data model.
The data standardization subsystem is connected with the model construction subsystem, is a process for carrying out standardization on definition, organization, supervision and protection of data by organization, and is used for formulating unified data service language, recording format, conversion, coding and other standard contents.
The data management operation subsystem is connected with the data standardization subsystem and is used for establishing a data management long-acting operation system.
The resource hierarchical management subsystem is shown in fig. 2, and comprises a personnel organization management module, an identity verification module, a right management module and a resource allocation module. The resources refer to data assets, the subsystem carries out hierarchical management on the data assets, different personnel organizations have different access rights, different personnel are set with different resource access rights and system operation rights through verification of identity information of users, and the system carries out resource allocation according to the user rights.
The staff organization includes: the data management committee member, the data asset manager, the information system manager, the business data manager, the common user and the like set and distribute different data access and operation authorities according to different using users, wherein the common user only has the authority for accessing the data resource catalog, and a higher-level user can obtain more data access and operation authorities.
The metadata management subsystem is shown in fig. 3, and comprises a technical metadata module, a business metadata module, an operation metadata module, a management metadata module, a metadata model module and a metadata analysis module. Metadata is data describing data, where there are two key points, one is data and the other is description data. Metadata includes attributes of the technology, management, business, etc. of the data. Technical metadata refers to data describing physical and chemical properties of the data, and the data type is a physical model. The service metadata is data describing the definition of the data. The operation metadata is data describing a data processing procedure. The management metadata is data describing data attributes. The meta-model module is a model conforming to the current situation of storing enterprise data. And the metadata analysis module analyzes the metadata in different dimensions according to the metadata models with different granularities and layers.
Technical metadata includes, but is not limited to, relational database physical models, noSQL, class database storage models, MPP type database physical models. Business metadata includes, but is not limited to, enterprise data standards, enterprise data quality standards, enterprise data indicators, enterprise data dictionaries, enterprise data codes, enterprise data security. The operation metadata includes, but is not limited to, ETL information, data processing policy data information, data processing scheduling information, data processing exception information. The management metadata includes, but is not limited to, data attribution information including, but not limited to, business attribution, system attribution, operation and maintenance attribution, data authority attribution and the like, and also includes technical attribute information and business attribute information. The meta-model module is a meta-data model which is obtained by combing technical meta-data, business meta-data, operation meta-data and management meta-data and accords with the current situation of the data of the storage enterprise. The metadata analysis module analyzes the metadata in different dimensions according to the metadata models with different granularities and layers, such as: data map, data lineage, impact analysis, etc. of the business. The embodiments are managed and implemented from three dimensions, the technical dimensions surrounding a source system, a data platform, a data mart, a data application, a data model, a database, a table, a field, a data relationship between fields and fields. The data of the business dimension, the technical dimension and the management dimension of the business dimension management index, chinese description of fields, processing strategy of the table, life cycle information of the table and security level of the table or the fields. And the application dimension realizes data platform change influence analysis, blood system analysis and a high-order data map.
The model construction subsystem is shown in fig. 4, and comprises a business concept identification module, a theme concept model construction module, an entity concept model construction module and a data model construction module. The service concept identification module is used for carrying out continuity analysis on the key service and identifying each key service unit. The topic concept model is a high-level classification of data representing a set of concepts of topics that are important to an organization. The entity concept model is an important transaction of each business domain from the enterprise perspective, and is a unique identification representing business terms. The data model building module is a data overview generated and consumed in the whole organization of the enterprise and is a single integrated definition of data.
The model construction subsystem can be effectively used in scenes such as business conception, demand writing, model design, data application and the like, and can promote and solve the problems of difficult business and technical communication cooperation, difficult use of data assets, difficult landing of data treatment results and the like. The service concept identification module is used for carrying out continuity analysis on the key service and identifying each key service unit. The topic concept model may represent general business concepts such as customers, products, employees, finance, etc., as well as industry-specific concepts. The entity conceptual model is a unique identification that represents business terms. The data model building module is a data overview generated and consumed in the whole organization of the enterprise, is a single integration definition of data, identifies functions and organization boundaries of sharable or redundant data, and has no data redundancy and data ambiguity. The implementation method can reduce business cognition cost and learning cost, reduce operation complexity, improve operation friendliness, firstly, completely restore the existing business process, adopt business departments related to the business process in series in a form of business main lines, one enterprise possibly comprises a plurality of business main lines, subdivide and comb the business process, business links and business units one by one according to the main lines, finally identify documents used or generated by each key business unit, business data can circulate in the documents, the state of the documents can be changed in the circulation process, each state change is caused by the operation of the key business unit, data model construction can be carried out by finding the business data, the construction of the data model needs to be constructed based on metadata relation of information items or fields from the angle of data application, and the value emphasis is embodied in the richness of the relation. The data model construction belongs to the content of data taxonomies and is not described in detail herein.
The data standardization subsystem is shown in fig. 5, and comprises a data attribute definition module, a data entity definition module and a data theme definition module. The data attributes include management attributes, business attributes, and technical attributes. The data entity refers to a collection of data attributes, and the data entity definition module refers to a service definition of each type of data object.
The specific implementation mode of the data standardization subsystem is that the data attribute is managed, the data attribute definition module, the data entity definition module and the data theme definition module define the data attribute, the related technical attribute is field type and field length, the related business attribute comprises contents such as data owner, business unit and business definition, and the related management attribute comprises contents such as version, security classification and grading, return management department and the like.
The data management operation subsystem is shown in fig. 6, and comprises a data standard full-flow management module, a data quality module, a data security module, a message pushing module and a data management evaluation module. The subsystem solves the problems of metadata management, data standard, data quality, data security, data life cycle and data service penetration. The data standard full-flow management module applies the bottom layer components of the integrated framework and comprises the functions of data dictionary management, metadata management and the like. The data quality module is built with the goal of exposing and improving the system data quality. The data security module is used for realizing free flow of data under proper security protection and meeting the requirement of data security use. The message pushing module is a feedback means for long-acting operation of data management, and the data management evaluation module is a module for forming closed-loop supervision and promoting formation of a stable data management system.
The data standard full-flow management module provides data dictionary service and metadata query service for various business personnel and application systems. The data quality module integrates data quality category management, quality standard management, quality check management, problem data display and data quality report components, and aims at exposing and improving system data quality. The method has the advantages that the data quality problems are used for carrying out the activities of identification, measurement, monitoring, early warning and the like, and the data quality is further improved by improving and enhancing the management level of organizations. The data security module combines the division of different categories and confidentiality of data defined by the data standardization subsystem, adopts different use and management principles according to the categories and confidentiality of the data, realizes the free flow of the data under proper security protection, and meets the requirement of data security use. The message pushing module is a feedback means for long-acting operation of data management, and the data standard full-flow management module generates standard revision, standard modification and standard deletion messages; the data quality module generates information such as inconsistent data, missing data and the like, the safety module generates information such as data compliance use, data acquisition and the like, and the information module pushes information forming tasks to a user and processes the information forming tasks by the user. When the user adopts the above modules to carry out treatment, the data treatment evaluation module has the condition that the program execution in certain treatment tasks fails or the treatment process is not completed, and in order to improve the success rate of data treatment, the evaluation module evaluates the data treatment result according to the data treatment requirement, generates an evaluation report, and sends the evaluation report to the user through the message pushing module so that the user can carry out proper adjustment measures, thereby forming closed-loop supervision and promoting the formation of a steady data treatment system.
The embodiment of the invention provides a data management method for improving the success rate of data management service, which comprises the following steps: the system comprises a resource hierarchical management subsystem, a metadata management subsystem, a model construction subsystem, a data standardization subsystem and a data management operation subsystem. The resource hierarchical management subsystem is used for constructing data management organizations and distributing authorities; the metadata management subsystem is used for collecting, managing and relating analysis of technical and business metadata; the model construction subsystem is used for constructing a data model; the data standardization subsystem is a process of standardization of definition, organization, supervision and protection of data by an organization and is used for formulating unified data service language, recording format, conversion, coding and other standard contents; the data management operation subsystem is used for establishing a data management long-acting operation system.
The technical scheme performs data management by organizing, metadata association analysis, data model construction, data standardization definition and establishing a data management operation system, and mainly performs closed-loop supervision through organization, model, standard and operation angles to gradually form a stable data management system. The resource hierarchical management subsystem in the technical scheme is mainly used for performing hierarchical management on data assets, setting different resource access rights for different personnel organizations, and performing resource allocation according to users by the system, and aims at data management committee members, data asset administrators, information system administrators, business data administrators and common users. The metadata in the technical scheme comprises technical metadata, business metadata, operation metadata, management metadata and a metadata model, each metadata comprises three attributes of technology, management and business, and the technical metadata comprises but is not limited to a relational database physical model, a NoSQL, a class database storage model and an MPP type database physical model. Business metadata includes, but is not limited to, enterprise data standards, enterprise data quality standards, enterprise data indicators, enterprise data dictionaries, enterprise data codes, enterprise data security. The operation metadata includes, but is not limited to, ETL information, data processing policy data information, data processing scheduling information, data processing exception information. The management metadata includes, but is not limited to, data attribution information including, but not limited to, business attribution, system attribution, operation and maintenance attribution, data authority attribution and the like, and also includes technical attribute information and business attribute information. The meta-model module is a meta-data model which is obtained by combing technical meta-data, business meta-data, operation meta-data and management meta-data and accords with the current situation of the data of the storage enterprise. The technical scheme is implemented in three layers of topics, entities and attributes and is applied from service dimension, technical dimension and management dimension, and aims at personnel such as data management committee members, data asset administrators, information system administrators, business data administrators, common users and the like.
In the technical scheme, metadata acquisition is acquisition of technology, service, operation and management metadata, and the technology, management and service attributes of the metadata are defined by means of a model construction subsystem, service identification and attribute combing of a standardized subsystem and the like. According to the technical scheme, the metadata analysis module analyzes metadata in three dimensions according to metadata models with different granularities and layers, wherein the technical dimensions surround a source system, a data platform, a data mart and a data application, and the data models, the databases, the tables, the fields and the data relations among the fields are analyzed. The business dimension surrounds the data, chinese description of the field, processing strategy of the table, life cycle information of the table, security level of the table or the field. The application dimension surrounds change influence analysis, blood system analysis and a high-order data map, wherein the change influence analysis is analysis of whether the change of metadata influences other metadata, and the blood system analysis is analysis of all metadata with a certain metadata as a termination node and related to the certain metadata.
Data governance services emphasize delivery of trusted, secure data, and it is seen that data governance focuses on the process methods that implement the services rather than the results. In the data management implementation process, the data management implementation can be effectively realized by organically correlating the data management implementation process with the data management implementation process. The method has the advantages that through effective data management, the flow system of data management organization is unified and standardized, a traceable data flow chart is provided, data redundancy and data ambiguity are reduced, consistency and correctness of data quality are improved, definition of general attributes and service concept identification definition are improved, finally, data tracking meeting service compliance is provided, the aim of supporting and improving service is achieved, and a long-acting operation system of data management is formed. Through the cooperation of the subsystems, the data management method of the embodiment of the invention solves the problem that the existing data management does not have an effective data management method with unified standard. Thereby realizing easy operation and greatly improving the data management service efficiency and the implementation success rate; meanwhile, a closed-loop steady system can be formed, and a data management long-acting operation system is established.
The data management control system for improving the success rate of the data management service is provided to solve the problem that the existing data management does not have an effective data management method with unified standard. Thereby realizing easy operation and greatly improving the service efficiency and the implementation success rate of data management; meanwhile, a closed-loop steady system can be formed, and a data management long-acting operation system is established.
In this specification, the invention has been described with reference to specific embodiments thereof. It will be apparent, however, that various modifications and changes may be made without departing from the spirit and scope of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
Claims (7)
1. A data governance control system for improving success rate of data governance services, said system comprising:
the resource hierarchical management subsystem is used for constructing data management organization, verifying user identity, managing authority and distributing resources;
the metadata management subsystem is connected with the resource hierarchical management subsystem and is used for collecting metadata of technology, service, operation and management, establishing a metadata model and associating and analyzing the metadata;
the model construction subsystem is connected with the metadata management subsystem and is used for identifying business concepts, constructing theme and entity concept models and constructing enterprise data models;
the data standardization subsystem is connected with the model construction subsystem and is used for formulating unified data service language, recording format and conversion and coding standard content;
the data management operation subsystem is connected with the data standardization subsystem and is used for establishing a data management long-acting operation system;
the data management operation subsystem comprises:
the data standard full-flow management module is connected with the data standardization subsystem and is used for providing data dictionary service and metadata query service for all business personnel and application systems;
the data quality module is connected with the data standard full-flow management module and is used for managing data quality category, quality standard and quality check and displaying a data quality report;
the data security module is connected with the data quality module and is used for realizing data security according to the data type and the security level;
the message pushing module is connected with the data security module and is used for pushing the message to a user and providing user processing;
the data treatment evaluation module is connected with the message pushing module and is used for evaluating the treatment process and generating an evaluation report;
the data standard whole-flow management module performs the activities of identification, measurement, monitoring and early warning according to the data quality problem, and improves the data quality by improving and enhancing the management level of an organization;
the data security module combines the division of different categories and the security classes of the data defined by the data standardization subsystem, adopts different use and management principles according to the categories and the security classes of the data, and realizes the free flow of the data under proper security protection;
the data management evaluation module evaluates the data management result according to the data management requirement, generates an evaluation report, and sends the evaluation report to a user through the message pushing module so that the user can take proper adjustment measures to form closed-loop supervision and promote the formation of a stable data management system.
2. The data governance control system for improving success rate of data governance services of claim 1, wherein said resource hierarchy management subsystem comprises:
the personnel organization management module is used for managing user organization;
the identity verification module is connected with the personnel organization management module and is used for verifying the identity information of the user;
the authority management module is connected with the identity verification module and is used for setting resource access authority and system operation authority for a user;
and the resource allocation module is connected with the authority management module and is used for allocating resources according to the user authority.
3. The data governance control system for improving success rate of data governance services of claim 1, wherein said metadata management subsystem comprises:
the technical metadata module is connected with the resource hierarchical management subsystem and is used for managing data describing physical and chemical data;
the business metadata module is connected with the technical metadata module and is used for managing the data defined by the description data;
the operation metadata module is connected with the business metadata module and is used for managing data describing a data processing process;
the management metadata module is connected with the operation metadata module and is used for managing data describing data attributes;
the meta model module is connected with the management meta data module and is used for managing the model conforming to the current situation of the stored enterprise data;
and the metadata analysis module is connected with the metadata model module and is used for carrying out multidimensional analysis on metadata according to the multi-granularity and hierarchical metadata model.
4. The data governance control system for improving success rate of data governance services of claim 1, wherein said model building subsystem comprises:
the business concept identification module is connected with the metadata management subsystem and is used for carrying out continuity analysis on key business and identifying each key business unit;
the theme conceptual model construction module is connected with the business conceptual identification module and used for constructing a general business conceptual model, and the theme conceptual model is a general business concept;
the entity concept model construction module is connected with the topic concept model construction module and is used for constructing an entity concept model which is a unique identifier of a business term;
and the data model construction module is connected with the entity concept model construction module and is used for constructing a data model generated and consumed in an organization.
5. The data governance control system for improving success rate of data governance services of claim 1, wherein said data normalization subsystem comprises:
the data attribute definition module is connected with the model construction subsystem and is used for carding data attributes;
the data entity definition module is connected with the data attribute definition module and is used for carrying out service definition of various data objects;
and the data theme definition module is connected with the data entity definition module and is used for defining the data theme.
6. The data governance control system for improving success rate of data governance services of claim 2, wherein said personnel organization comprises a data governance committee member, a data asset manager, an information system manager, a business data manager and a general user.
7. The data governance control system for improving success rate of data governance services according to claim 3, wherein said metadata comprises a technical attribute, a management attribute and a business attribute.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910288687.6A CN110019176B (en) | 2019-04-11 | 2019-04-11 | Data management control system for improving success rate of data management service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910288687.6A CN110019176B (en) | 2019-04-11 | 2019-04-11 | Data management control system for improving success rate of data management service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019176A CN110019176A (en) | 2019-07-16 |
| CN110019176B true CN110019176B (en) | 2023-08-18 |
Family
ID=67190977
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910288687.6A Active CN110019176B (en) | 2019-04-11 | 2019-04-11 | Data management control system for improving success rate of data management service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019176B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111190881A (en) * | 2019-11-13 | 2020-05-22 | 深圳市华傲数据技术有限公司 | Data management method and system |
| CN110765337B (en) * | 2019-11-15 | 2021-04-06 | 中科院计算技术研究所大数据研究院 | Service providing method based on internet big data |
| CN111143616B (en) * | 2019-12-24 | 2023-09-15 | 北京中盾安全技术开发公司 | Video image data management method |
| CN113032376A (en) * | 2019-12-25 | 2021-06-25 | 陕西云基华海信息技术有限公司 | Mass data quality management and treatment system |
| CN111861790A (en) * | 2020-07-24 | 2020-10-30 | 国网安徽省电力有限公司 | Electric power full-service data management system |
| CN112434071B (en) * | 2020-12-15 | 2021-07-20 | 北京三维天地科技股份有限公司 | Metadata blood relationship and influence analysis platform based on data map |
| CN112527774A (en) * | 2020-12-18 | 2021-03-19 | 通号智慧城市研究设计院有限公司 | Data center building method and system and storage medium |
| CN112800046A (en) * | 2021-02-26 | 2021-05-14 | 上海帕科信息科技有限公司 | Artificial intelligence platform applied to field data management |
| CN113222740A (en) * | 2021-05-27 | 2021-08-06 | 中国工商银行股份有限公司 | Asset management method, apparatus, computing device and medium executed by computing device |
| US20230105207A1 (en) * | 2021-10-06 | 2023-04-06 | Bank Of America Corporation | System and methods for intelligent entity-wide data protection |
| CN114298550A (en) * | 2021-12-28 | 2022-04-08 | 安徽海螺信息技术工程有限责任公司 | Method for treating cement production operation data |
| CN115080758A (en) * | 2022-05-10 | 2022-09-20 | 兴业银行股份有限公司 | Data life cycle management method and system based on data blood relationship map |
| CN114881802B (en) * | 2022-07-11 | 2022-10-04 | 湖南三湘银行股份有限公司 | Metadata-based data asset management method and system |
| CN116431646A (en) * | 2023-04-20 | 2023-07-14 | 北京瑞风协同科技股份有限公司 | Modeling and hierarchical control device and method for data model |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002054171A2 (en) * | 2000-12-06 | 2002-07-11 | Biosentients, Inc. | System, method, software architecture and business model for an intelligent object based information technology platform |
| CN104143122A (en) * | 2013-05-07 | 2014-11-12 | 天津冠创科技有限公司 | Intelligent service approval scheme |
| CA2867589A1 (en) * | 2013-10-15 | 2015-04-15 | Coho Data Inc. | Systems, methods and devices for implementing data management in a distributed data storage system |
| CN105760980A (en) * | 2015-11-27 | 2016-07-13 | 国网山东省电力公司潍坊供电公司 | Intelligent operation system based on intelligent power grid framework |
| CN106664224A (en) * | 2014-08-20 | 2017-05-10 | 华为技术有限公司 | System and method for metadata enhanced inventory management of a communications system |
| CN107247788A (en) * | 2017-06-15 | 2017-10-13 | 山东浪潮云服务信息科技有限公司 | A kind of method of the comprehensive regulation service based on government data |
| CN107430712A (en) * | 2014-12-19 | 2017-12-01 | 艾诺茨Ip公司 | Resource management system is can access with the network that can be allocated management of |
| CN107506462A (en) * | 2017-08-30 | 2017-12-22 | 中国建设银行股份有限公司 | Data processing method, system, electronic equipment, the storage medium of Enterprise Data |
| CN107657052A (en) * | 2017-10-17 | 2018-02-02 | 上海计算机软件技术开发中心 | A kind of data governing system based on metadata management |
| CN108171638A (en) * | 2017-12-20 | 2018-06-15 | 中国电子科技集团公司信息科学研究院 | A kind of Urban Data governing system |
| WO2018236886A1 (en) * | 2017-06-21 | 2018-12-27 | Opera Solutions Usa, Llc | System and method for code and data versioning in computerized data modeling and analysis |
| CA3072045A1 (en) * | 2017-08-02 | 2019-02-07 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
-
2019
- 2019-04-11 CN CN201910288687.6A patent/CN110019176B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002054171A2 (en) * | 2000-12-06 | 2002-07-11 | Biosentients, Inc. | System, method, software architecture and business model for an intelligent object based information technology platform |
| CN104143122A (en) * | 2013-05-07 | 2014-11-12 | 天津冠创科技有限公司 | Intelligent service approval scheme |
| CA2867589A1 (en) * | 2013-10-15 | 2015-04-15 | Coho Data Inc. | Systems, methods and devices for implementing data management in a distributed data storage system |
| CN106664224A (en) * | 2014-08-20 | 2017-05-10 | 华为技术有限公司 | System and method for metadata enhanced inventory management of a communications system |
| CN107430712A (en) * | 2014-12-19 | 2017-12-01 | 艾诺茨Ip公司 | Resource management system is can access with the network that can be allocated management of |
| CN105760980A (en) * | 2015-11-27 | 2016-07-13 | 国网山东省电力公司潍坊供电公司 | Intelligent operation system based on intelligent power grid framework |
| CN107247788A (en) * | 2017-06-15 | 2017-10-13 | 山东浪潮云服务信息科技有限公司 | A kind of method of the comprehensive regulation service based on government data |
| WO2018236886A1 (en) * | 2017-06-21 | 2018-12-27 | Opera Solutions Usa, Llc | System and method for code and data versioning in computerized data modeling and analysis |
| CA3072045A1 (en) * | 2017-08-02 | 2019-02-07 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for detection in an industrial internet of things data collection environment with large data sets |
| CN107506462A (en) * | 2017-08-30 | 2017-12-22 | 中国建设银行股份有限公司 | Data processing method, system, electronic equipment, the storage medium of Enterprise Data |
| CN107657052A (en) * | 2017-10-17 | 2018-02-02 | 上海计算机软件技术开发中心 | A kind of data governing system based on metadata management |
| CN108171638A (en) * | 2017-12-20 | 2018-06-15 | 中国电子科技集团公司信息科学研究院 | A kind of Urban Data governing system |
Non-Patent Citations (1)
| Title |
|---|
| 大数据视域下高校数据治理方案研究;余鹏;《现代教育技术》;第28卷(第6期);60-66 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019176A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019176B (en) | Data management control system for improving success rate of data management service | |
| CN112699175B (en) | Data management system and method thereof | |
| CN110765337B (en) | Service providing method based on internet big data | |
| US10599684B2 (en) | Data relationships storage platform | |
| CN112163724A (en) | Environment information data resource integration system | |
| Pardillo et al. | Using ontologies for the design of data warehouses | |
| CN110119395B (en) | Method for realizing association processing of data standard and data quality based on metadata in big data management | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| CN112527774A (en) | Data center building method and system and storage medium | |
| US20110040805A1 (en) | Techniques for parallel business intelligence evaluation and management | |
| CN110928963B (en) | Column-level authority knowledge graph construction method for operation and maintenance service data table | |
| CN106528828A (en) | Multi-dimensional checking rule-based data quality detection method | |
| CN112651218A (en) | Automatic generation method and management method of bidding document, medium and computer | |
| CN117472874A (en) | Government affair data resource integrated management system and method based on big data analysis | |
| CN115617776A (en) | Data management system and method | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN115221337A (en) | Data weaving processing method and device, electronic equipment and readable storage medium | |
| CN110134688B (en) | Hot event data storage management method and system in online social network | |
| CN115952160A (en) | Data checking method | |
| Chen | Database Design and Implementation | |
| CN111291029B (en) | Data cleaning method and device | |
| Santos et al. | Using relational algebra on the specification of real world ETL processes | |
| CN118396087B (en) | Block chain-based enterprise digital map construction method and system | |
| Li et al. | Research on Relational Database Fusion Method for Data Mining | |
| CN114139979A (en) | Service platform for specific research and development mechanism |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |