CN111143322A - Data standard treatment system and method - Google Patents
Data standard treatment system and method Download PDFInfo
- Publication number
- CN111143322A CN111143322A CN201911104345.0A CN201911104345A CN111143322A CN 111143322 A CN111143322 A CN 111143322A CN 201911104345 A CN201911104345 A CN 201911104345A CN 111143322 A CN111143322 A CN 111143322A
- Authority
- CN
- China
- Prior art keywords
- data
- standard
- rule
- file
- data standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data standard management system, which comprises a standard management module, a business object management module, a data standard mapping module, a standard execution task creation module, a standard execution module, a first task analysis module and a second task analysis module, and solves the problem that the data apertures of different departments are inconsistent: the meaning, the representation mode and the code of the service data are not uniform, and the credibility of the data is low; the shared data among different services can not be effectively shared, and the scientificity of management decision is influenced.
Description
Technical Field
The invention relates to the technical field of data standards, in particular to a data standard treatment system.
Background
Government departments, internet enterprises, and large group enterprises have accumulated and precipitated a large amount of data resources. China has become one of the countries with the largest data generation and accumulation amount and the most abundant data types, and data has become the first resource from the perspective of national strategy and urban strategy. However, in the process of informatization construction, enterprises and governments face the same problem, and data is wasted. In the initial stage of information construction, organizations construct a large number of systems which meet the current requirements of services, unified planning and management are lacked, data standards are not unified, different services describe the same data differently, and data formats are different, so that a large amount of data cannot be used for decision making, and the quality is low.
The government affair IT system lacks unified planning and deployment in the initial stage of construction, leads to different departments data bore inconsistency: the meaning, the representation mode and the code of the service data are not uniform, and the credibility of the data is low; shared data among different services cannot be effectively shared, and scientificity of management decision is influenced. By constructing the data standard packet, the data quality problem of the coding property is solved, so that the data verification is more comprehensive and accurate, the problem that the original ubiquitous data standard and data are separated into two pieces of data is solved, and the high unification of the data standard and the data is achieved.
Disclosure of Invention
In view of the above problems, the present invention provides a data standard management system and method, which overcome the problem that data is difficult to share and exchange because different services have different definitions, different data descriptions, and different data formats for the same data in an organization.
In order to achieve the above object, an embodiment of the present invention provides a data standard governance system, including:
the standard management module is used for creating or modifying the data standard file and the data rule file and is also used for associating the data source with the data standard file;
the business object management module is used for respectively creating corresponding business objects for different data sources and classifying the created business objects;
the data standard mapping module is used for correlating the created service object with the data standard file;
the standard execution task creating module is used for creating a task for enabling the business object to execute the standard of the data standard file and the rule of the data rule file;
the standard execution module is used for executing the standard execution task created by the standard execution task creation module;
the first task analysis module is used for searching for the business objects which are inconsistent with the data standards corresponding to the standard execution tasks;
and the second task analysis module is used for supervising the standard execution task and determining an execution error log of the standard execution task.
Further, the data standard file comprises a standard packet and reference data, and the data rule file comprises a value domain analysis rule, a dictionary rule, a function dependency rule, an SQL rule, a regular expression rule and an inclusion dependency rule.
Further, the modifying the data standard file comprises dynamically expanding each type of data standard.
Further, the data standard mapping module determines an attributed data standard topic according to the field meaning of the created service object, and searches data standards in the corresponding data standard topic for association to form a mapping result table.
A data standard governance method, comprising:
creating or modifying a data standard file and a data rule file, and associating a data source with the data standard file;
respectively creating corresponding business objects for different data sources, and classifying the created business objects;
correlating the created business object with the data standard file;
a task of creating a rule that causes the business object to execute the standard of the data standard file and the data rule file;
executing the created standard execution task;
searching for a business object with inconsistent data standard corresponding to the standard execution task;
and supervising the standard execution task, and determining an execution error log of the standard execution task.
Further, the data standard file comprises a standard packet and reference data, and the data rule file comprises a value domain analysis rule, a dictionary rule, a function dependency rule, an SQL rule, a regular expression rule and an inclusion dependency rule.
Further, the modifying the data standard file includes dynamically extending the data standards of each type.
Further, an attributive data standard theme is determined according to the field meaning of the created service object, and data standards are searched in the corresponding data standard theme for association to form a mapping result table.
The embodiment of the invention provides a data standard management system and a method, comprising a standard management module, a data standard file and a data rule file, wherein the standard management module is used for creating or modifying the data standard file and the data rule file and is also used for associating a data source with the data standard file; the business object management module is used for respectively creating corresponding business objects for different data sources and classifying the created business objects; the data standard mapping module is used for correlating the created service object with the data standard file; the standard execution task creating module is used for creating a task for enabling the business object to execute the standard of the data standard file and the rule of the data rule file; the standard execution module is used for executing the standard execution task created by the standard execution task creation module; the first task analysis module is used for searching for the business objects which are inconsistent with the data standards corresponding to the standard execution tasks; and the second task analysis module is used for supervising the standard execution task and determining an execution error log of the standard execution task. The problem that the data apertures of different departments are not consistent is solved: the meaning, the representation mode and the code of the service data are not uniform, and the credibility of the data is low; the shared data among different services can not be effectively shared, and the scientificity of management decision is influenced.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 shows a data standard abatement application schematic.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
In the description and claims of the present invention and in some of the flows described in the above drawings, a plurality of operations are included in a specific order, but it should be clearly understood that these operations may be executed out of the order they appear herein or in parallel, and it should be noted that "first", "second", etc. are described herein for distinguishing different messages, devices, modules, etc. without representing a sequential order, and without limiting "first" and "second" to be different types.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In a multi-index evaluation system, the evaluation indexes are different in size and magnitude due to different properties. When the levels of the indexes are greatly different, if the original index values are directly used for analysis, the function of the indexes with higher numerical values in the comprehensive analysis is highlighted, and the function of the indexes with lower numerical levels is relatively weakened. Therefore, in order to ensure the reliability of the result, the raw index data needs to be standardized.
Normalization of data (normalization) is to scale data to fall within a small specific interval. In some index processing for comparison and evaluation, unit limitation of data is removed and converted into a dimensionless pure numerical value, so that indexes of different units or orders can be compared and weighted conveniently. The most typical of these is the normalization of the data, i.e. the uniform mapping of the data onto the [0,1] interval.
At present, there are various data standardization methods, which can be classified into a linear method (e.g., extreme method, standard deviation method), a broken line method (e.g., three-broken line method), and a curve method (e.g., semi-normal distribution). Different normalization methods will have different effects on the system evaluation results, but unfortunately no general rule can be followed in the selection of the data normalization method.
In fig. 1, the standard architecture management provides functions of data standard documents, data meta standards, standard phrases, standard terms, etc. to help clients establish global standard management. Mapping of data standard system and service object, and associating defined service object in current system with standard packet and rule in data standard. The data standard subject to which the field belongs is usually determined according to the service meaning of the system field, the defined data standard is searched and associated in the corresponding subject standard (the cleaning and fusion of the data is realized through a data cleaning production line, a standard library is established, and thus a basic library and a subject library and a special library of each office are formed), and a mapping result table is formed.
With reference to fig. 1, a data standard abatement system comprising: the standard management module is used for creating or modifying the data standard file and the data rule file and is also used for associating the data source with the data standard file;
the data standard file comprises a standard packet and reference data, and the data rule file comprises a value domain analysis rule, a dictionary rule, a function dependence rule, an SQL rule, a regular expression rule and an inclusion dependence rule.
The standard packet management + reference data provides a data standard creating function, and various types of data standards can be dynamically expanded. The maintenance management of specific data standards is realized, the functions of editing, modifying and deleting the names, descriptions and the like of the standard packages are included, the functions of configuring jar packages and importing reference data of custom data standards can be realized, and the functions of previewing, editing and updating the jar packages and the reference data can be realized.
The data element management is to manage the data elements uniformly according to the standard and associate the data elements with the standard packet at the same time so as to map the data elements with the service objects.
The rule management realizes the unified management of the data rules. The main management rules are: value domain analysis, dictionary rules, function dependency rules, SQL rules, regular expression, contain dependencies, and the like.
The standard management in fig. 1 is versioning management of standard files, and the standard files are classified according to national standards, local standards, international standards, and industry standards. And standard version management is provided, so that clients can consult standard contents in different periods and compare the standard contents.
The business object management module is used for respectively creating corresponding business objects for different data sources and classifying the created business objects; the business object management defines corresponding business objects according to the data source and carries out grouping and classification on the defined business objects; substring generation (i.e., data table attribute customization) may also be performed on fields of business objects.
The data standard mapping module is used for correlating the created service object with the data standard file; the mapping of the data standards architecture to the business objects in fig. 1, the process of associating the defined business objects in the current system with the standard packages and rules in the data standards. Generally, a data standard subject to which a field belongs is determined according to the service meaning of the system field, a defined data standard is searched in a corresponding subject standard for association, and a mapping result table is formed.
The standard execution task creating module is used for creating a task for enabling the business object to execute the standard of the data standard file and the rule of the data rule file; and after the standard mapping is established, establishing a standard execution scheme. The standard execution scheme supports the scheduling in the system and also can support a third party outside the system to schedule products. And after the execution scheme is designed, the approval is carried out according to the relevant flow, and the execution scheme after the approval is implemented in the relevant system.
The standard execution module is used for executing the standard execution task created by the standard execution task creation module;
the first task analysis module is used for searching for the business objects which are inconsistent with the data standards corresponding to the standard execution tasks; the method mainly comprises the following two aspects of data standard difference analysis and data standard execution log analysis: and (3) data standard difference analysis, wherein the front-end analysis evaluates through the difference analysis, compares the difference of the standard execution and identifies the specific situation which does not meet the definition of the data standard. The analysis content is mainly carried out from two dimensions of a standard and a business object, and the standard dimension mainly reflects that the attribute of the business object violates the current specified inspection standard; from the business object dimension, how much question data the current business object has, and what criteria or rules the question data violates. And meanwhile, the difference analysis result is exported.
And the second task analysis module is used for supervising the standard execution task and determining an execution error log of the standard execution task. And performing log analysis by the data standard, and analyzing the execution condition of the current business object. The main analysis is the execution error log.
The modifying data standard file comprises dynamically expanding each type of data standard. And the data standard mapping module determines an attributive data standard theme according to the field meaning of the created service object, searches data standards in the corresponding data standard theme for association, and forms a mapping result table.
And the system can also be subjected to standard system conformance detection to help a client clearly see the standard operation condition in the organization. The governance method can also comprise standard data query, standard global query providing, standard content display, subscription, detection, evaluation, standard coverage and other conditions. And standard change management can be provided, and change application of the data element standard is provided, so that the checked data element standard becomes more standard.
When the data standard treatment system is applied, the data system standard can be subjected to custom control, such as a basic layer in fig. 1, and the data standard treatment system has a user management function, manages users of a platform, and provides addition, modification, deletion and user authority distribution; and (3) authority management: unified management is carried out on the authority of the platform;
role management function: adding and deleting roles, and assigning authority to the roles: the method comprises function authorization, data authorization and data source authorization, wherein the data authorization is to give access rights (addition, deletion, modification and check) of specified data standards to specified roles;
and (4) function management: the main function is to configure a function menu;
data source management: managing data sources related to the platform; providing addition, modification and deletion;
system logging: the reference system log mainly comprises the operation time, the user name, the IP address and the like.
As in the presentation layer in fig. 1, the presentation layer presents a data standard deviation analysis report, a data standard execution log analysis report, and a data standard deviation analysis report.
The invention discloses a standard data governance system, which comprises a standard management module, a standard data governance module and a data rule management module, wherein the standard management module is used for creating or modifying a standard data file and a data rule file and is also used for associating a data source with the standard data file; the business object management module is used for respectively creating corresponding business objects for different data sources and classifying the created business objects; the data standard mapping module is used for correlating the created service object with the data standard file; the standard execution task creating module is used for creating a task for enabling the business object to execute the standard of the data standard file and the rule of the data rule file; the standard execution module is used for executing the standard execution task created by the standard execution task creation module; the first task analysis module is used for searching for the business objects which are inconsistent with the data standards corresponding to the standard execution tasks; and the second task analysis module is used for supervising the standard execution task and determining an execution error log of the standard execution task. The problem that the data apertures of different departments are not consistent is solved: the meaning, the representation mode and the code of the service data are not uniform, and the credibility of the data is low; the shared data among different services can not be effectively shared, and the scientificity of management decision is influenced.
In the embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.
It will be understood by those skilled in the art that all or part of the steps in the method for implementing the above embodiments may be implemented by hardware that is instructed to implement by a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
While the data analysis method and system provided by the present invention have been described in detail, those skilled in the art will appreciate that the present invention is not limited to the above embodiments, and that various modifications, additions, substitutions, and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.
Claims (8)
1. A data standard abatement system comprising:
the standard management module is used for creating or modifying the data standard file and the data rule file and is also used for associating the data source with the data standard file;
the business object management module is used for respectively creating corresponding business objects for different data sources and classifying the created business objects;
the data standard mapping module is used for correlating the created service object with the data standard file;
the standard execution task creating module is used for creating a task for enabling the business object to execute the standard of the data standard file and the rule of the data rule file;
the standard execution module is used for executing the standard execution task created by the standard execution task creation module;
the first task analysis module is used for searching for the business objects which are inconsistent with the data standards corresponding to the standard execution tasks;
and the second task analysis module is used for supervising the standard execution task and determining an execution error log of the standard execution task.
2. The data standard governance system of claim 1, wherein the data standard file comprises a standard package and reference data, and the data rule file comprises a value domain analysis rule, a dictionary rule, a function dependency rule, an SQL rule, a regular expression rule, and an inclusion dependency rule.
3. The data standard governance system of claim 2, wherein modifying the data standard file comprises dynamically extending each type of data standard.
4. The data standard governance system of claim 1, wherein the data standard mapping module determines an attributed data standard topic according to the field meaning of the created business object, and searches for a data standard in the corresponding data standard topic for association to form a mapping result table.
5. A data standard governance method, comprising:
creating or modifying a data standard file and a data rule file, and associating a data source with the data standard file;
respectively creating corresponding business objects for different data sources, and classifying the created business objects;
correlating the created business object with the data standard file;
a task of creating a rule that causes the business object to execute the standard of the data standard file and the data rule file;
executing the created standard execution task;
searching for a business object with inconsistent data standard corresponding to the standard execution task;
and supervising the standard execution task, and determining an execution error log of the standard execution task.
6. The data standard governance method according to claim 5, wherein the data standard file comprises a standard package and reference data, and the data rule file comprises a value domain analysis rule, a dictionary rule, a function dependency rule, an SQL rule, a regular expression rule and an inclusion dependency rule.
7. The data standard governance method according to claim 6, wherein said modifying the data standard file comprises dynamically extending each type of data standard.
8. The data standard governance method according to claim 5, wherein an attributive data standard topic is determined according to the field meaning of the created business object, and the data standard is searched in the corresponding data standard topic for association to form a mapping result table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911104345.0A CN111143322A (en) | 2019-11-13 | 2019-11-13 | Data standard treatment system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911104345.0A CN111143322A (en) | 2019-11-13 | 2019-11-13 | Data standard treatment system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111143322A true CN111143322A (en) | 2020-05-12 |
Family
ID=70517058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911104345.0A Pending CN111143322A (en) | 2019-11-13 | 2019-11-13 | Data standard treatment system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143322A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117540353A (en) * | 2023-11-20 | 2024-02-09 | 和创（北京）科技股份有限公司 | System and method for managing service authority based on RBAC model |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102024199A (en) * | 2010-06-04 | 2011-04-20 | 西本新干线股份有限公司 | Service rule engine and generation method of service process |
| CN102034194A (en) * | 2009-09-29 | 2011-04-27 | 上海博科资讯股份有限公司 | Logistics billing method based on rule |
| CN102592203A (en) * | 2012-03-18 | 2012-07-18 | 西北工业大学 | Rule engine based KPI (Key Performance Indicator) generation method in business activity monitoring |
| CN104361221A (en) * | 2014-10-31 | 2015-02-18 | 沈阳锐易特软件技术有限公司 | Heterogeneous system data mapping template-based medical data acquisition system and method |
| CN108492028A (en) * | 2018-03-21 | 2018-09-04 | 徐欣 | Demand data standardized method and standardized system |
| CN110069633A (en) * | 2019-04-24 | 2019-07-30 | 普元信息技术股份有限公司 | Big data realizes that auxiliary formulates the system and method for data standard in administering |
-
2019
- 2019-11-13 CN CN201911104345.0A patent/CN111143322A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102034194A (en) * | 2009-09-29 | 2011-04-27 | 上海博科资讯股份有限公司 | Logistics billing method based on rule |
| CN102024199A (en) * | 2010-06-04 | 2011-04-20 | 西本新干线股份有限公司 | Service rule engine and generation method of service process |
| CN102592203A (en) * | 2012-03-18 | 2012-07-18 | 西北工业大学 | Rule engine based KPI (Key Performance Indicator) generation method in business activity monitoring |
| CN104361221A (en) * | 2014-10-31 | 2015-02-18 | 沈阳锐易特软件技术有限公司 | Heterogeneous system data mapping template-based medical data acquisition system and method |
| CN108492028A (en) * | 2018-03-21 | 2018-09-04 | 徐欣 | Demand data standardized method and standardized system |
| CN110069633A (en) * | 2019-04-24 | 2019-07-30 | 普元信息技术股份有限公司 | Big data realizes that auxiliary formulates the system and method for data standard in administering |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117540353A (en) * | 2023-11-20 | 2024-02-09 | 和创（北京）科技股份有限公司 | System and method for managing service authority based on RBAC model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11755628B2 (en) | Data relationships storage platform | |
| US8108367B2 (en) | Constraints with hidden rows in a database | |
| US8615768B2 (en) | Dependency-ordered resource synchronization across multiple environments using change list created based on dependency-ordered graphs of the multiple environments | |
| CN111190881A (en) | Data management method and system | |
| CN111858615B (en) | Database table generation method, system, computer system and readable storage medium | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| CN106682096A (en) | Method and device for log data management | |
| US10885087B2 (en) | Cognitive automation tool | |
| CN114357088A (en) | Nuclear power industry data warehouse system | |
| US11615076B2 (en) | Monolith database to distributed database transformation | |
| CN115617776A (en) | Data management system and method | |
| CN115794839B (en) | Data collection method based on Php+Mysql system, computer equipment and storage medium | |
| CN114880405A (en) | Data lake-based data processing method and system | |
| Al-Barak et al. | Database design debts through examining schema evolution | |
| CN112988919A (en) | Power grid data market construction method and system, terminal device and storage medium | |
| CN110955801B (en) | Knowledge graph analysis method and system for cognos report indexes | |
| US20200387802A1 (en) | Dynamically adaptable rules and communication system for managing process controls | |
| CN111143322A (en) | Data standard treatment system and method | |
| CN116578614A (en) | Data management method, system, medium and equipment for pipeline equipment | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template | |
| US20220121721A1 (en) | Custom types controller for search engine support | |
| CN114124977A (en) | Cross-tenant data sharing method and device and electronic equipment | |
| CN104040537A (en) | Systems and methods of automatic generation and execution of database queries | |
| US11977997B1 (en) | Tagging systems and methods for efficient cloud service provisioning | |
| JP2015204057A (en) | Data processor, data processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |