CN114417859A - Data standardization method and system based on cloud block chain technology - Google Patents

Abstract

The invention discloses a data standardization method and system based on a cloud block chain technology, which comprises data standard formulation, data standard approval, data standard countersigning, data standard modification, data standard confirmation, data standard sharing and data standard application; the system applying the standardization method comprises a data input port, a data standard module, a data dictionary, a data model, a data quality module and a data assessment module. Wherein the data normalization method is coordinated with the blockchain technique.

Description

Data standardization method and system based on cloud block chain technology

Technical Field

The invention relates to the technical field of data standardization, in particular to a data standardization method and system based on a cloud block chain technology.

Background

Along with the advent of the big data era, the value of the data is widely agreed by people, and the importance of the data is improved to an unprecedented level. Data is becoming the most important strategic asset in the era of digital economy, and scientific data management and control can change data into production elements. The data standard is a precondition and an important means for data management and control. However, at present, each unit defines the standard content of the unit by itself, and a great deal of manpower, material resources and financial resources are repeatedly input. The standard formulation, standard management and standard landing levels of all units are different, and the fragmented data standard seriously blocks the data sharing capability among all units and cannot give full play to the due value of data.

Disclosure of Invention

The invention provides a data standardization method and system based on a cloud block chain technology, which can improve data standard management capacity, circulation capacity and landing capacity and improve data quality.

The technical scheme of the invention is as follows:

a data standardization method based on a cloud block chain technology comprises the steps of standard making, examining and approving, countersigning, modifying, confirming, sharing and applying.

Furthermore, a standard special committee consisting of a plurality of users is used as a main standard management user, the standard is established, and the data standard is updated according to market changes or customer demand changes in the fields of the world, the country, the region, the industry, the place and the like.

Furthermore, the method completes the preliminary analysis of standard contents by standard word splitting, automatically completes the supplement of standard English names and English abbreviations, preliminarily determines the standard data types according to the split data elements, and submits the approval by standard operation and maintenance personnel after the increase according to the actual conditions.

Furthermore, the standard management users receive the standard countersigning notification at the same time, after all the standard management users pass the audit, the standard is changed, the state to be issued is entered, and the standard becomes effective after the operation and maintenance personnel issue; and if the audit of the management user fails, the standard application returns to the standard operation and maintenance personnel for continuous adjustment.

Furthermore, a standard user can migrate the data standard in the public network environment to the system environment of the unit for use through a network or file mode, meanwhile, the user can upload the standard defined by the unit, the platform collects the user standards, compares the differences, and forms a standard knowledge base to serve as a basis for updating the subsequent standard.

Further, the uplink operation of the data standard is completed, specifically as follows: the method comprises the steps of utilizing a client account to carry out intelligent contract deployment, using an SDK to realize interaction of data and a block chain network, initiating a transaction by a client, transferring the SDK to submit a transaction request, verifying whether a format of a request is correct, whether a signature is valid, whether an initiator has the right to initiate a corresponding request, simulating and executing the transaction, generating a read set and a write set for updating, carrying out the signature, and returning a response to the client. Secondly, the method comprises the following steps: further, the client responds and queries the request result and the endorsement information of the query response to decide whether to submit the update or not. The client submits the collected endorsement responses and requests to an ordering server sorting node, and the ordering server sorts the collected requests according to channels and forms blocks according to an agglomeration strategy. Thirdly, the method comprises the following steps: further, the editing server broadcasts the tile to all peers of the corresponding channel, and the peers independently check whether each transaction in the tile is valid and mark the corresponding tile. And the peer checks whether the content of the transaction conforms to the endorsement policy, if so, the peer adds the block to the tail of the local block chain, updates the write set in the valid transaction into the state database, and simultaneously informs the client.

Further, the method for applying the data standard comprises the following steps:

and using the Chinese name of the object to be standardized, carrying out complete keyword search in the language library, returning the language if the Chinese name is searched, replacing the object to be standardized by using English abbreviation and domain (including data type, length and precision) of the standard language, and filling the object to be standardized into a model tool.

If the search is not successful, using the Chinese name of the object to be standardized to perform complete keyword search on the alias in the language library, returning the standard term corresponding to the alias if the search is successful, replacing the object to be standardized by using the Chinese name, English abbreviation and domain of the standard term, and filling the object to be standardized into a model tool.

If the search is not successful, performing word segmentation processing on the Chinese name of the object to be standardized by using a standard word and a non-standard word set in a standard word library to obtain a word list after word segmentation, if the word list is completely contained in the standard word and the non-standard word set, determining that all word segmentation is successful, and if the word list contains the non-standard words, converting the word list into the corresponding standard words. And after the conversion is finished, replacing the combination sequence based on the words in the word list to obtain the expression of all the words combined in different sequences. And using the expressions combined in different orders to perform complete keyword search in the standard expression library and the alias, returning the standard expressions if the expressions are searched, replacing the standard objects by using Chinese names, English abbreviations and domains of the standard expressions, and filling the objects into a model tool.

If the search is not successful, removing words which are not contained in the word library from the word list obtained by word segmentation, calculating the combination condition of all standard words, and performing fuzzy matching in the standard language library, wherein the fuzzy matching uses the standard words as the minimum complete matching unit. And scoring the matching result, wherein the scoring standard is calculated by accumulating the number of standard words contained in the searched expression, namely the score containing a large number of standard words is high, the score is finally calculated by giving a weight based on the matching condition, when the weight is calculated, the fuzzy matching of the combined word is higher than the sum of the successful matching of all the standard words, the scoring result list is finally output, the standard expression with the highest score is used as the default matching result, and other standard expressions with low scores are used as alternative results. In order to improve the intelligent matching degree, the words which are not contained in the standard word library can be output as missing words, and the word library of the data standard knowledge library is supplemented.

The invention intelligently recommends and quotes standard items in the model design and realizes the automatic label falling in the model design stage. After the model design is finished, the logical model and the physical model can be automatically checked, and a review report is generated by one key, wherein the review report comprises the steps of label falling check, definition integrity check, similarity check of an entity, an identifier, a main key and the like, isolation analysis check, physical information normalization check and the like of the model.

The invention also comprises a system applying the data standardization method, which comprises a data input port, a data standard module, a data dictionary, a data model, a data quality module and a data assessment module. The data input port inputs data into the system in an autonomous acquisition mode, a manual input mode and the like; the data standard module is used in a public network environment, is combined with a block chain technology, and provides the establishment and the release of public, transparent, tamper-proof, multi-party consensus service standards, technical standards, index standards, service terms and other contents; the data dictionary module is used for acquiring a base table structure of a specified database through an acquisition tool under a network environment where an enterprise is located, and can establish a corresponding database data dictionary through system processing and manual data supplement, and realize primary standard landing; the data model module provides a visual modeling tool, and automatically applies the existing standard to realize standard landing while modeling data; the data quality module is used for checking the data quality of the specified data in the specified database by self-defining checking content under the network environment of the enterprise, and can distribute the found data quality problems to related responsible persons for problem repair; the data examination module can define an examination scheme by self, examine the data quality problem generation and repair conditions of unit workers, and achieve the aim of improving the data quality in a reward and punishment mode.

The technical scheme provided by the invention has the beneficial effects that:

the data standard is an important basic tool in the digital era, is an important foundation for promoting the market construction of data elements, the platform adheres to the openness principle of 'once construction and reuse', adopts an operation mechanism of 'openness and compatibility, cooperation and sharing and advancement with time', continuously adapts to the working requirements of the data standard in development, gathers the wisdom of people in an open mode, and creates a high-quality data standard which has strong universality, good landing performance, high flexibility and convenient application in a timely and innovative manner.

The consistency of terms and domains in the data use process is guaranteed, and the manual error rate is reduced. Inconsistency of design specifications among systems in the enterprise information construction process can cause increase of process costs of communication, conversion, cleaning and the like in the data integration process. Meanwhile, in the process of designing the data model, in order to ensure the consistency of terms and domains, the construction of an enterprise-level unified data standard knowledge base and the stage of designing the data model need to be realized by an automatic application means, so that in the technical scheme provided by the invention, the application is realized by the automatic means on the basis of establishing the data standard knowledge base, and the error rate of manual conversion is avoided. The efficiency of artifical selection and matching is improved, the human input is reduced. The unification of terms and domains in the design process of the data model is not easy once and for all, even if the terms and the domains are applied in a partial automatic mode for the first time, the conditions of new addition and change still need to be concerned continuously in the process of continuously expanding the system functions, therefore, the method comprises the characteristic of automatic association of the data standard knowledge base, not only effectively reduces the labor input in the process of first application, but also plays a role in improving the efficiency in the process of continuous maintenance. The invention combines standard formulation and use process with block chain technology, utilizes the characteristics of block chain disclosure, transparency, tamper resistance, multi-party consensus and the like, ensures legal compliance of data standard formulation and application in key processes of authority and management control of data standard approval, confirmation and sharing of data standard, audit and supervision of data standard and the like by evidence storage and traceability of data standard formulation, approval, modification, confirmation, sharing, application and the like, improves the consistency of enterprise understanding professional terms, and improves communication efficiency. The construction of the enterprise-level data standard knowledge base plays a role in unifying the service aperture and the data interaction aperture, avoids the communication efficiency in the process of system integration and data exchange, and also reduces the difficulty and the error rate of integration and interaction.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of a standard release according to an embodiment of the present invention;

FIG. 2 is a block diagram of a data standard fortune pipe and block chain combination according to the present invention;

FIG. 3 is a schematic overall flow chart of the intelligent model design method based on the data standard knowledge base of the present invention;

FIG. 4 is a detailed flow diagram of the intelligent standardized model design of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

The first embodiment is as follows:

in the logical model of the data model design tool, the attributes to be standardized are selected as the objects to be standardized. The standard synchronization of the objects to be standardized in the selected range according to the words, domains and expressions in the custom data standard knowledge base comprises the following steps:

and (3) using the Chinese name of the object to be standardized, carrying out complete keyword search in the language library, returning the language if the keyword is searched, and replacing the object to be standardized by using English abbreviation and domain (including data type, length and precision) of the standard language. For example, the object to be standardized is 'e-mail', the standard phrase library is searched for the existence of 'e-mail', and the model tool is filled with English abbreviations, data types and lengths of 'e-mail' in the standard phrase library.

If the matching is not successful, the Chinese name of the object to be standardized is used, the alias is subjected to complete keyword search in the language library, if the alias is searched, the standard term corresponding to the alias is returned, and the Chinese name, English abbreviation and domain of the standard term are used for replacing the object to be standardized. For example, the object to be standardized is "Email", and the "Email" searched in the standard language library is an alias of the standard term "Email", the object to be standardized is replaced by the "Email" in the standard language library, and the english abbreviation, data type and length of the "Email" are filled into the model tool.

If the matching is not successful, performing word segmentation processing on the Chinese name of the object to be normalized by using a standard word and a non-standard word set in a standard word library, and obtaining a word list after word segmentation. And after the conversion is finished, replacing the combination sequence based on the words in the word list to obtain the expression of all the words combined in different sequences. For example, when the word to be standardized is "employee entry number", the standard word library includes three words "employee", "entry", and "number", and meanwhile, the "employee" in the non-standard word library corresponds to the standard word "employee". The method comprises the steps of using employees, numbers and employees to perform word segmentation on words to be standardized, obtaining three words of employees, employees and numbers as word segmentation results, and obtaining the employees, the employees and the numbers after performing non-standard word replacement. After the combination sequence is replaced, a plurality of terms consisting of standard words, such as the employee attendance number, the employee attendance number, the employee attendance number, and the employee number, are obtained.

And performing complete keyword search processing on the language library and the alias of the standard knowledge base by using all the combined terms obtained by the above-mentioned participle. For example, if the standard phrase library includes the standard phrase "employee number", and the obtained phrases are completely matched, the standard phrase is searched in the phrase library of the standard knowledge base, and after the "employee number" is replaced with the "employee number", the english abbreviation, data type, and length of the "employee number" are filled in the model tool.

If the matching is not successful, removing words which are not contained in the word library from the obtained word list, carrying out score calculation recommendation, and outputting the words which are not contained in the standard word library as missing words. For example, the object to be standardized, namely "security code information", is processed in multiple steps to obtain a word structure consisting of three words, namely "security + code + information", and since neither the standard word bank nor the non-standard word bank contains "information", the word structure is removed from the combination and output as a missing word, namely, two standard words of "security + code" are input in the next step.

And calculating the combination condition of all standard words according to the word list obtained by word segmentation, and then performing fuzzy matching in a standard language library, wherein the fuzzy matching takes the standard words as the minimum complete matching unit. And scoring the matching result, wherein the scoring reference is accumulated by the number of standard words contained in the searched expression, namely the score containing a large number of standard words is high. For example, when two standard words "security + code" are entered, all combinations of these two words are first constructed, resulting in "security", "code", "security code" and "code security", wherein "security" and "code" belong to the original standard words and "security code" and "code security" belong to the combined words of the original words. The four words are used for searching in the standard phrase library, if the standard phrase library contains a 'listed security code', the fuzzy matching of the combination words is successful, if the 'listed security unique code' exists in the standard phrase library, the fuzzy matching of the combination words fails, but the standard words are successfully matched, and two standard words of 'securities' and 'codes' are contained, if the 'listed security name' exists in the standard phrase library, the fuzzy matching of the combination words fails, but the standard words are successfully matched, and only one standard word is contained. And finally, giving a weight calculation score based on the matching condition, wherein when the weight value is calculated, the fuzzy matching of the combined word is higher than the sum of successful matching of all standard words, the standard word matching result is calculated according to the number of the standard words, the calculation is carried out according to the condition, the number of the input standard words is 2, the combined word matching result is larger than 2, and for a plurality of combined words, which are successfully matched, the combination word is calculated according to the condition that the positive sequence combined result score is larger than the negative sequence combined score, namely, the score of the security code is higher than the code security, and the combined word matching result is larger than the scores of the standard word matching result, namely, the unique code of the security on the market and the name of the security on the market. And outputting the final scoring result in a list, wherein the standard expression with the highest score is used as a default matching result, and other standard expressions with low scores are used as alternative results. After the model design and the standard automatic reference are completed, the quality of the model is checked, including the falling standard check, the definition integrity check, the similarity check of entities, identifiers, main keys and the like, the isolation analysis check, the physical information normalization check and the like of the model, and the check report of the model is automatically generated.

Example two:

a system applying the data standardization method comprises a data input port, a data standard module, a data dictionary, a data model, a data quality module and a data assessment module. The data input port is a computer and a device which can automatically identify preset data of a user, and the data is input into the system in the modes of automatic acquisition, manual input and the like; the data standard module is used in a public network environment, is combined with a block chain technology, and provides the establishment and the release of public, transparent, tamper-proof, multi-party consensus service standards, technical standards, index standards, service terms and other contents; the data dictionary module is used for acquiring a base table structure of a specified database through an acquisition tool under a network environment where an enterprise is located, and can establish a corresponding database data dictionary through system processing and manual data supplement, and realize primary standard landing; the data model module provides a visual modeling tool, and automatically applies the existing standard to realize standard landing while modeling data; the data quality module is used for checking the data quality of the specified data in the specified database by self-defining checking content under the network environment of the enterprise, and can distribute the found data quality problems to related responsible persons for problem repair; the data examination module can define an examination scheme by self, examine the data quality problem generation and repair conditions of unit workers, and achieve the aim of improving the data quality in a reward and punishment mode.

Claims (3)

1. A data standardization method based on a cloud block chain technology comprises data standard formulation, data standard approval, data standard countersigning, data standard modification, data standard confirmation, data standard sharing and data standard application, and specifically comprises the following steps of S1: a user establishes a standard, and updates the data standard according to market change or customer demand change in the fields of the world, the country, the region, the industry, the place and the like; s2: completing the preliminary analysis of standard contents by standard word splitting, automatically completing the supplement of standard English names and English abbreviations, preliminarily determining the standard data type according to the split data elements, and submitting for approval after increasing according to actual conditions; s3: the user receives the standard signing notice, after all the users pass the verification, the standard is changed and passed, the state to be issued is entered, and the standard becomes effective after the operation and maintenance personnel issue; if the audit of the management user fails, the standard application returns to the standard operation and maintenance personnel for continuous adjustment; s4: a user migrates the data standard in the public network environment to the system environment of the unit for use in a network or file mode, meanwhile, the user can upload the standard defined by the unit, and the platform collects the standards of the users, compares the differences and forms a standard knowledge base as the basis for updating the subsequent standard; s5: sharing and application of data standards, characterized by further comprising uplink operations of the data standards as follows:

firstly, the method comprises the following steps: the method comprises the steps of utilizing a client account to carry out intelligent contract deployment, using an SDK to realize interaction of data and a block chain network, initiating a transaction by a client, transferring the SDK to submit a transaction request, verifying whether a format of a request is correct, whether a signature is valid, whether an initiator has the right to initiate a corresponding request, simulating and executing the transaction, generating a read set and a write set for updating, carrying out the signature, and returning a response to the client. Secondly, the method comprises the following steps: further, the client responds and queries the request result and the endorsement information of the query response to decide whether to submit the update or not. The client submits the collected endorsement responses and requests to an ordering server sorting node, and the ordering server sorts the collected requests according to channels and forms blocks according to an agglomeration strategy. Thirdly, the method comprises the following steps: further, the editing server broadcasts the tile to all peers of the corresponding channel, and the peers independently check whether each transaction in the tile is valid and mark the corresponding tile. And the peer checks whether the content of the transaction conforms to the endorsement policy, if so, the peer adds the block to the tail of the local block chain, updates the write set in the valid transaction into the state database, and simultaneously informs the client.

2. The data standardization method based on the cloud blockchain technology of claim 1, wherein the data standardization application comprises the following steps:

s1: and using the Chinese name of the object to be standardized, carrying out complete keyword search in the language library, returning the language if the Chinese name is searched, replacing the object to be standardized by using English abbreviation and domain (including data type, length and precision) of the standard language, and filling the object to be standardized into a model tool.

S2: if the search is not successful, using the Chinese name of the object to be standardized to perform complete keyword search on the alias in the language library, returning the standard term corresponding to the alias if the search is successful, replacing the object to be standardized by using the Chinese name, English abbreviation and domain of the standard term, and filling the object to be standardized into a model tool.

S3: if the search is not successful, performing word segmentation processing on the Chinese name of the object to be standardized by using a standard word and a non-standard word set in a standard word library to obtain a word list after word segmentation, if the word list is completely contained in the standard word and the non-standard word set, determining that all word segmentation is successful, and if the word list contains the non-standard words, converting the word list into the corresponding standard words. And after the conversion is finished, replacing the combination sequence based on the words in the word list to obtain the expression of all the words combined in different sequences.

S4: and using the expressions combined in different orders to perform complete keyword search in the standard expression library and the alias, returning the standard expressions if the expressions are searched, replacing the standard objects by using Chinese names, English abbreviations and domains of the standard expressions, and filling the objects into a model tool.

S5: if the search is not successful, removing words which are not contained in the word library from the word list obtained by word segmentation, calculating the combination condition of all standard words, and performing fuzzy matching in the standard language library, wherein the fuzzy matching uses the standard words as the minimum complete matching unit. And scoring the matching result, wherein the scoring standard is calculated by accumulating the number of standard words contained in the searched expression, namely the score containing a large number of standard words is high, the score is finally calculated by giving a weight based on the matching condition, when the weight is calculated, the fuzzy matching of the combined word is higher than the sum of the successful matching of all the standard words, the scoring result list is finally output, the standard expression with the highest score is used as the default matching result, and other standard expressions with low scores are used as alternative results.

3. An application data standardization system comprises a data input port, a data standard module, a data dictionary, a data model, a data quality module and a data assessment module. The data input port inputs data into the system in an autonomous acquisition mode, a manual input mode and the like; the data standard module is used in a public network environment, is combined with a block chain technology, and provides the establishment and the release of public, transparent, tamper-proof, multi-party consensus service standards, technical standards, index standards, service terms and other contents; the data dictionary module is used for acquiring a base table structure of a specified database through an acquisition tool under a network environment where an enterprise is located, and can establish a corresponding database data dictionary through system processing and manual data supplement, and realize primary standard landing; the data model module provides a visual modeling tool, and automatically applies the existing standard to realize standard landing while modeling data; the data quality module is used for checking the data quality of the specified data in the specified database by self-defining checking content under the network environment of the enterprise, and can distribute the found data quality problems to related responsible persons for problem repair; the data examination module can define examination schemes by self, and can achieve the purpose of improving data quality through reward and punishment by examining the data quality problems of unit workers and repairing conditions, and is characterized in that: the method used by the data standard module is the method of claim 1.

Images