CN108256074B - Verification processing method and device, electronic equipment and storage medium - Google Patents
Verification processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN108256074B CN108256074B CN201810045917.1A CN201810045917A CN108256074B CN 108256074 B CN108256074 B CN 108256074B CN 201810045917 A CN201810045917 A CN 201810045917A CN 108256074 B CN108256074 B CN 108256074B
- Authority
- CN
- China
- Prior art keywords
- standard
- field
- morpheme
- definition
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Machine Translation (AREA)
Abstract
The embodiment of the invention provides a verification processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining models of a data warehouse to be verified, wherein each model comprises a plurality of field information, and the field information comprises field definitions and field types; checking the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type; and if the field definition is matched with the standard definition and the field type is not matched with the standard type, modifying the field type to be consistent with the standard type. The method verifies the model of the data warehouse according to the standard expression, and when the field definition is matched with the standard definition and the field type is not matched with the standard type, the field type is pertinently modified to be consistent with the standard type, so that a standard consistent model is obtained.
Description
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a verification processing method, a verification processing device, electronic equipment and a storage medium.
Background
In order to make a decision better, a data warehouse is created, and the data warehouse provides data support for decision making.
The data warehouse comprises a large amount of data, wherein the data is obtained by extracting and cleaning the data of a plurality of original scattered databases, and carrying out system processing, summarizing and sorting on the data.
Because the data of the data warehouse has a plurality of data sources (databases), and names of the data sources may be different for one same field, if the same field is sorted into one data warehouse, a plurality of inconsistent names exist in one same field, which results in low quality of the data warehouse, and in the following, confusion in use when data is stored and read.
In the prior art, a manual checking mode is mainly adopted, so that the naming of each data is standard and consistent.
Due to the fact that experience and capability of each person are different, omission and wrong judgment can occur, and the data naming consistency in the data warehouse cannot be achieved.
Disclosure of Invention
In order to overcome the defects in the prior art, embodiments of the present invention provide a method and an apparatus for verification processing, an electronic device, and a storage medium.
In one aspect, an embodiment of the present invention provides a method for verification processing, where the method includes:
obtaining models of a data warehouse to be verified, wherein each model comprises a plurality of field information, and the field information comprises field definition and field type;
checking the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type;
and if the field definition is matched with the standard definition and the field type is not matched with the standard type, modifying the field type to be consistent with the standard type.
In another aspect, an embodiment of the present invention provides an apparatus for verification processing, where the apparatus includes:
the system comprises an acquisition module, a verification module and a verification module, wherein the acquisition module is used for acquiring models of a data warehouse to be verified, each model comprises a plurality of field information, and the field information comprises field definitions and field types;
the verification module is used for verifying the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type;
and the modification module is used for modifying the field type to be consistent with the standard type if the field definition is matched with the standard definition and the field type is not matched with the standard type.
In another aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the above method when executing the program.
In another aspect, an embodiment of the present invention further provides a storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the above method.
It can be known from the foregoing technical solutions that, in the verification method, the verification apparatus, the electronic device, and the storage medium provided in the embodiments of the present invention, the method verifies the model of the data warehouse according to the standard expression, and when the field definition matches the standard definition and the field type does not match the standard type, the field type is modified to be consistent with the standard type in a targeted manner, so as to obtain a standard consistent model.
Drawings
Fig. 1 is a schematic flowchart of a verification processing method according to an embodiment of the present invention;
fig. 2 is a schematic overall structure diagram of a verification processing apparatus according to another embodiment of the present invention;
fig. 3 is a schematic flowchart of a verification processing method according to another embodiment of the present invention;
FIG. 4 is a flowchart illustrating operation of an initialization phase according to another embodiment of the present invention;
FIG. 5 is a diagram of a portion of an example of a verify operation according to yet another embodiment of the present invention;
FIG. 6 is a diagram of a portion of an example of a verify operation according to yet another embodiment of the present invention;
FIG. 7 is a flowchart illustrating a verification operation according to another embodiment of the present invention;
fig. 8 is a schematic structural diagram of an apparatus for verification processing according to yet another embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to yet another embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.
Fig. 1 is a schematic flowchart illustrating a method for verification processing according to an embodiment of the present invention.
As shown in fig. 1, the method provided in the embodiment of the present invention specifically includes the following steps:
step 11, obtaining models of a data warehouse to be verified, wherein each model comprises a plurality of field information, and the field information comprises field definitions and field types;
alternatively, the construction of a data warehouse may be divided into two steps: first, a model of the data warehouse is designed, and second, data is written to a corresponding model (data table).
After the model design is completed, the method provided by the embodiment of the invention is applied to verify the model.
Optionally, at least one designed model is uploaded to the verification processing device, and a model is understood to be a data table, wherein the data table comprises a plurality of rows of data, and each row of data comprises corresponding field information.
Optionally, the field information includes a field definition and a field type, the field definition is a description of the meaning of the field, and may include a field name and a field description. The field type is a description of the type of the field, for example, the field is double or int, where double is a double-precision floating-point number, that is, the field may be a number with a decimal point, and int represents integer, that is, the field is an integer.
optionally, a data dictionary is created in advance, and the data dictionary comprises a plurality of standard expressions, and each standard expression is agreed and can be used as a standard expression of a unified standard.
Alternatively, the standard expression is collected from an industry professional expression dictionary, historical data warehouse data, wiki (Wikipedia), various professional books, and data.
Optionally, the standard wording includes a standard definition that is a standard description of a field and a standard type that indicates a type that the field may be used.
For example, the standard is defined as amount, the standard type double of the amount created in advance, and the amount does not use int as the standard type after the standard type is determined to be double.
Optionally, for the field definition of the model, the standard expression of the data dictionary is queried for whether there is a standard definition matching the field definition of the model.
And if the field definition is successfully matched with the standard definition of the standard expression, inquiring the standard type corresponding to the standard definition matched with the field definition of the model in the standard expression aiming at the field type of the model.
If the field definition does not match the standard definition of the standard expression successfully, the verification result is output as failure.
And step 13, if the field definition is matched with the standard definition and the field type is not matched with the standard type, modifying the field type to be consistent with the standard type.
And if the field definition of the model is consistent with the standard definition and the field type is not consistent with the standard type, remarking the model, wherein the remarking content is as follows: and if the field type is not consistent with the standard type, outputting a verification result, wherein the verification result comprises the remark.
In the verification process of the embodiment of the invention, remarks are added to provide modification suggestions, so that modification is executed subsequently according to the verification result, and the field type is modified to be consistent with the standard type.
If the field definition of the model is consistent with the standard definition and the field type is consistent with the standard type, the model is proved to be in accordance with the specification, and the verification result of the field information is successful.
It can be understood that if each data warehouse performs the method of the embodiment of the present invention during modeling, and checks are performed according to the data dictionary to obtain a consistent and standard data table, then the data warehouse can be directly filled into the standard data table during data filling.
According to the verification processing method provided by the embodiment, the model of the data warehouse is verified according to the standard expression, and when the field definition is matched with the standard definition and the field type is not matched with the standard type, the field type is pertinently modified to be consistent with the standard type, so that a standard consistent model is obtained.
On the basis of the foregoing embodiment, in a method for performing verification processing according to another embodiment of the present invention, the field definition includes a field name and a field description, and the standard definition includes a standard name and a standard description, and accordingly, according to a pre-stored data dictionary, the step of performing verification on field information specifically includes:
if the field name is matched with the standard name, checking whether the field description is consistent with the standard description or not, and checking whether the field type is consistent with the standard type or not;
or;
and if the field description is matched with the standard description, checking whether the field name is consistent with the standard name or not, and checking whether the field type is consistent with the standard type or not.
Optionally, the contents of a model include as shown in table 1:
TABLE 1
| Name of field | Field description | Type of field |
| Paidup_perf_amount | Achievement of real income | Double |
| …… | …… | …… |
Optionally, if the field name is successfully matched with the standard name, checking other fields (field description and field type) of the field information to determine whether the field name is consistent with the standard description and standard type corresponding to the successfully matched standard name.
If the field information is consistent with the standard expression, the field information is completely consistent with the standard expression, and the verification result is successful.
If not, the remark content is as follows: the field description and the field type are inconsistent with the standard expression for subsequent modification of the field description and the field type to be consistent with the standard expression.
Similarly, if the field description matches the standard description, a check is made for other fields (field name and field type) of the field information to see if they match the standard name and standard type corresponding to the standard field that successfully matches.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
According to the method for checking processing provided by the embodiment, the field name and the field description are checked respectively, so that the checking result can be accurately obtained.
On the basis of the foregoing embodiment, in a method for verification processing provided by another embodiment of the present invention, after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method includes:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes;
acquiring a pre-stored rule management base, wherein the rule management base comprises a plurality of replacement rules, and each replacement rule comprises a modifier and a classification word;
if the morpheme is matched with the modifier, judging whether the classified word of the morpheme exists or not;
and if not, replacing the morpheme with the morpheme and the corresponding classified word.
Optionally, if the field definition is not successfully matched with the standard definition of the standard expression, indicating that the verification using the data dictionary fails, the embodiment of the present invention is executed, and the verification is continued using the rule management base.
Optionally, a morpheme is the smallest word with a particular meaning, not resolvable, such as: day, month, income, city, etc.
For example, the sequence "customer-source-side performance" is split into three morphemes "customer source", "end", and "performance".
Optionally, word segmentation processing may be performed on the field information according to a prior art manner to obtain morphemes, and a rule management base is used to verify each morpheme.
Optionally, the rule management library includes a plurality of replacement rules, each replacement rule includes a modifier and a classifier, a relationship between the modifier and the classifier is a relationship between a fixed phrase and a core phrase, that is, a modified and modified relationship, the modifier is a morpheme used as the fixed phrase for adjectively classifying words, and the classifier is a morpheme used as the core phrase of the modifier.
For example, the "performance amount" is two morphemes, the "amount" is a central word, indicating that the "performance amount" belongs to the category of money, and is a money value, and the "performance" indicates that the value is a performance value, and not other values.
Optionally, the replacement rule is used to replace the modifiers with the modifiers and the classification words when determining that the morphemes split from the field information include the modifiers and do not include the classification words, which is equivalent to adding the classification words to the modifiers when only the modifiers do not include the classification words in the field information.
And searching for a modifier of the replacement rule aiming at each morpheme of the field information, and judging whether the morpheme has a classification word of the morpheme in the field information if one morpheme is matched with the modifier in the replacement rule in a consistent manner.
If the field information does not have the classified word of the modifier, remarks are added, and the morpheme in the model is replaced by two morphemes, namely the modifier is replaced by the modifier and the classified word corresponding to the modifier.
For example, when the morpheme "performance" is included in the model and the "amount" is not included, the "performance" is converted to the "amount of performance" according to the replacement rule.
It can be understood that shorthand may be used in designing the model, only the modifiers are used, and the categorical words are omitted, and for this irregular writing, the preset categorical words are added to the modifiers by replacing the rules.
If the field information has the classified word of the modifier, the field information is normalized, and the verification result is successful.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
In the verification processing method provided by this embodiment, if the field definition does not match the standard definition, the rule management base is used for verification, and if a morpheme matches a modifier and there is no classifier, the morpheme is replaced with the morpheme and the corresponding classifier, so that irregular abbreviation is modified to be standard.
On the basis of the foregoing embodiment, in the verification processing method according to another embodiment of the present invention, if the field definition does not match the standard definition, the data preprocessing is performed on each field information, and the step of obtaining a plurality of morphemes specifically includes:
analyzing each field information to generate a corresponding json character string;
and performing word segmentation processing on each json character string to obtain a plurality of morphemes.
Optionally, a parser is used to parse each field information.
Optionally, regarding a json character string as a sequence, and calling a word segmentation component to perform word segmentation on the sequence to obtain morphemes.
Optionally, the word segmentation component is a function, and is configured to segment a sequence into individual words, i.e., morphemes, and check the obtained morphemes by using a rule management base.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
In the verification processing method provided by this embodiment, the field information is analyzed to generate corresponding json character strings, and word segmentation processing is performed on each json character string to obtain a morpheme, so as to subsequently implement morpheme-level verification.
On the basis of the foregoing embodiment, a method for verification processing according to another embodiment of the present invention is a method for verification processing, where the morpheme includes a chinese morpheme and/or an english morpheme, and accordingly, after the step of determining whether or not there is a classified word of the morpheme if the morpheme is matched with a modifier, the method includes:
if the morpheme is not matched with the modifier, a pre-stored service dictionary is obtained, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises a Chinese expression and an English expression;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
And if the morpheme is not matched with the modifier, the verification of the rule management library is failed, and the service dictionary is continuously adopted for verification.
Optionally, each business expression includes two morphemes, a chinese expression and an english expression. Such as (performance, perf), (broker, agent), etc.
For example, if the morpheme includes a morpheme of performance, and matches with a chinese expression in a business expression (performance, perf), and does not include an english expression perf, the remarks are added: adding English wording perf. Similarly, if perf is included in the model and performance is not included, then notes are added: performance is added.
It can be understood that when designing a model, shorthand writing may be used, and only chinese does not have english, or only english does not have chinese, and for this irregular writing method, completion is performed by business terms, so that subsequent data, whether chinese or english, can be correctly identified and filled.
Optionally, if the morpheme only includes a chinese morpheme, matching the chinese morpheme, if the morpheme only includes an english morpheme, matching the english morpheme, if the morpheme includes a chinese morpheme and an english morpheme, the matching order is not limited, and matching the chinese morpheme or matching the english morpheme may be performed first.
If the Chinese morpheme is matched with the Chinese language but the English morpheme is not matched with the English language, remarks are added to explain the situation, and a verification result is output.
Similarly, if the English morpheme matches the English word, but the Chinese morpheme does not match the Chinese word, the remark description is added.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
In the verification processing method provided by this embodiment, if the morpheme is not matched with the modifier, the verification is continued by using the chinese term and the english term, and the chinese morpheme and the english morpheme can be completed.
On the basis of the foregoing embodiment, a method for verification processing according to another embodiment of the present invention is provided, where the morphemes include chinese morphemes and/or english morphemes, and after the step of replacing the morphemes with the morphemes and corresponding classification words, the method includes:
the method comprises the steps of obtaining a pre-stored service dictionary, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises Chinese expressions and English expressions;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
And after the simplified morphemes are replaced by using the rule management library for verification, the service dictionary is continuously adopted for verification.
Optionally, each business expression includes two morphemes, a chinese expression and an english expression. Such as (performance, perf), (broker, agent), etc.
For example, the original morpheme includes a morpheme of performance, and after checking by using the rule management base, the morpheme is replaced by a performance amount, in the embodiment of the present invention, the performance is matched with a chinese expression in a business expression (performance, perf), and an english expression perf is not included, and remarks are added: adding English expression perf, matching the money amount with Chinese expression in the business expression (money amount, amount) and not including English expression, adding remarks: the english term amount is added.
And if no matched morpheme exists after the service expression is traversed, adding a corresponding remark and outputting a verification result.
It can be understood that when designing a model, shorthand writing may be used, and only chinese does not have english, or only english does not have chinese, and for this irregular writing method, completion is performed by business terms, so that subsequent data, whether chinese or english, can be correctly identified and filled.
Alternatively, if the Chinese morpheme matches the Chinese phrase but the English morpheme does not match the English phrase, a remark is added to explain the situation, and the verification result is output. Similarly, if an English morpheme matches an English word, but a Chinese morpheme does not match a Chinese word, a remark is added.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
In the verification processing method provided by this embodiment, after the rule management base is used for verification, the chinese vocabulary and the english vocabulary are continuously used for verification, and the chinese morpheme and the english morpheme can be completed.
On the basis of the foregoing embodiment, in a method for verification processing provided by another embodiment of the present invention, after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method includes:
training a field definition if the field definition does not match a standard definition;
and if the preset condition is met, taking the field definition as a standard definition.
Alternatively, if the field definition does not match the standard definition, there may be two cases: one is the nonstandard definition of the field and the other is the canonical expression of the unified standard, but the data dictionary does not write the field definition into the standard definition.
Optionally, training the field definitions refers to specifically re-matching the field definitions to determine whether the field definitions can be defined as a standard.
Optionally, according to the latest data, matching is performed with the field definition, if there is a standardized standard expression matching with the field definition, the field definition is used as a standard definition, and the field type of the field definition is used as a standard type, so as to obtain a new standard expression.
It can be understood that the newly added standard expressions are classified and placed into the data dictionary, and finally a closed-loop verification management is formed to gradually expand the verification range and reduce the misjudgment.
Other steps of this embodiment are similar to those of the previous embodiment, and are not described again in this embodiment.
In the verification processing method provided in this embodiment, if the field definition does not match the standard definition, the field definition is trained, and after it is determined that the field definition can be used as the standard definition, the field definition is stored in the data dictionary to extend the verification range.
In order to more fully understand the technical content of the present invention, on the basis of the above embodiments, the method of the verification processing provided in the present embodiment is explained in detail.
The method of the embodiment of the invention mainly aims at solving the problem that the data standard of the existing data warehouse is difficult to implement on the ground, and provides the following solution:
learning data are obtained from historical data and a data model by adopting a word segmentation technology and experience matching mode, experience learning is carried out, common standard words and service terms are extracted and corrected, and the common standard words and service terms are respectively stored into corresponding dictionary managers according to different types of standard terms or specifications.
In the subsequent modeling development, the generalized standard terms (specifications) are used as experience and standard to detect and correct the subsequent model, so that the problem of model data naming standardization is solved, and the closed-loop management mode of model verification, data dictionary accumulation and model verification is used for continuously adjusting and optimizing, so that the continuous improvement is realized, the aim of automatically maintaining data standard management is fulfilled, the data quality of a data warehouse is improved, and the operation cost is reduced.
The embodiment of the invention can be used for unifying data naming, data definition and data type standardization constraint and is used for solving the problem that the terms are disordered or how to name the terms in the modeling process.
The standardized object in the embodiment of the invention refers to data used in the engineering project range, and can be understood as a target object needing data standardization.
Fig. 2 is a schematic overall structure diagram of a verification processing apparatus according to another embodiment of the present invention.
As shown in fig. 2, the overall structure of the device for verification processing is divided into three parts:
the data storage layer is used for storing a data standard specification dictionary and comprises a rule management library, a data dictionary and a service dictionary; standard check-up layer: the system is used for performing standard object verification and generating a standard expression dictionary and consists of a vocabulary training system, a word segmentation component and a standard verifier; a data interface layer: the system is used for receiving and analyzing the standard object model document and providing a verification report for the outside and comprises a model analyzer and a verification report generator respectively.
The standardized verification is mainly performed from the following aspects:
1. morphemes: the smallest unit word with a certain meaning can be generally understood as a word obtained by dividing a standardized object by using a word dividing component, and when the standardization work is performed, the first step is to decompose the current expression into the smallest unit meaning and then perform standard word confirmation, such as: day, month, income, city, etc., belonging to the category of service dictionary.
2. The standard word is: the standard word is the smallest unit word in the dictionary meaning and is the basic component element of the business expression. The standard words are composed of Chinese names and English abbreviation labels, and each standard word has one English abbreviation matched with the standard word, such as (achievement, perf), (broker, agent) and the like, and belongs to the category of service dictionaries.
3. Classifying words: the classification word identifies a standard word of an entity or entity attribute type from which a standard word of an internal data value type can be inferred. Such as amount, quantity, PV, UV, etc., belong to the category of the service dictionary.
4. Standard domain: the data is divided into a code field, a number field, a group field, etc., and standard data types (character strings, data, dates, etc.) and lengths are defined to clarify data ranges. Such as (amount, double), (amount, num, int), etc., belong to the category of the service dictionary.
5. Standard wording: all standard item names generated by using standard words according to naming rules (modifiers + classified words) comprise entity names, attribute names of entities, table names, column names, domain names and the like, such as (pt, time partition, string), (house _ ID, house source ID, int) and the like, and belong to the category of a data dictionary.
6. Rule conversion: the method is characterized in that some merging and converting operations of standard words, classified words and standard domains are combined, some vocabulary and phrases with high use frequency are spliced in a mode of modifiers and classified words, when name shorthand appears in a standard target object, full name conversion can be carried out according to a conversion rule, and information such as English identification corresponding to the conversion rule is attached, for example: (Perf _ account, double), when the word of 'achievement' appears in the standard object, the rule manager will convert 'achievement' into 'achievement amount' according to the conversion rule, belonging to the category of the rule management base.
The embodiment of the invention relates to two parts:
a first part: and (5) an initialization phase. In order to ensure the correct judgment and verification accuracy of data standardization, data standard initialization is required, which mainly comprises the work of collecting a data source, determining a data dictionary, a rule management library, a service dictionary and the like. The part of work is composed of automatic software implementation and manual intervention.
A second part: and (5) a data standardization checking stage. After the data standard initialization work is carried out, the data standard is checked, and a detection report is generated for standard modification before the model is on line; and after the model is online, performing vocabulary analysis on the newly added model, adding a new standard expression, and forming closed-loop management of standard object verification- > adding the standard expression- > standard object verification.
Fig. 3 is a flowchart illustrating a verification processing method according to another embodiment of the present invention.
As shown in fig. 3, the embodiment of the present invention specifically includes a plurality of steps: model parsing, word segmentation processing, model checking, output checking report, model modification, model submission and parsing, model training and model online.
Can be understood to comprise 3 steps: initialization, verification and the following steps.
Fig. 4 is a flowchart of an initialization phase operation according to another embodiment of the present invention.
As shown in fig. 4, step 1: the standard object range is selected and generally collected from industry professional word dictionary, existing data warehouse data, wiki, various professional books and data.
After the material is collected, it is treated in two ways: directly performing word segmentation, searching and scoring on the text data, performing word frequency statistical sequencing, and then filtering from high to low according to the word frequency sequence; for the existing database data information, the recursive combination is split and sequenced according to the English name and the Chinese name of the field, and then the existing naming specification is sequentially filtered and modified according to the occurrence frequency of the pairing. And finally, putting different words into different data dictionaries, rule management libraries and service dictionaries according to manual filtering.
Wherein, in the process of empirically matching existing model metadata information, as shown in figure 3,
acquiring field attribute information of all models, and splitting words into morphemes; then recursively splicing three parts of each English code, each Chinese name and the field type of the current field into a character string, performing word frequency statistics on all the spliced character strings, and counting data with more occurrence times of each group of corresponding relations as an experience pairing standard; and finally, after manual inspection, the data are recorded into three storage information bases (a data dictionary, a rule management base and a service dictionary).
Step 2: after the initialization step is completed, a standard verification program can be operated online.
FIG. 5 is a diagram of a portion of an example of a verify operation according to yet another embodiment of the present invention.
FIG. 6 is a diagram of a portion of an example of a verify operation according to yet another embodiment of the present invention.
As shown in fig. 5 and 6, when a new design model is subjected to data standard verification, its model information, including table name, field type, field description, etc., is first uploaded.
Then the uploaded model is analyzed by a model analyzer, and each row of data is analyzed into a json string structure of a structure of { origin (field name), name (field description), type (field type) };
after the model is completely analyzed, json information and native information are sent to a standard checker for checking;
after receiving the json string, the [ word segmentation components ] are called line by line to carry out word morpheme splitting, and after splitting, checking work is carried out on each row of data.
Fig. 7 is a flowchart illustrating a verification operation according to another embodiment of the present invention.
As shown in fig. 7, the whole verification process is divided into three determinations:
1. firstly, calling data in a data dictionary to perform field code/name matching on native information, inquiring whether field names or field descriptions can be matched, if one party is successfully matched, acquiring information to perform standard verification on other data to be verified currently, and if the party is not successfully matched, using the fields in the data dictionary to perform remark identification; and if the matching is not successful, entering the next judgment.
2. After the data dictionary passes, judging whether the morphemes in the json string have elements which can correspond to each other or not in the rule management base for the second time, taking out the rule to judge and fill the json data to be verified, and entering the next judgment if the elements which correspond to each other are not found.
3. After the judgment of the rule management base, the method enters a business dictionary to search and match morphemes in the json string, if matched words exist, the matched words are obtained to carry out standard verification on elements in the json data, standard filling remarks are carried out on places which are not satisfied, if not, the next round of verification is finished, and the verification result of the data in the current row is temporarily stored. And when all the fields of the checked model are checked, acquiring all temporary storage checking data results, sending the temporary storage checking data results to a checking report generator to generate a remark checking report and a modification suggestion, and returning the remark checking report and the modification suggestion to the submitter. And finishing the model verification of the current round.
And step 3: after the online model is checked, the model is named according to a check report, after the structure is modified, the model can be submitted again, the model is clicked, the current model can be subjected to vocabulary training and word extraction, maintenance personnel of a final storage library can classify and place newly added and extracted standard vocabulary words into three different dictionary libraries, and finally closed-loop check management is formed to gradually complete the coverage range.
The patent application has several innovation points as follows:
1. the data standard checking process and method with higher automation degree are realized through less manual operation and software programs. At the beginning of starting, multiple data sources are adopted, and multiple initialization modes are used for identifying the range of a standard object; and after the system is on-line, a closed-loop working mode of model verification, standard expression vocabulary addition and model verification is formed, and the standard dictionary base is continuously expanded, so that the matching coverage and the verification degree of the verification method and the verification system are continuously improved.
2. The data standard verification mode of the invention has comprehensiveness, and comprehensively covers data naming, data definition and data type verification of entity names, entity attribute names, table names, column names, index names and the like in a data warehouse. The standardized objects are verified from multiple aspects of morphemes, standard words, standard phrases, classified words, standard domains and conversion rules by using standard conditions of multiple levels of granularity so as to meet the maximum matching degree and the verification accuracy rate.
3. The data standard verification mode of the invention has stronger flexibility, and is abstracted into 3 categories according to different situations of data standardization: a data dictionary, a rule management library and a service dictionary library. The data dictionary is used for verifying standard domain verification of fixed field names and strictly defining the names, types, descriptions and data ranges of the data fields; the rule management library is used for converting standard expression vocabularies, successfully matches the standard expression vocabularies, and directly converts the irregular vocabularies into the standard expressions; the service dictionary is used for providing standardized naming reference opinions for the morphemes subjected to word segmentation and splitting. The user can construct different dictionaries according to different standard specifications, and the method has great independence and flexibility.
The verification processing method provided by the embodiment provides a scientific method, flow and implementation scheme for the standardized implementation of the construction of the data warehouse of the enterprise; the construction quality of the data warehouse is improved, the correctness of the data is ensured, and the consistency of the enterprise model is maintained; the development productivity and the management efficiency of the data warehouse are improved, and the resource and labor waste caused by repeated invalid labor is reduced; and the operation cost of enterprise data warehouse maintenance is reduced.
Fig. 8 is a schematic structural diagram of an apparatus for verification processing according to yet another embodiment of the present invention.
Referring to fig. 8, on the basis of the foregoing embodiment, the apparatus for verification processing provided in this embodiment includes an obtaining module 81, a verifying module 82, and a modifying module 83, where:
the obtaining module 81 is configured to obtain models of a data warehouse to be verified, where each model includes a plurality of field information, and the field information includes a field definition and a field type; the checking module 82 is configured to check the field information according to a pre-stored data dictionary, where the data dictionary includes a plurality of standard expressions, and each standard expression includes a standard definition and a standard type; the modification module 83 is configured to modify the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type.
Alternatively, the construction of a data warehouse may be divided into two steps: first, a model of the data warehouse is designed, and second, data is written to a corresponding model (data table).
After the model design is completed, the device provided by the embodiment of the invention is applied to verify the model.
The checking device is a computer with an iBATIS architecture. The iBATIS is an open source code project based on Java, and can automatically realize object relation mapping.
Optionally, the obtaining module 81 uploads at least one designed model, each model comprising a plurality of lines of data, to the means for checking.
Alternatively, a model may be understood as a data table carrying headers, the data table comprising a plurality of rows of data, each row of data comprising corresponding field information.
Optionally, the field information includes a field definition and a field type, the field definition is a description of the meaning of the field, and may include a field name and a field description. The field type is a description of the type of the field, for example, the field is double or int, where double is a double-precision floating-point number, that is, the field may be a number with a decimal point, and int represents integer, that is, the field is an integer.
The verification module 82 verifies the field information according to a pre-stored data dictionary.
Optionally, a data dictionary is created in advance, and the data dictionary comprises a plurality of standard expressions, and each standard expression is agreed and can be used as a standard expression of a unified standard.
Alternatively, the standard expression is collected from an industry professional expression dictionary, historical data warehouse data, wiki (Wikipedia), various professional books, and data.
Optionally, the standard wording includes a standard definition that is a standard description of a field and a standard type that indicates a type that the field may be used.
For example, the standard is defined as amount, the standard type double of the amount created in advance, and the amount does not use int as the standard type after the standard type is determined to be double.
Optionally, for the field definition of the model, the standard expression of the data dictionary is queried for whether there is a standard definition matching the field definition of the model.
And if the field definition is successfully matched with the standard definition of the standard expression, inquiring the standard type corresponding to the standard definition matched with the field definition of the model in the standard expression aiming at the field type of the model.
If the field definition of the model is consistent with the standard definition and the field type is not consistent with the standard type, the modification module 83 notes the model, and the contents of the notes are as follows: and if the field type is not consistent with the standard type, outputting a verification result, wherein the verification result comprises the remark.
In the verification process of the embodiment of the invention, remarks are added to provide modification suggestions, so that modification is executed subsequently according to the verification result, and the field type is modified to be consistent with the standard type.
If the field definition of the model is consistent with the standard definition and the field type is consistent with the standard type, the model is proved to be in accordance with the standard, and the verification result of the field information is directly output to be successful.
If the field definition does not match the standard definition of the standard expression successfully, the verification result is output as failure.
It can be understood that, by creating a data dictionary in advance, if the apparatus of the embodiment of the present invention is applied to each data warehouse during modeling, and verification is performed according to the data dictionary to obtain a consistent and standard data table, then the consistent and standard data table can be directly filled in the standard data table during data filling.
The apparatus for checking provided in this embodiment may be configured to execute the method in the foregoing method embodiment, and this implementation is not described again.
In the device for checking processing provided by this embodiment, the checking module checks the model of the data warehouse according to the standard expression, and when the field definition matches the standard definition and the field type does not match the standard type, the modification module modifies the field type to be consistent with the standard type in a targeted manner, so as to obtain a standard consistent model.
Fig. 9 is a schematic structural diagram of an electronic device according to yet another embodiment of the present invention.
Referring to fig. 9, an electronic device provided by the embodiment of the present invention includes a memory (memory)91, a processor (processor)92, a bus 93, and a computer program stored on the memory 91 and operable on the processor. The memory 91 and the processor 92 complete communication with each other through the bus 93.
The processor 92 is used to call the program instructions in the memory 91 to implement the method of fig. 1 when executing the program.
In another embodiment, the processor, when executing the program, implements the method of:
the field definition comprises a field name and a field description, the standard definition comprises a standard name and a standard description, and correspondingly, the step of verifying the field information according to the pre-stored data dictionary specifically comprises the following steps:
if the field name is matched with the standard name, checking whether the field description is consistent with the standard description or not, and checking whether the field type is consistent with the standard type or not;
or;
and if the field description is matched with the standard description, checking whether the field name is consistent with the standard name or not, and checking whether the field type is consistent with the standard type or not.
In another embodiment, the processor, when executing the program, implements the method of:
after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes;
acquiring a pre-stored rule management base, wherein the rule management base comprises a plurality of replacement rules, and each replacement rule comprises a modifier and a classification word;
if the morpheme is matched with the modifier, judging whether the classified word of the morpheme exists or not;
and if not, replacing the morpheme with the morpheme and the corresponding classified word.
In another embodiment, the processor, when executing the program, implements the method of:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes specifically:
analyzing each field information to generate a corresponding json character string;
and performing word segmentation processing on each json character string to obtain a plurality of morphemes.
In another embodiment, the processor, when executing the program, implements the method of:
the morphemes include Chinese morphemes and/or English morphemes, and correspondingly, after the step of judging whether the classified words of the morphemes exist or not if the morphemes are matched with the modified words, the method comprises the following steps:
if the morpheme is not matched with the modifier, a pre-stored service dictionary is obtained, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises a Chinese expression and an English expression;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
In another embodiment, the processor, when executing the program, implements the method of:
the morphemes include chinese morphemes and/or english morphemes, and accordingly, after the step of replacing the morphemes with the morphemes and the corresponding classified words, the method includes:
the method comprises the steps of obtaining a pre-stored service dictionary, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises Chinese expressions and English expressions;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
In another embodiment, the processor, when executing the program, implements the method of:
after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
training a field definition if the field definition does not match a standard definition;
and if the preset condition is met, taking the field definition as a standard definition.
The electronic device provided in this embodiment may be configured to execute the program corresponding to the method in the foregoing method embodiment, and this implementation is not described again.
In the electronic device provided by this embodiment, when the processor executes the program, the processor verifies the model of the data warehouse according to the standard expression, and when the field definition matches the standard definition and the field type does not match the standard type, the field type is modified to be consistent with the standard type in a targeted manner, so as to obtain a standard consistent model.
A further embodiment of the invention provides a storage medium having a computer program stored thereon, which when executed by a processor performs the steps of fig. 1.
In another embodiment, the program when executed by a processor implements a method comprising:
the field definition comprises a field name and a field description, the standard definition comprises a standard name and a standard description, and correspondingly, the step of verifying the field information according to the pre-stored data dictionary specifically comprises the following steps:
if the field name is matched with the standard name, checking whether the field description is consistent with the standard description or not, and checking whether the field type is consistent with the standard type or not;
or;
and if the field description is matched with the standard description, checking whether the field name is consistent with the standard name or not, and checking whether the field type is consistent with the standard type or not.
In another embodiment, the program when executed by a processor implements a method comprising:
after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes;
acquiring a pre-stored rule management base, wherein the rule management base comprises a plurality of replacement rules, and each replacement rule comprises a modifier and a classification word;
if the morpheme is matched with the modifier, judging whether the classified word of the morpheme exists or not;
and if not, replacing the morpheme with the morpheme and the corresponding classified word.
In another embodiment, the program when executed by a processor implements a method comprising:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes specifically:
analyzing each field information to generate a corresponding json character string;
and performing word segmentation processing on each json character string to obtain a plurality of morphemes.
In another embodiment, the program when executed by a processor implements a method comprising:
the morphemes include Chinese morphemes and/or English morphemes, and correspondingly, after the step of judging whether the classified words of the morphemes exist or not if the morphemes are matched with the modified words, the method comprises the following steps:
if the morpheme is not matched with the modifier, a pre-stored service dictionary is obtained, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises a Chinese expression and an English expression;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
In another embodiment, the program when executed by a processor implements a method comprising:
the morphemes include chinese morphemes and/or english morphemes, and accordingly, after the step of replacing the morphemes with the morphemes and the corresponding classified words, the method includes:
the method comprises the steps of obtaining a pre-stored service dictionary, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises Chinese expressions and English expressions;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
In another embodiment, the program when executed by a processor implements a method comprising:
after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
training a field definition if the field definition does not match a standard definition;
and if the preset condition is met, taking the field definition as a standard definition.
In the storage medium provided in this embodiment, when the program is executed by the processor, the method in the foregoing method embodiment is implemented, and details of this implementation are not described again.
According to the storage medium provided by the embodiment, the model of the data warehouse is verified according to the standard expression, and when the field definition is matched with the standard definition and the field type is not matched with the standard type, the field type is pertinently modified to be consistent with the standard type, so that a standard consistent model is obtained.
Yet another embodiment of the present invention discloses a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the methods provided by the above-mentioned method embodiments, for example, comprising:
obtaining models of a data warehouse to be verified, wherein each model comprises a plurality of field information, and the field information comprises field definition and field type;
checking the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type;
and if the field definition is matched with the standard definition and the field type is not matched with the standard type, modifying the field type to be consistent with the standard type.
Those skilled in the art will appreciate that although some embodiments described herein include some features included in other embodiments instead of others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments.
Those skilled in the art will appreciate that the steps of the embodiments may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (9)
1. A method of verification processing, the method comprising:
obtaining models of a data warehouse to be verified, wherein each model comprises a plurality of field information, and the field information comprises field definition and field type;
checking the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type;
if the field definition is matched with the standard definition and the field type is not matched with the standard type, modifying the field type to be consistent with the standard type;
after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes;
acquiring a pre-stored rule management base, wherein the rule management base comprises a plurality of replacement rules, and each replacement rule comprises a modifier and a classification word;
if the morpheme is matched with the modifier, judging whether the classified word of the morpheme exists or not;
and if not, replacing the morpheme with the morpheme and the corresponding classified word.
2. The method of claim 1, wherein: the field definition comprises a field name and a field description, the standard definition comprises a standard name and a standard description, and correspondingly, the step of verifying the field information according to the pre-stored data dictionary specifically comprises the following steps:
if the field name is matched with the standard name, checking whether the field description is consistent with the standard description or not, and checking whether the field type is consistent with the standard type or not;
or;
and if the field description is matched with the standard description, checking whether the field name is consistent with the standard name or not, and checking whether the field type is consistent with the standard type or not.
3. The method of claim 1, wherein: if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes specifically:
analyzing each field information to generate a corresponding json character string;
and performing word segmentation processing on each json character string to obtain a plurality of morphemes.
4. The method of claim 1, wherein the morphemes comprise Chinese morphemes and/or English morphemes, and wherein the method comprises, after the step of determining whether a classified word of the morphemes exists if the morphemes match modifiers:
if the morpheme is not matched with the modifier, a pre-stored service dictionary is obtained, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises a Chinese expression and an English expression;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
5. The method of claim 3, wherein: the morphemes include chinese morphemes and/or english morphemes, and accordingly, after the step of replacing the morphemes with the morphemes and the corresponding classified words, the method includes:
the method comprises the steps of obtaining a pre-stored service dictionary, wherein the service dictionary comprises a plurality of service expressions, and each service expression comprises Chinese expressions and English expressions;
if the Chinese morpheme is matched with the Chinese expression and the corresponding English expression does not exist in the morpheme, remarking the Chinese morpheme for increasing the English expression of the Chinese morpheme;
and if the English morpheme is matched with the English expression and the morpheme does not have the Chinese expression corresponding to the English expression, remarking the English morpheme so as to increase the Chinese expression of the English morpheme.
6. The method of claim 1, wherein: after the step of modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, the method comprises:
training a field definition if the field definition does not match a standard definition;
and if the preset condition is met, taking the field definition as a standard definition.
7. An apparatus for verification processing, the apparatus comprising:
the system comprises an acquisition module, a verification module and a verification module, wherein the acquisition module is used for acquiring models of a data warehouse to be verified, each model comprises a plurality of field information, and the field information comprises field definitions and field types;
the verification module is used for verifying the field information according to a pre-stored data dictionary, wherein the data dictionary comprises a plurality of standard expressions, and each standard expression comprises a standard definition and a standard type;
the modification module is used for modifying the field type to be consistent with the standard type if the field definition is matched with the standard definition and the field type is not matched with the standard type;
the modification module, after modifying the field type to be consistent with the standard type if the field definition matches the standard definition and the field type does not match the standard type, is further configured to:
if the field definition is not matched with the standard definition, performing data preprocessing on each field information to obtain a plurality of morphemes;
acquiring a pre-stored rule management base, wherein the rule management base comprises a plurality of replacement rules, and each replacement rule comprises a modifier and a classification word;
if the morpheme is matched with the modifier, judging whether the classified word of the morpheme exists or not;
and if not, replacing the morpheme with the morpheme and the corresponding classified word.
8. An electronic device comprising a memory, a processor, a bus, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of any of claims 1-6.
9. A storage medium having a computer program stored thereon, characterized in that: the program when executed by a processor implementing the steps of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810045917.1A CN108256074B (en) | 2018-01-17 | 2018-01-17 | Verification processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810045917.1A CN108256074B (en) | 2018-01-17 | 2018-01-17 | Verification processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108256074A CN108256074A (en) | 2018-07-06 |
| CN108256074B true CN108256074B (en) | 2020-06-23 |
Family
ID=62741174
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810045917.1A Active CN108256074B (en) | 2018-01-17 | 2018-01-17 | Verification processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108256074B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109165209B (en) * | 2018-08-14 | 2021-06-08 | 上海达梦数据库有限公司 | Data verification method, device, equipment and medium for object types in database |
| CN109408510A (en) * | 2018-10-19 | 2019-03-01 | 中国银行股份有限公司 | A kind of method for normalizing and device of data model |
| CN109388685B (en) * | 2018-10-23 | 2022-01-07 | 泰华智慧产业集团股份有限公司 | Method and device for warehousing spatial data used by planning industry |
| CN109766436A (en) * | 2018-12-04 | 2019-05-17 | 北京明略软件系统有限公司 | A kind of matched method and apparatus of data element of the field and knowledge base of tables of data |
| CN109597763B (en) * | 2018-12-04 | 2022-02-25 | 北京广利核系统工程有限公司 | Consistency verification method and device for normalizing multiple items of data |
| CN109656912B (en) * | 2018-12-13 | 2020-08-07 | 成都四方伟业软件股份有限公司 | Data model control method and device and server |
| CN111488327B (en) * | 2019-01-29 | 2023-08-22 | 卓望数码技术（深圳）有限公司 | Data standard management method and system |
| CN110673888B (en) * | 2019-08-27 | 2023-04-07 | 贝壳技术有限公司 | Verification method and device for configuration file |
| CN110795464B (en) * | 2019-08-28 | 2022-03-04 | 腾讯科技（深圳）有限公司 | Method, device, terminal and storage medium for checking field of object marker data |
| CN110532561B (en) * | 2019-08-30 | 2022-12-09 | 北京明略软件系统有限公司 | Data detection method and device, storage medium and electronic device |
| CN110795482B (en) * | 2019-10-16 | 2022-11-22 | 浙江大华技术股份有限公司 | Data benchmarking method, device and storage device |
| CN110909003B (en) * | 2019-11-25 | 2022-06-10 | 车智互联（北京）科技有限公司 | Method for creating data table and computing equipment |
| CN112164481B (en) * | 2020-08-17 | 2023-09-29 | 北京广利核系统工程有限公司 | Intelligent verification method and system for nuclear power safety control display equipment database |
| CN112035451A (en) * | 2020-08-25 | 2020-12-04 | 上海灵长软件科技有限公司 | Data verification optimization processing method and device, electronic equipment and storage medium |
| CN114091426A (en) * | 2020-10-19 | 2022-02-25 | 北京沃东天骏信息技术有限公司 | Method and device for processing field data in data warehouse |
| CN112733199A (en) * | 2020-12-28 | 2021-04-30 | 北京极豪科技有限公司 | Data processing method and device, electronic equipment and readable storage medium |
| CN113642327A (en) * | 2021-10-14 | 2021-11-12 | 中国光大银行股份有限公司 | Method and device for constructing standard knowledge base |
| CN114416832B (en) * | 2022-01-26 | 2022-11-15 | 重庆允丰科技有限公司 | Method for configuring formula field and report and computer storage medium |
| CN115186650B (en) * | 2022-09-07 | 2022-12-09 | 中国中金财富证券有限公司 | Data detection method and related device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102346785A (en) * | 2011-11-15 | 2012-02-08 | 北京创腾科技有限公司 | Method and device for directly self-defining field of database |
| CN104598598A (en) * | 2015-01-23 | 2015-05-06 | 浙江协同数据系统有限公司 | Method for evaluating relational data standard |
| US9135309B2 (en) * | 2006-02-01 | 2015-09-15 | Oracle International Corporation | System and method for building decision trees in a database |
| CN107193681A (en) * | 2016-03-15 | 2017-09-22 | 阿里巴巴集团控股有限公司 | Data verification method and device |
-
2018
- 2018-01-17 CN CN201810045917.1A patent/CN108256074B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9135309B2 (en) * | 2006-02-01 | 2015-09-15 | Oracle International Corporation | System and method for building decision trees in a database |
| CN102346785A (en) * | 2011-11-15 | 2012-02-08 | 北京创腾科技有限公司 | Method and device for directly self-defining field of database |
| CN104598598A (en) * | 2015-01-23 | 2015-05-06 | 浙江协同数据系统有限公司 | Method for evaluating relational data standard |
| CN107193681A (en) * | 2016-03-15 | 2017-09-22 | 阿里巴巴集团控股有限公司 | Data verification method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108256074A (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108256074B (en) | Verification processing method and device, electronic equipment and storage medium | |
| US11113477B2 (en) | Visualizing comment sentiment | |
| CN110580308B (en) | Information auditing method and device, electronic equipment and storage medium | |
| CN109492106B (en) | Automatic classification method for defect reasons by combining text codes | |
| AU2019204444B2 (en) | System and method for enrichment of ocr-extracted data | |
| CN112231431B (en) | Abnormal address identification method and device and computer readable storage medium | |
| CN112036842A (en) | Intelligent matching platform for scientific and technological services | |
| CN112632982A (en) | Dialogue text emotion analysis method capable of being used for supplier evaluation | |
| CN115062117A (en) | Method for automatically generating and classifying documents based on natural language processing technology | |
| CN114548072A (en) | Automatic content analysis and information evaluation method and system for contract files | |
| CN107943785B (en) | PDF document processing method and device based on big data | |
| CN115482075A (en) | Financial data anomaly analysis method and device, electronic equipment and storage medium | |
| US10120652B2 (en) | System and method for representing software development requirements into standard diagrams | |
| CN110928535A (en) | Derivative variable deployment method, device, equipment and readable storage medium | |
| CN112037027B (en) | Automatic method and system for production change compliance check of banking system | |
| CN117332286A (en) | System, method and device for data mapping verification | |
| CN118093650A (en) | Data supervision method and system based on rule engine | |
| CN117875706A (en) | Digital management method for grading process based on AI | |
| CN115454985A (en) | Data quality checking method | |
| DK202070594A1 (en) | A method for address matching and related electronic device | |
| CN118037294A (en) | Financial certificate generation method and device based on business form data | |
| CN117827923A (en) | Query demand processing method and device, computer equipment and storage medium | |
| CN112667666A (en) | SQL operation time prediction method and system based on N-gram | |
| CN117635138A (en) | Abnormal business transaction account identification method and system | |
| CN114493175A (en) | Content recommendation method for auditing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100085 Floor 102-1, Building No. 35, West Second Banner Road, Haidian District, Beijing Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: 100085 Floor 102-1, Building No. 35, West Second Banner Road, Haidian District, Beijing Applicant before: LIANJIA(BEIJING) TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |