CN104268247A - Master data imputation method based on fuzzy analytic hierarchy process - Google Patents

Abstract

The invention belongs to the technical field of metallurgical industry informationization and discloses a master data imputation method based on a fuzzy analytic hierarchy process. Aiming at the industrial characteristic that enterprise master data have features of diversity and complexity of metadata, more data source hierarchies, difficulty in data collection, high data change frequency, failure in timeliness of system updating and the like, the method includes that on the basis of the fuzzy analytic hierarchy process, adopting methods of data hierarchy processing, rule weight allocation and multi-keyword fuzzy matching to realize modularization by dynamic assembly of limited instruction sets, and combining specification rechecking and dynamic analysis manners to realize a data imputation technique with advantages of once collection, few maintenance and dynamic logic updating in terms of data collection. Complaints about lots of problems in data collection can be avoided, simplicity and quickness in data maintenance are achieved, and efficiency can be improved continuously as well as serviceability of quickness in response to user needs.

Description

A kind of master data collecting method based on Fuzzy Level Analytic Approach

Technical field

The invention belongs to metallurgy industry informationization technology field, relate to a kind of master data based on Fuzzy Level Analytic Approach and collect technology.

Background technology

The invention discloses a kind of master data based on Fuzzy Level Analytic Approach and collect technology.Corporate boss's data have the various complexity of metadata, Data Source level more collection difficulty, data frequent variations rate high, the system update characteristic such as not in time, traditional forms of enterprises's master data is collected basic by artificially collecting, very high to data quality requirements, error rate is comparatively large, and repetitive operation is more.Although the data collection software that industry has part shaping, as SAP mdm, Oracle EBS etc., although data consistency, integrality can be accomplished, but the data of collecting are only limitted to built-in system, and internal logic is not strong, once changes the data volume that need adjust large, and consuming time longer, face the problem that follow-up data maintenance work is heavy.The present invention looks for another way, for industry characteristic, on the basis of Fuzzy Hierarchy Method, adopt the process of data hierarchy level, rule weight allocation, the method of multiple key fuzzy matching, then by realizing modularization to the changeable assembling of limited instruction set, finally verify again in conjunction with specification, dynamic analysis mode realizes the once collection of Data Collection aspect, less maintenance, the purpose data classifying technology that dynamic logic upgrades, allow user need not complain problems in Data Collection again, and data post is safeguarded simple, fast, efficiency and the service ability responding user's request fast can be improved constantly.

Summary of the invention

Emphasis application Fuzzy Level Analytic Approach of the present invention is theoretical, by collecting the basic comprising unit arranging out master data, i.e. metadata, attribute definition and layout are carried out to it, index is set up to all data, extract the key word of Representative properties value, semantic analysis is carried out to metadata source document, and then word segmentation processing, in conjunction with index, fuzzy matching.From traditional artificially collecting merely, manually check and correction change that basic data artificially collects, profound Dynamic Matching into, reference surface is many, avoids repeated work, serious forgiveness low, with a high credibility, and data post is safeguarded simple, quick.

For achieving the above object, a kind of master data based on Fuzzy Level Analytic Approach collects technology, adopts the strategy of first distinguishing hierarchy, keywording formation rule, then fuzzy matching again.This technology comprises following key components:

(1) metadata definition: business data carries out hierarchical classification, level definition needs rigorous, easily extensible, period introduces significant assignment coding unique definition metadata, and coding itself has Special Significance, is level and divides corresponding rule.

(2) property value definition: for the data of Water demand process, define its property value, here to represent the technical parameter of metallurgy industry production equipment for attribute, clear definition, avoid repetition, classification five property values.

(3) rule extraction: for the property value sorted, according to semantic analysis, does a step and disassembles processing, be called participle, special symbol is rejected, form traditional key word, in conjunction with ordering rule, key word is duplicate removal again, carries out weight allocation before duplicate removal, what adopt here is Judgement Matricies method, according to property value first 5, by Paired Comparisons, list matrix, be two 5X5 matrixes and do multiplying, the applicating geometric method of average (root method):

Calculate the product of each element of each row of judgment matrix A mi;

Calculate the n th Root of mi;

Vector is normalized;

This vector is required weight.

(4) index is set up: carry out all data index to object matching data, and this index changes its index file in real time along with the increase and decrease of target data, and index stores is in server file.

(5) language fuzzy matching: for the rule extracted and key word, metadata and target data are carried out fuzzy matching, its rule is priority weight, again secondary key, again key word quantity, final according to customer service requirement, choose metadata to mate with N number of desired value, the automatic record matching relation of system, relation is once set up, an initial step desk checking, get up as primary data store, metadata or target data change afterwards, do not need manual maintenance, original matching relationship resets by system automatically, optimum combination.

Wherein, the structural matrix of described employing analytical hierarchy process determines attribute weight, is a kind of mathematical theory, sketches here: compare i-th element and a jth unit

During the importance of relative certain factor of last layer of element, the relative weighting a of usage quantity _ijdescribe.If total n element participates in comparing, then

A=(a _ij) n × n is called pairwise comparison matrix.Form the quantity basis that pairwise comparison matrix is analytical hierarchy process, for particular problem, the expert that is experienced, good sense by every field provides.

In addition, this technology encapsulates the citable universal method lucene of expression formula and carries out the participle index establishing method of target data for calling, and its feature is:

(1) index file form is independent of application platform.Lucene defines a set of index file form based on octet, makes the application of compatible system or different platform can share the index file of foundation.

(2) on the basis of the inverted index of traditional full-text search engine, achieve block index, for new file set up small documents index, index speed can be promoted.Then by the merging with original index, the object of optimization is reached.

(3) outstanding OO system architecture, makes the learning difficulty for Lucene expansion reduce, conveniently expands New function.

(4) devise the text analyzing interface independent of language and file layout, index completes the foundation of index file by accepting Token stream, user expands new language and file layout, only needs the interface realizing text analyzing.

(5) given tacit consent to and achieved a set of powerful query engine, even if user can obtain powerful query capability without the need to oneself writing code system, in the inquiry realization of Lucene, acquiescence achieves boolean operation, fuzzy query (Fuzzy Search [11]), Querying by group etc.

The concrete steps of master data collecting method are as follows: (following steps are the content increased)

Step one, carry out hierarchical layered definition for shop equipment, according to POF as partitioning standards, carry out the fractionation of stratification component units, the data split are metadata, the field of metadata comprises: metadata unique encodings ls_sid, device name ls_name, device type ls_sblx, technical parameter jsdxlx, device coding ls_code, father encode p_code, specifications and models specification, figure number png_num.

Step 2, carry out the setting of technical parameter attribute for the every kind equipment of each level, according to machinery, hydraulic pressure, electrically, automation equipment carries out Classifying Sum, for each subdivision subclass, until subclass can the technical data of sake of clarity equipment, property value is at least 5 does not have ceiling restriction, subclass can be foundation according to various kinds of equipment instructions technical parameter information, technical parameter field is according to distinct device type, classification is different, concrete field at least comprises: technical parameter attribute unique encodings jsdxlx_sid, technical object coding cs_code, parameter name cs_name, clock rate cs_lx, parent coding cs_code_p, attribute coding sx_code, (parameter object adds multiple property value such as technical parameter and is Property Name sx_name: ACA10 automation class frequency converter, property value is ACA1001 title, ACA1002 specifications and models, ACA1003 manufacturer, ACA1004 factory number, ACA1005 figure item, these concrete values collect masterplate by equipment to import in system)

Be example according to a class automation equipment, to its attribute according to selected front 5 property values of sequence, each property value of each equipment compares in pairs by significance level, compares degree and determines by 1-9 mathematics scale, draw the weight of 5 property values of every kind equipment;

Described scale from 1-9, represent two property values the former than the significance level of the latter, numerical value is larger, and significance level is stronger; If inverse is property value i is a with the ratio of the importance of property value j _ij, so property value j with the ratio of property value i importance is $a_{\mathrm{ji}}=\frac{1}{a_{\mathrm{ij}}}.$

Construct all judgment matrixs in each level and try to achieve the approximate solution of its maximal eigenvector; Middle layer A ₁-A _neach property value is to destination layer Z constituent ratio comparatively judgment matrix

N is the quantity of technical parameter property value, provides a in paired multilevel iudge matrix A _ijscale numerical value, according to the size order of scale numerical value, draw property value ordering rule, and property value ordering rule be saved in database table.The structure of table is: device coding, technical parameter, property value ordering rule value.Draw most important front 5 property values sequence rule value of sequence.

Step 3, set up material data storehouse, its texture field is: material unique encodings (unique identification) wlbm, material description wlms, specifications and models ggxh, supplier gys, unit price price, quantity count, stock ground storedreess, contract number contactnumber (initial spare part data are entered in system by ERP system interface).The material description corresponding for all material unique encodings and specifications and models make the full table lucene index according to Chinese word segmenting, and index file is stored in application server end.

Step 4, to be mated with key word in index file by the ordering rule value taken out, show that matching degree=both mate identical characters number/sequence rule value number of characters * 100%, then sort according to matching degree, 50% is more than or equal to for matching degree, the material data of first of matching degree being sorted is known as principal mark, gets first 5 and preserves fuzzy matching records; If matching degree is all less than 50%, be then judged to be without fuzzy matching target.

When for metadata and spare part data movement, processing rule is:

1) the no longer record that repeats of metadata, has the then rule extraction again of new record, note: this rule is same as the rule extracted scarcely, because do not have repetition values, relevant with concrete each equipment 5 property values in rule.

2) after spare part Data Update, need to re-establish index file, remain the not record of repetition herein, after having new record, re-establish index, although the index again set up is different from the previous case, do not affect former fuzzy matching record.

Beneficial effect of the present invention:

A kind of master data based on Fuzzy Level Analytic Approach collects technology, based on Fuzzy Hierarchy Method in theory, adopt the process of data hierarchy level, rule weight allocation, the method of multiple key fuzzy matching, then by realizing modularization to the changeable assembling of limited instruction set, finally verify again in conjunction with specification, dynamic analysis mode realizes the once collection of Data Collection aspect, less maintenance, the purpose data classifying technology that dynamic logic upgrades, allow user need not complain problems in Data Collection again, and data post is safeguarded simple, fast, efficiency and the service ability responding user's request fast can be improved constantly.Technically, following income is achieved:

1. metadata masterplate definable: the masterplate of metadata collecting can configure according to business rule.

2. Service Component is configurable: undertaken dynamically by instruction set, flexible configuration.The dirigibility of increase system and agility.

3. after business changes, only need the data division of more new change, need not again collecting of other, can realize the business demand of dynamic change, promotes the ability of response user's request fast

4. business datum maintenance is little, has been Job that some regularly perform, and is used for the variation of periodic scanning metadata, and general objectives data variation amount is relatively less here, and frequency is not very large.

Accompanying drawing illustrates:

Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described further.

Fig. 1 is step analysis and the fuzzy matching activity chart of the specific embodiment of the invention;

Embodiment:

The present invention proposes a kind of master data based on Fuzzy Level Analytic Approach and collect technology, as follows with example in detail by reference to the accompanying drawings:

Be explained in detail it for embodiment describes in detail with steel industry master data management below, this example traffic demand is:

Each function production equipment of factory is carried out tree-like management, collects its technical parameter, and carry out coupling with level Four spare part data and associate.

Performing step:

1. hierarchical layered definition is carried out for shop equipment, this according to POF as partitioning standards, carry out the fractionation of stratification component units, the data of fractionation are the metadata mentioned in invention, and in this business model, metadata is the device model of a tree structure in fact.Here for certain steel mill's asset equipment master data, the field that metadata needs comprises: unique identification sid, device name ls_name, device type ls_sblx, technical parameter jsdxlx, device coding ls_code, father encode p_code, specifications and models specification, figure number png_num etc.

2. the setting of technical parameter attribute is carried out for the every kind equipment of each level, here according to machinery, hydraulic pressure, electrically, automation equipment carries out Classifying Sum, for each subdivision subclass, until subclass can the technical data of sake of clarity equipment, property value is at least 5, does not have ceiling restriction.Technical parameter field is according to distinct device type, classification is different, concrete field at least comprises: jsdxlx_sid, technical object coding cs_code, parameter name cs_name, clock rate cs_lx, parent coding cs_code_p, attribute coding sx_code, (parameter object adds multiple property value such as technical parameter and is Property Name sx_name: ACA10 automation class frequency converter, property value is ACA1001 title, ACA1002 specifications and models, ACA1003 manufacturer, ACA1004 factory number, ACA1005 figure item, these concrete values collect masterplate by equipment to import in system)

3. be example according to a class automation equipment, to its attribute according to selected front 5 property values of sequence, each property value of each equipment compares in pairs, draws the weight of 5 property values of every kind equipment, as figure:

Table 1 ~ 9 quantity scale value and implication

Construct all judgment matrixs in each level and try to achieve the approximate solution of its maximal eigenvector

Middle layer A ₁-A ₅each factor is to destination layer Z constituent ratio comparatively judgment matrix

$A=\left(\begin{array}{ccccc}{a}_{11}& a_{12}& a_{13}& a_{14}& a_{15}\\ a_{21}& a_{22}& a_{23}& a_{24}& a_{25}\\ a_{31}& a_{32}& a_{33}& a_{34}& a_{35}\\ a_{41}& a_{42}& a_{43}& a_{44}& a_{45}\\ a_{51}& a_{52}& a_{53}& a_{54}& a_{55}\end{array}\right)$

Provide a in paired multilevel iudge matrix A _ijscale numerical value:

Such as:

Z	A1	A2	A3	A4	A5
						A1	1	5	4	4	1
A2	1/5	1	1/2	1/2	1/3
						A3	1/4	2	1	1	1/4
A4	1/4	2	1	1	1/4
						A5	1	3	4	4	1

4. then can obtain:

$A=\left(\begin{array}{ccccc}1& 5& 4& 4& 1\\ 1/5& 1& 1/2& 1/2& 1/3\\ 1/4& 2& 1& 1& 1/4\\ 1/4& 2& 1& 1& 1/4\\ 1& 3& 4& 4& 1\end{array}\right)$

Therefore draw the weight of 5 attributes, it is relevant that this is rule.In conjunction with actual parameters respectively:

Sequence number	Project	Concrete device value
			A1	Title	AC converter
A2	Specifications and models	6DV62-0
			A3	Manufacturer	Siemens
A4	Factory number	120823N23
			A5	Figure item	0876221

Rule is herein: " AC converter 6DV62-0 Siemens 120823N230876221 " 31 characters, rank successively, this rule is saved in inside database table, list structure is: device coding (RYZJQ001E001), technical parameter (ACA10), rule value (AC converter 6DV62-0 Siemens 120823N230876221).

5. carry out full index foundation for spare part data, adopt method lucene participle of increasing income to set up here, do not set up special segmenter here, because can comprise at present for the Chinese grammer generally applied.Here the material data of certain steel mill is exemplified as, its texture field is: material code (unique identification) wlbm, material description wlms, specifications and models ggxh, supplier gys, unit price price, quantity count, stock ground storedreess, contract number contactnumber. (initial spare part data are entered in system by ERP system interface).The material description corresponding for all material code and specifications and models make the full table lucene index according to Chinese word segmenting, and index file is stored in application server end.

6. go to mate with index key by the rule that takes out and keyword, in practical business, user is required of more than two keywords and is relation establishment, therefore the data relationship in fuzzy matching procedure is at least two Keywords matching, because multi-keyword matching quantity is more, here when specific implementation, known by the principal marks that does maximum for coupling key word, both expressions matching relationship mates completely or be similar to mate completely.Here because quantity is meaningless too much, choose front 6 matching relationships meeting rule relevant and store.This completes step analysis fuzzy matching.4. such as walk the rule drawn is: " AC converter 6DV62-0 Siemens 120823N230876221 " 31 characters, as fuzzy matching desired value, the index file of this string value and foundation is carried out system Auto-matching, adopt the participle coupling of lucene, the material description 5. set up is found to comprise this record of the identical material code of above desired value, then matching degree (both mate identical characters number/31*%) is made, here 10 records are found, according to matching degree, (matching degree need in 50%-100% interval, if be less than 50%, be judged to be without fuzzy matching target) sequence, (material code is 740005158) is known as principal mark using first, then get first 5 and preserve fuzzy matching record.So far level fuzzy matching completes.

7., when for metadata and spare part data movement, processing rule is:

3) the no longer record that repeats of metadata, has the then rule extraction again of new record, note: this rule is same as the rule extracted scarcely, because do not have repetition values, relevant with concrete each equipment 5 property values in rule.

4) after spare part Data Update, need to re-establish index file, remain the not record of repetition herein, after having new record, re-establish index, although the index again set up is different from the previous case, do not affect former fuzzy matching record.

Claims (2)

1., based on a master data collecting method for Fuzzy Level Analytic Approach, it is characterized in that:

Step one, carry out hierarchical layered definition for shop equipment, according to POF as partitioning standards, carry out the fractionation of stratification component units, the data split are metadata, the field of metadata comprises: metadata unique encodings, device name, device type, technical parameter, device coding, father's coding, specifications and models, figure number;

Step 2, carry out the setting of technical parameter attribute for the every kind equipment of each level, according to machinery, hydraulic pressure, electrically, automation equipment carries out Classifying Sum, for each subdivision subclass, until subclass can the technical data of sake of clarity equipment, technical parameter property value is at least 5 does not have ceiling restriction; Technical parameter field at least comprises: technical parameter attribute unique encodings, technical object coding, parameter name, clock rate, parent coding, attribute coding, Property Name;

To technical parameter attribute according to selected front 5 property values of sequence, each property value of each equipment compares in pairs by significance level, compares degree and determines by 1-9 mathematics scale, draw the weight of 5 property values of every kind equipment;

Construct all judgment matrixs in each level and try to achieve the approximate solution of its maximal eigenvector; Middle layer A ₁-A _neach property value is to destination layer Z constituent ratio comparatively judgment matrix:

N is the quantity of technical parameter property value, provides a in paired multilevel iudge matrix A _ijscale numerical value, according to the size order of scale numerical value, draw property value ordering rule, and property value ordering rule be saved in database table; The structure of table is: device coding, technical parameter, property value ordering rule value, draws most important front 5 the property value ordering rule values of sequence;

Step 3, set up material data storehouse, its texture field is: material unique encodings, material description, specifications and models, supplier, unit price, quantity, stock ground, contract number; The material description corresponding for all material unique encodings and specifications and models make the full table lucene index according to Chinese word segmenting, and index file is stored in application server end;

Step 4, to be mated with key word in index file by the ordering rule value taken out, show that matching degree=both mate identical characters number/ordering rule value number of characters * 100%, sort according to matching degree, 50% is more than or equal to for matching degree, the material data of first of matching degree being sorted is known as principal mark, gets first 5 and preserves fuzzy matching records; If matching degree is all less than 50%, be then judged to be without fuzzy matching target.

2., as claimed in claim 1 based on the master data collecting method of Fuzzy Level Analytic Approach, it is characterized in that: described scale from 1-9, represent two property values the former than the significance level of the latter, numerical value is larger, and significance level is stronger; If inverse is property value i is a with the ratio of the importance of property value j _ij, so property value j with the ratio of property value i importance is