CN112883022A - XML template-based object data cleaning and converting method - Google Patents

Abstract

The invention relates to the technical field of information technology, and provides an object data cleaning and converting method based on an XML template. The method aims to solve the problems of hard coded data cleaning and processing, the problem that restarting service is needed when coding is completed, the problems of expanding service scene cleaning, repeatedly developing codes and the like. Acquiring json messages of table operation information of other databases; performing data source filtering on the JSON message through a data source filtering tag of an XML rule template, screening out data to be cleaned, and cleaning the data to be cleaned through an XML rule template data cleaning rule to obtain data information to be operated; in the data information to be operated, selecting a table name to be operated and a field in the table as an XML source data service main key, and defining in an XML rule template; and associating the XML source data service main key with the table and the field of the local database to generate an SQL statement which can insert, delete, update and query the table corresponding to the local database.

Description

XML template-based object data cleaning and converting method

Technical Field

The invention relates to the technical field of information technology, and provides an object data cleaning and converting method based on an XML template.

Background

In current software system architectures, object transformation and data cleansing are frequently encountered problems. At present, networks are developed abnormally, interface call of cross-system and service is often encountered, and data in other systems are converted and stored in the system so that the system can handle business transaction. And in combination with the specificity of other system data structures, the requirement data format of the system is cleaned. Although the method is feasible, the flexibility is poor, too much maintenance and development amount is generated under the condition that the service expansion demand is continuously changed, a large amount of manpower and physical cost are consumed, and the cost cannot be tolerated in the internet financial industry with a rapidly developed network, so that the data cleaning and converting method is developed, and the cleaning is the ideal field attribute aiming at different databases and different table structures and table fields.

The XML template-based object data cleaning and converting method can request data through the configured cleaning message, and solves the problems of cleaning and processing of hard coded data, restarting of service required for coding completion, expanding of service scene cleaning, repeated code development and the like. The flexibility is greatly improved, the service expansion is simple, and the maintenance and operation cost is greatly reduced.

Disclosure of Invention

The invention aims to solve the problems of hard coded data cleaning and processing, restarting service required when coding is completed, expanding a cleaning service scene, repeatedly developing codes and the like.

In order to solve the technical problems, the invention adopts the following technical means:

an object data cleaning and converting method based on an XML template comprises the following steps:

step 1, acquiring json messages of a data bus, which contain table alignment operation information of other databases, through kafka;

step 2, performing data source filtering on the JSON message through a data source filtering tag of the XML rule template, and screening out data to be cleaned, which needs to be cleaned and contains table names and field information;

step 3, cleaning the data to be cleaned through XML rule template data cleaning rules to obtain data information to be operated;

step 4, in the data information to be operated, selecting the name of the table to be operated and one or more fields in the table as XML source data service main keys and defining the XML source data service main keys in an XML rule template;

and 5, associating the XML source data service main key with the table and the field of the local database to generate an SQL statement which can insert, delete, update and query the table corresponding to the local database.

In the above technical solution, the data cleaning rule includes:

simple cleaning rules: analyzing the received json message, and copying the appointed field to a corresponding value;

complex cleaning rules: and analyzing the received json message, and performing mathematical operation, type conversion or logic processing on the specified field to obtain a corresponding value.

Self-defining a cleaning rule: analyzing the received json message, extending the self-defined function method by the attribute, and obtaining the corresponding value.

In the above technical solution, the simple cleaning rule is as follows:

declaring a simple mapping relation simple label, which contains mapping description and is used for remarking the attribute of desc of the current simple mapping relation;

the attribute value relation of the result table of field mapping in the declaration request parameter table comprises the following attributes:

col: including a statement cleansing target table field;

the relationship value: a request source table field is declared.

4. The XML template-based object data cleaning and converting method according to claim 1, wherein the complex cleaning rule is as follows:

declaring a simple mapping relation complete label;

the mapping relation label of the field mapping result table in the declaration request parameter table comprises the following attributes:

col: declaring a cleaning target table field;

declaring a complex mapping processing rule rel tag, containing the attributes:

handle: declaring a processing mode;

ref-type: declaring a value type;

data-Type: declaring a value type or a conversion mode;

rel tag value: a request parameters table field indicating a flush.

In the above technical solution, the custom cleaning rule is as follows:

< querylndex >: declaring a mapping table query condition, having at least one < or > sub-tag, and having the following attributes: desc: mapping description is used for remarking the current simple mapping relation;

< or >: declaring a query condition mapping relationship having at least one < rel > sub-label, the attributes comprising:

rel: and declaring a corresponding attribute value field in the mapping result table.

Because the invention adopts the technical scheme, the invention has the following beneficial effects:

because a large amount of interface calls and data requests exist in the internet and financial industry. Different data structure differences exist among different mechanisms and services. For example, xx loan products of organization A and xxx loan products of organization B are summarized into a statistical statement of a business system, besides normal communication cost, interface debugging cost is also included, and through using a cleaning engine, only the operation of a corresponding table needs to be lost to kafka, and the business system accesses kafak to call the cleaning engine, so that the data updating of the business system can be automatically realized. In addition, due to the expansibility of the service, a hard coding mode is adopted, the operation requirement difficulty is deepened and the input of manpower is increased in a service scene that transaction data is frequently changed, and if the service needs to be expanded, the services of other mechanisms are added, and the recoding debugging is needed. The data cleaning method engine is used, and great help can be obtained in the deployment of the server, the expansion of the service and the maintenance of the operation.

Drawings

FIG. 1 is a logic block diagram;

FIG. 2 is an example of a source object (json message input);

FIG. 3 is an example XML template configuration;

FIG. 4 is an SQL output example.

Detailed Description

The invention provides an object conversion method based on an XML template, which comprises the following steps:

step S1: defining a tag of an XML rule template;

step S2: defining XML source data filter labels;

step S3: defining XML source data cleaning rules;

s3.1: simple cleaning rules (attribute direct mapping);

s3.2: complex cleaning rules (attribute handling mapping);

s3.3: self-defining cleaning rules (attribute extension self-defining function method);

step S4: defining an XML source data query condition;

step S5: generating corresponding cleaned SQL sentences (including addition, deletion, modification and check);

the request parameters are JSON objects of the source table, and corresponding SQL sentences are output through types (adding, deleting, changing and checking) needing cleaning. (request JSON object, including operation type (I, U, D))

Step S1 specifically describes:

in the above technical solution, the tags of the XML rule template are as follows:

< nodes >: a root node, containing no attributes, no other type tags, having to or at least containing a < node > sub-tag;

< node >: the washing node must or at least contains a < mapping > sub-tag, and the attributes include the following:

clean result: SQL table names representing output cleansing;

< mapping >: matching tags, must include sub-tags: relations and querylndex. No or only one sub-tag: < filters >. The attributes include the following:

tableName: representing the table name requested in the input parameter (source table name);

in the above technical solution, the field value rule descriptor in step S1 is as follows:

tableName: matching tableme in the Json attribute of the request parameter;

clean result: table showing the results of the final cleaning;

and (4) business rules:

the clearresult can only have one configuration in one configuration file (namely, only one clearresult configuration at most exists in one cleaning target table); tableName can have multiple configurations (representing washing into different service tables (clearresult) according to different service scenarios (filter attribute);

the processing logic:

when a project is started, the xml configuration is converted into Java object NodeTablesVOs through an xml analysis tool Digeter;

after a kafka request is obtained, converting the Json character string into Map, and obtaining a corresponding node configuration set List < NodeTableVO > from NodeTableVOs through a tableName in a request parameter;

step S2 specifically describes:

the XML source data filter tags are as follows:

< filters >: and configuring a filtering configuration parent tag and configuring a filtering attribute in the request parameter. There is no or at least one < or > sub-tag. The attributes include the following:

desc: filtering description for remarking the current filtering relation;

< or >: a filter group configuration tag having at least one < filter > sub-tag;

< filter >: filtering attribute configuration without sub-tags. The attributes include the following:

source: declaring a table value field in the request parameter;

match: declare a comparison type, such as: !! (support equal and not equal to both);

the filter value: filtering the matched attribute values;

the field value rule descriptor is as follows:

source: declare a table value field in the request parameter (acquired in a request message Json and acquiescent with the attribute after the after node is acquired);

match: declare a comparison type, such as: !! (support equal and not equal to both); and (3) service comparison shows that: requesting that an attribute in Json be of type (e.g., String type would have an attribute value with a double quote ("), and numeric type would have no quote (where a decimal number can be matched from a numeric value))

String comparison logic:

when the filter value configures null, the source attribute value obtained from the entry is "" and returns true:

when the filter value is not null, compare the equals value! Comparing the results with the results of the next equials;

digital comparison logic:

integer type

When the source attribute gets null, return false! Returning to true;

when the source attribute acquisition is not null, the convert Integer comparison returns ═ heel! A value of;

double type

When the source attribute gets null, return false! Returning to true;

when the source attribute get is not null, convert to BigDecimal (precision bit 4) compare back ═ with! A comparative value;

the filter value: filtering the matched attribute values;

and (4) business rules:

the or label inner sub-labels are filters, and the relationship among a plurality of the or labels is that the or label inner filters meet one condition, and the current or filtration meets the condition;

the or labels are in an and relationship (namely, one filter configured by a node contains a plurality of peer or labels, and the matching values of all the or labels need to be met at the same time);

the processing logic:

i. judging whether List < NodeTableVO > is empty (or the size is 0) or not, if so, finishing cleaning, and prompting that cleaning configuration does not exist; if not, traversing List < NodeTableVO > set;

ii, obtaining a filter configuration List < NodeTableFiltererssVO > in the NodeTableVO, and traversing the List < NodeTableFilterersVO > (if the filter is empty or the size is 0, the filter is passed, and no corresponding filter value exists is prompted);

obtaining the filters set List < NodeTableFiltersOrVO > in NodeTableFilterers 0 rsVO:

traversing List < NodeTableFiltersOrVO >, obtaining corresponding xml configuration according to NodeTableFilters0rVO, obtaining source values (obtaining types) in request parameters, obtaining configured xml filtering values, matching filtering types in xml, matching one filtering type, namely returning to true, and traversing break;

v. in step iii the NodeTableFiltersOrsVO alignment results are returned to step ii, if a false occurs, the loop of step ii is ended, and the filtering is not passed. Traversing List < NodeTableVO > in the step i;

vi, the whole List < NodeTableFiltersOrsVO > is traversed and passed, and the traversal value NodeTableVO in the current step i is returned, namely the currently matched xml cleaning configuration is obtained;

step S3 specifically describes:

the XML source data washing tag is as follows:

< relationships >: a mapping relationship is declared. There is no or at least one < simple >, < complex >, < customer-define > sub-tag. The attributes include the following:

desc: filtering description for remarking the current filtering relation;

s3.1 simple cleaning rules

That is, the received json message is parsed and the specified fields are copied to corresponding values

Defining a label:

< simple >: a simple mapping relationship is declared, with no or at least one < relation > sub-tag. The attributes include the following:

desc: mapping description is used for remarking the current simple mapping relation;

< relationship >: declare the result table attribute values of the field mappings in the request parameter table without word tags. The attributes include the following:

col: declaring a cleaning target table field;

the relationship value: a claim request source table field;

and (4) business rules:

under a single label, a plurality of relation sub-labels are contained, the corresponding col is the final cleaning target, and the relation sub-label value is the source table after-table attribute field of the after-table node.

The processing logic:

i. traversing and filtering the passed NodeTableVO attribute, and acquiring a corresponding simple cleaning rule attribute configuration List < NodeTableRelationsSimRelVO > set;

traverse List<NodeTableRelationsSimRelVO>Gather, get c of xml configuration_oA value l and a relation tag value, acquiring a corresponding key value through a map in the relation tag value acquisition request parameter, and storing a corresponding value in the acquired map into a washed mappingSourceValue (the mappingSourceValue is the map after temporary attribute washing (the key with the target col as the map, and the mapping value is the value));

s3.2 Complex cleaning rules

Defining a label:

< complete >: a complex mapping relationship is declared, with no or at least one < relation > sub-tag. The attributes include the following:

desc: mapping description is used for remarking the current complex mapping relation;

< relationship >: the request parameter table is declared to have a field mapping result table mapping relationship with no or at least one < rel > sub-label. The attributes include the following:

col: declaring a cleaning target table field;

< rel >: a complex mapping process rule is declared without sub-tags. The attributes include the following:

handle: declaring a processing mode;

ref-type: declaring a value type;

data-Type: declaring a value type or a conversion mode;

rel tag value: a request parameters table field indicating a flush;

and (4) business rules:

under one complete label, there are several relation sub labels, the corresponding col is the final cleaning target, and the relation sub label value is the source table after-table attribute field of the after-table node.

The processing logic:

i. traversing and filtering the passed NodeTableVO attributes, and acquiring a corresponding attribute configuration List < NodeTableRelationsComplexVO > set of the complex cleaning rule;

traversing List < NodeTableRelationsComplexVO > set, obtaining col value of xml configuration and relation tag value set List < NodeTableRelationsComRelV0>

And iii, traversing the NodeTableRelationsComReIVO set to obtain configured handle, refType, dataType and rel values:

a) if handle is +/addTarget, col representing complex mapping adds an attribute value, the current applicable scene is String type and digit type, and col column adds an attribute

b) If handle is-,/, it means that the complex mapping is mathematically operated

c) If handle is turn, it indicates to do type conversion

d) If handle is dataFormat, the representation is time formatted

e) If handle is subString, it indicates to do string interception (the configured dataType at this time is the number of bits intercepted, such as: 0,10)

f) If the refType is local, the value configured for the relationship tag (the value in the xml configuration file) is represented

g) If refType is source, the value is expressed as the key value in the request parameter

h) If dataType is NUM, NUM1, NUM2, NUM, it indicates the exact bit after NUM is converted (the number following NUM);

i) if dataType is STRING: a character string operation is indicated (handle is + indicates that an additional operation is performed);

j) if dataType is yyyy-mm-dd, yyyy/mm/dd (or other date format): indicating the format for converting a configuration attribute from comma to front to comma to back (to be used in conjunction with handle for dataFormat)

k) If dataType is ═ Λ, ═ λ, · λ: here, it may be defined that after reading the configuration attribute value, an arbitrary attribute value (e.g., 1 ═ S, 2 ═ N, 1, 2, 3 ═ S, 3, 4, 5, 6 ═ N) or the like indicates that if the attribute value before the inner equal sign of the configuration is read, the attribute value is converted to the attribute value after the equal sign (i.e., check _ flag ═ 3, configured to 3, 4, 5 ═ N, the check _ flag is converted to N);

acquiring a relationship configuration value of the complex cleaning according to the handle, refType, dataType and rel processing logic (when the handle of the first relationship is +, -, +,/a, multiple operations can be performed on the same attribute, namely multiple operations can be performed on the same attribute in the complex cleaning), and storing the configuration value into the cleaned mappingSourceValue (the mappingSourceValue is the key with the target col as the map and the mapping value is value);

example 1, as follows: setting the service type field as EOS + original table service main key, and configuring as follows:

example 2, as follows: the formatted original time format yyyy-MM-dd is formatted as yyyMMdd and configured as follows:

<relation col＝″SYS_DATE〞>

<rel handle＝〞dateFormat″refType＝″source″dataType＝〞yyyy-MM-dd，yyyyMMdd″>TRANS_DATE</rel></relation>

example 3, as follows: the conversion of transaction states 1, 3 to F (failure), 0, 2, 4 to S (success), and null to P are configured as follows:

example 4, as follows: and (3) remitting the transaction amount principal with the transaction amount interest into the transaction amount, wherein the configuration is as follows:

s3.3 custom cleaning rules

Defining a label:

< customer-define >: a custom cleansing rules (extended implementation) tag is declared. There is and only one < rel > sub-tag. The attributes include the following:

col: declaring a target implementation class of a custom cleaning rule;

and (4) business rules:

when the cleaning rules in the step 2 are not matched, an extension method for custom realization is defined, the method only needs to inherit a parent class TransAnalyssCustomerSupFacade, and a handleCluster method of the parent class needs to be rewritten (the input parameter is the current request parameter, the xml configuration, the cleaned attribute Map, and the output parameter is the temporary variable Map < String, 0 object >);

the processing logic:

i. traversing and filtering the passed NodeTableeV 0 attribute to obtain a corresponding custom rule configuration attribute NodeTableRelationsCusDefineVO;

calling spring Util.getBeam to obtain a TransAnalyssCustomerSupFacade object through a value configured by col in xml;

iii, calling a self-defining processing method of TransAnalyssCustomerSupFacade to obtain the cleaned map

Traversing the map, and storing the map, the key and the value which are cleaned by self-definition into the mappingSourceValue (the mappingSourceValue is the map which is cleaned by the temporary attribute (the key takes the target col as the map, and the mapping value is the value);

s4 specifically states:

in the above technical solution, the tags of the XML rule template in step S4 are as follows:

< querylndex >: a mapping table query condition is declared, with at least one < or > sub-tag. The attributes include the following:

desc: mapping description is used for remarking the current simple mapping relation;

< or >: a query condition mapping relationship is declared, with at least one < rel > sub-tag. The attributes include the following:

rel: declaring a corresponding attribute value field in a mapping result table;

and (4) business rules:

acquiring a corresponding value from a temporary map (mapped Source value) through configured rel configuration, assembling an attribute after SQL is assembled, generating an or relation by an intra-or label rel, and generating an and relation by an or label at the same level;

the processing logic:

i. traversing and filtering the passed NodeTableVO attribute to obtain a NodeTableQueryndex attribute;

obtaining the set List of xml configuration from NodeTableQueryndex < NodeTableQueryndexors >

if the List < NodeTableQueryDirectDeordors > is null, the configuration is prompted to be null, the configuration is checked, and the method ends

if not, go through List < NodeTableQueryndexros > to get the collection object (multiple rel configured by xml) (the temporal And ad relationship between each NodeTableQueryndexros, or temporal And or relationship between the sub-labels rel)

v. the configured rel value is a key for cleaning a temporary map (mappingsourcevalue), a corresponding attribute value is obtained from the map (mappingsourcevalue), and a condition character String after the where is generated (if the value type in the map is not String, a single quotation mark is not added when sql is spliced);

the step S5 processing is as follows:

and (4) business rules:

generating corresponding SQL by table name (table _ name), operation type (op _ type), map (mapping Source value) after washing and condition character string after person in request parameter

The processing logic:

i. obtaining operation type op _ type in request parameter

if the result is D (deletion), taking the conditional character string after the where, and combining the table name to generate an SQL statement corresponding to the delete;

if not D (I, U), traversing mappingSource value, taking the corresponding key, value, combining table name and where condition string, generating querySql, updateSql and insertSql strings, and packaging into a parameter TransAnalysisSQLVO.

Claims (5)

1. An object data cleaning and converting method based on an XML template is characterized by comprising the following steps:

step 1, acquiring json messages of a data bus, which contain table alignment operation information of other databases, through kafka;

step 2, performing data source filtering on the JSON message through a data source filtering tag of the XML rule template, and screening out data to be cleaned, which needs to be cleaned and contains table names and field information;

step 3, cleaning the data to be cleaned through XML rule template data cleaning rules to obtain data information to be operated;

step 4, in the data information to be operated, selecting the name of the table to be operated and one or more fields in the table as XML source data service main keys and defining the XML source data service main keys in an XML rule template;

and 5, associating the XML source data service main key with the table and the field of the local database to generate an SQL statement which can insert, delete, update and query the table corresponding to the local database.

2. The XML template-based object data cleaning and converting method according to claim 1, wherein the data cleaning rules include:

simple cleaning rules: analyzing the received json message, and copying the appointed field to a corresponding value;

complex cleaning rules: and analyzing the received json message, and performing mathematical operation, type conversion or logic processing on the specified field to obtain a corresponding value.

Self-defining a cleaning rule: analyzing the received json message, extending the self-defined function method by the attribute, and obtaining the corresponding value.

3. The XML template-based object data cleaning and converting method according to claim 1, wherein the simple cleaning rule is as follows:

declaring a simple mapping relation simple label, which contains mapping description and is used for remarking the attribute of desc of the current simple mapping relation;

the attribute value relation of the result table of field mapping in the declaration request parameter table comprises the following attributes:

col: including a statement cleansing target table field;

the relationship value: a request source table field is declared.

4. The XML template-based object data cleaning and converting method according to claim 1, wherein the complex cleaning rule is as follows:

declaring a simple mapping relation complete label;

the mapping relation label of the field mapping result table in the declaration request parameter table comprises the following attributes:

col: declaring a cleaning target table field;

declaring a complex mapping processing rule rel tag, containing the attributes:

handle: declaring a processing mode;

ref-type: declaring a value type;

data-Type: declaring a value type or a conversion mode;

rel tag value: a request parameters table field indicating a flush.

5. The method according to claim 1, wherein the custom cleansing rule is as follows:

< querylndex >: declaring a mapping table query condition, having at least one < or > sub-tag, and having the following attributes: desc: mapping description is used for remarking the current simple mapping relation;

< or >: declaring a query condition mapping relationship having at least one < rel > sub-label, the attributes comprising:

rel: and declaring a corresponding attribute value field in the mapping result table.

Images