CN107643956A - The method and apparatus for positioning the abnormal origin of abnormal data - Google Patents
The method and apparatus for positioning the abnormal origin of abnormal data Download PDFInfo
- Publication number
- CN107643956A CN107643956A CN201710722887.9A CN201710722887A CN107643956A CN 107643956 A CN107643956 A CN 107643956A CN 201710722887 A CN201710722887 A CN 201710722887A CN 107643956 A CN107643956 A CN 107643956A
- Authority
- CN
- China
- Prior art keywords
- data
- abnormal
- node
- bore
- pretreatment layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a kind of method and apparatus for the abnormal origin for positioning abnormal data, it is related to field of computer technology.One embodiment of this method includes：The data of leaf node are made comparisons with the data of corresponding pretreatment layer node, when the corresponding leaf node of some pretreatment layer node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned；When abnormal data is not more than a reference value, then the complete situation of each intermediate node in addition to pretreatment layer node is checked, when some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin and returned；Whether the bore of each intermediate node of the inspection in addition to pretreatment layer node and the standard gauge of abnormal data are consistent, if the bore of some intermediate node and the bore of abnormal data are inconsistent, it is determined that the intermediate node is abnormal origin and returned.The embodiment can effectively evade human error, reduce requirement to data abnormality processing person and rapidly and efficiently.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of method and dress of the abnormal origin for positioning abnormal data
Put.
Background technology
Data warehouse is to show for the ease of multidimensional analysis and multi-angle and data are carried out into storage institute by specific pattern
The relevant database set up, it is used to support the Analysis of Policy Making of enterprise or tissue to handle.
The judgement of the reason for data exception occurred for data warehouse, currently the only processing method is complete artificial pair
Data warehouse is investigated, and after data warehouse downstream feedback data are problematic, engineer starts should from the front end of data warehouse
Use bottom data source down to investigate in layer, find a problem points and handle one, then run again again, or asked all
Topic point, which is found out, to be uniformly processed, and is then run again again.
In process of the present invention is realized, inventor has found that at least there are the following problems in the prior art：It is existing for number
According to warehouse data exception the reason for determination methods, by it is pure it is artificial carry out, if running into nonstandard script, (such as script is complete
Text annotation is very few) will be to judging and handling work increase difficulty and cost, human cost is high, human error possibility is high and speed
Degree is slower.And whole process requires higher for investigating the people of problem, it is desirable to which its source to problem data, bottom processing are patrolled
Volume and professional knowledge it is quite known, otherwise can waste many times, cause operating efficiency to substantially reduce, in addition confuse direction,
Make a futile effort.
Therefore, need one kind badly rapidly and efficiently, can effectively evade human error, reduce the requirement to data abnormality processing person
Positioning abnormal data abnormal origin method and apparatus.
The content of the invention
In view of this, the embodiment of the present invention provides a kind of method and apparatus for the abnormal origin for positioning abnormal data, can
Effectively evade human error, reduce requirement to data abnormality processing person and rapidly and efficiently.
To achieve the above object, one side according to embodiments of the present invention, there is provided a kind of to position the different of abnormal data
The method often to originate from, the abnormal data corresponding logical relationship tree, the root node of the logical relation tree are the abnormal datas, leaf
Node is the tables of data of data source, and intermediate node is the intermediate data table being related to during the abnormal data produces,
Methods described includes：
Step 1, the data of the leaf node are made comparisons with the data of corresponding pretreatment layer node, the pretreatment layer
Node is the intermediate node that the data source of corresponding leaf node generates after pretreatment, wherein, when some pretreatment layer node and its
Corresponding leaf node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned, and otherwise performs step 2；
Step 2, judges whether the abnormal data is more than corresponding a reference value, when the abnormal data is no more than described
A reference value, then the complete situation of each intermediate node in addition to the pretreatment layer node is checked, otherwise performs step 3, its
In, when some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin and returned, and otherwise performs step 3；
Step 3, check the bore of each intermediate node and the mark of the abnormal data in addition to the pretreatment layer node
Whether quasi- bore is consistent, if the bore of the bore of some intermediate node and the abnormal data is inconsistent, it is determined that the middle node
Point is abnormal origin and returns.
Optionally, the logical relation tree be in service logic relational tree original corresponding to the abnormal data with
The unrelated part of positioning abnormal origin cuts off acquisition.
Further, the method for the abnormal origin of positioning abnormal data provided in an embodiment of the present invention also includes：Output institute
State the inventory of the abnormal origin of determination.
Further, the data of the leaf node are made comparisons with the data of corresponding pretreatment layer node including：
Based on the logical relation tree obtain the abnormal data with its produce during the pretreatment layer section that is related to
The direct mapping relations tree of point；
The data of the corresponding leaf node of the data of the pretreatment layer node in the directly mapping relations tree
Make comparisons.
Further, the bore of each intermediate node of the inspection in addition to the pretreatment layer node and the abnormal number
According to standard gauge whether unanimously include：
The bore inventory of each intermediate node in addition to the pretreatment layer node is obtained according to the logical relation tree；
Check whether the bore of the intermediate node in the bore inventory is consistent with the bore of the abnormal data.
To achieve the above object, other side according to embodiments of the present invention, a kind of positioning abnormal data is additionally provided
Abnormal origin device, the abnormal data corresponding logical relationship tree, the root node of the logical relation tree is the abnormal number
According to, leaf node is the tables of data of data source, and intermediate node is the intermediate data table being related to during the abnormal data produces,
Described device includes：
Sentence duty module, for step 1, the data of the leaf node made comparisons with the data of corresponding pretreatment layer node,
The pretreatment layer node is the intermediate node that the data source of corresponding leaf node generates after pretreatment, wherein, when some pre- place
It is inconsistent to manage the corresponding leaf node of node layer, it is determined that the pretreatment layer node is abnormal origin and returned, and is otherwise performed
Step 2；
Integrity check module, for step 2, judge whether the abnormal data is more than corresponding a reference value, when described
Abnormal data is not more than a reference value, then checks the complete situation of each intermediate node in addition to the pretreatment layer node,
Otherwise step 3 is performed, wherein, when some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin and returned, no
Then perform step 3；
Bore checks module, for step 3, checks the bore of each intermediate node in addition to the pretreatment layer node
It is whether consistent with the standard gauge of the abnormal data, if the bore of the bore of some intermediate node and the abnormal data differs
Cause, it is determined that the intermediate node is abnormal origin and returned.
Further, the device of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention also includes：Export mould
Block, the inventory of the abnormal origin for exporting the determination.
Further, it is described to sentence duty module and be further used for based on the logical relation tree acquisition abnormal data and its
The direct mapping relations tree for the pretreatment layer node being related to during generation, the pre- place in the directly mapping relations tree
The data for managing the corresponding leaf node of data of node layer are made comparisons.
Further, the bore checks that module is further used for obtaining according to the logical relation tree and removes the pretreatment
The bore inventory of each intermediate node outside node layer, check the bore of intermediate node in the bore inventory and the exception
Whether the bore of data is consistent.
To achieve the above object, other side according to embodiments of the present invention, additionally provide one kind and judge data exception
The electronic equipment of reason, the electronic equipment include：
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are by one or more of computing devices so that one or more of processing
The method that device realizes the abnormal origin of positioning abnormal data provided in an embodiment of the present invention.
To achieve the above object, other side according to embodiments of the present invention, a kind of computer-readable Jie is additionally provided
Matter, is stored thereon with computer program, realizes that positioning provided in an embodiment of the present invention is abnormal when described program is executed by processor
The method of the abnormal origin of data.
The method and apparatus of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention, are related to based on abnormal data
Original service logic relational tree carry out beta pruning and obtain its corresponding logical relation tree, cause the abnormal data so as to sort out
The tables of data set where abnormal direct factor occurs, the possibility and the complexity of investigation that then basis goes wrong,
The problem of and investigation big since comparatively possibility is easy, the data that may be present in the logical relation tree are investigated successively
The imperfect problem of source problem, tables of data and bore inconsistence problems, so as to navigate to the abnormal origin of abnormal data.Pass through this
Invention provides the above method, related personnel can it is self-service, quickly navigate to abnormal origin, and provide abnormal related tables of data
Information, in order to it is follow-up the problem of handle and repair, so as to shorten party in request's stand-by period, and can in time informing business handle into
Degree.Relative to existing localization method need to carry out it is pure be positioned manually abnormal with processing data, the inventive method can be advised effectively
Keep away human error, reduce to problem investigation processor requirement so that not can only developer could handle.
Further effect adds hereinafter in conjunction with embodiment possessed by above-mentioned non-usual optional mode
With explanation.
Brief description of the drawings
Accompanying drawing is used to more fully understand the present invention, does not form inappropriate limitation of the present invention.Wherein：
Fig. 1 is the method flow diagram of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention；
Fig. 2 is the schematic diagram of logical relation tree corresponding to abnormal index data F provided in an embodiment of the present invention；
Fig. 3 is the application flow schematic diagram of the method for the abnormal origin of positioning abnormal data provided in an embodiment of the present invention；
Fig. 4 is the mapping that abnormal index F provided in an embodiment of the present invention directly relies on Data Warehouse cleaning layer
The schematic diagram of relational tree；
Fig. 5 is the schematic diagram of the device of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention；
Fig. 6 is adapted for the structural representation of the computer system of the electronic equipment for realizing the embodiment of the present invention.
Embodiment
The one exemplary embodiment of the present invention is explained below in conjunction with accompanying drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
Arrive, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, the description to known function and structure is eliminated in following description.
The embodiment of the present invention provides a kind of method for the abnormal origin for positioning abnormal data, and the inventive method can apply to
In the database of data warehouse or other similar structures, the abnormal origin of abnormal data is positioned.For example, when discovery industry
When some achievement data of business application based on data warehouse generation has abnormal, you can used the method provided by the present invention is to the exception
Abnormal origin of the achievement data in data warehouse is positioned, it is determined that the tables of data of problem occurs, consequently facilitating targetedly
Carry out follow-up repair.Certainly, the use of the abnormal data of the inventive method positioning abnormal origin can also be data warehouse
The middle intermediate data generated involved by some achievement data.
In the methods of the invention, abnormal data corresponding logical relationship tree, the logical relation tree are by the abnormal number of the generation
The set with level formed according to involved tables of data during generation according to the logical relation for generating the data,
The root node of logical relation tree is the abnormal data, and leaf node is the tables of data of data source corresponding to the abnormal data, middle node
Point is the intermediate data table being related to during the abnormal data produces.The field processing that logical relation tree is included between each table is patrolled
Collect and the condition of subquery, logical relation tree can be obtained by the logical relation document for the business development that the abnormal data is related to
.
In the present invention, service logic original corresponding to abnormal data is included in the logical relation document of business development
Relational tree, logical relation tree be in service logic relational tree original corresponding to the abnormal data with positioning abnormal origin without
The part of pass cuts off acquisition.It is therein because service logic relational tree original corresponding to abnormal data is often very complicated
Some minor matters parts may not cause the abnormal generation of the abnormal data, or the possibility very little itself to go wrong,
These minor matters parts are i.e. it is believed that be unrelated with positioning abnormal origin.Therefore, in the present invention can be according to correlation experience pair
Original service logic relational tree carries out beta pruning, cuts off part wherein unrelated with positioning abnormal origin, obtains subsequent step use
In the logical relation tree of positioning abnormal origin.
In the present invention, can also be according to going wrong while beta pruning is carried out to original service logic relational tree
The size of probability, a certain degree of change is carried out to the position of the intermediate node of original service logic relational tree so that this hair
The inspection consistent with bore of the bright integrality that subsequently carries out can be gone wrong generally based on obtained logical relation tree priority check
The larger node of rate, so as to faster navigate to the abnormal origin of abnormal data.For example, checking the integrality of intermediate node
With bore it is whether consistent when, successively checked to data upstream node from the data downstream node of logical relation tree, then can be to original
While the service logic relational tree of beginning carries out beta pruning, the position that will appear from the node that problem may be big adjusts to data accordingly
Downstream, so as to can faster navigate to the node of abnormal origin when checking.
As shown in figure 1, the method for the abnormal origin of positioning abnormal data provided by the invention comprises the following steps one, step
Two and step 3.
Wherein, in step 1, the data of leaf node are made comparisons with the data of corresponding pretreatment layer node, wherein, when
The corresponding leaf node of some pretreatment layer node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned.
Pretreatment layer node is the intermediate node that the data source of corresponding leaf node generates after pretreatment, is generating abnormal number
During, data source data is then loaded into the system of generation abnormal data after being extracted by pretreatment, is stored in
In the pretreatment layer tables of data of system, for example, for data warehouse, data source data passes through ETL (Extract-Transform-
Load in the tables of data) extract, change, being loaded onto the pretreatment layer of data warehouse.
The corresponding pretreatment layer data table data of data source data is made whether consistent comparison by step 1, wherein
The content contrasted includes：Information, the data volumes such as the numerical value related to abnormal data and data volume in tables of data refer to spy
Determine data acknowledgment number related to abnormal data in the table in timing statisticses.By above-mentioned comparison so as to judging whether to be
The problem of the problem of data source or preprocessing process caused data exception.For example, due to pretreatment layer be when isolating by
Extracted according to certain logic increment or full dose, if data source changes, and the logic of isolating of pretreatment layer is not made
Respective change, may result in pretreatment layer data and production system data it is inconsistent, so as to cause to produce by subsequent logic
Raw data occur abnormal.
Therefore, when the corresponding data source data of the data of some pretreatment layer tables of data is inconsistent, it is determined that this is pre-
Process layer tables of data is abnormal origin, wherein, abnormal origin may be multiple.In the present invention, it is determined that pretreatment layer it is different
After normal origination data table, the inventory of the abnormal origin of determination, i.e. output and the inconsistent pretreatment of corresponding data source data are exported
The inventory of layer data table, the inventory can be sent to corresponding director, notify it to be handled so that corresponding director being capable of root
According to the inventory, for property the data source problem of system bottom or extraction, pretreatment and loading procedure are repaired.
In the present invention, step 1, the process that the data of leaf node are made comparisons with the data of corresponding pretreatment layer node
Specifically include：First, logic-based relational tree obtain abnormal data with its produce during be related to pretreatment layer node it is straight
Mapping relations tree is connect, the purpose of step 1 is whether the bottom data source of checking system and preprocessing process occur problem, because
This only need to find pretreatment layer tables of data corresponding to problem data in step 1, and its corresponding data source data is carried out
Contrast.In this step, directly reflecting for its corresponding pretreatment layer node is obtained by the logical relation tree of problem data
Relational tree is penetrated, other Rotating fields nodes between problem data and pretreatment layer node have been neglected in the tree so that by this
Tree quickly can directly find pretreatment layer node corresponding to problem data.Then, pre- in the direct mapping relations tree
The data for handling the corresponding leaf node of data of node layer are made comparisons.
In the application scenarios that the inventive method is faced, due to abnormal data, there is a strong possibility that property is due to bottom data
Caused by source problem, in the process of the present invention position abnormal data abnormal origin when first to bottom data source problem carry out
Investigation, the pretreatment layer section being related to directly is quickly found by the direct mapping relations tree of abnormal data and pretreatment layer node
Point carries out corresponding comparison check, it is determined that after abnormal origin, returns to the caller of the inventive method process, terminates positioning, make
For it is most of there is abnormal data in the case of can be transferred through step 1 and quickly determine abnormal origin and carry out follow-up repair
Multiple processing.
When the comparison Jing Guo step 1, the data of leaf node corresponding to abnormal data and the data of corresponding pretreatment layer node
It is all consistent, then illustrate that system bottom data source and extraction, pretreatment and loading procedure for data source have no problem, simultaneously
The scope of abnormal origin can also be reduced among other Rotating fields into logical relation tree between pretreatment layer and abnormal data
Node.
In step 2, judge whether abnormal data is more than corresponding a reference value, when abnormal data is not more than a reference value, then
The complete situation of each intermediate node in addition to pretreatment layer node is checked, wherein, when some intermediate node is imperfect, it is determined that
The intermediate node is abnormal origin and returned.In this step, size of the abnormal data than its a reference value is judged first,
The a reference value of abnormal data refers to standard value of the abnormal data under non-abnormal conditions, abnormal data and its a reference value not phase
Deng, a reference value can with empirically determined or obtained by the other systems of correlation, for example, can according to it is passing daily/
The normal value of weekly/monthly this data, forecast assessment go out a reference value of the value as the data.
When abnormal data is less than a reference value, then the complete situation of each intermediate node in addition to pretreatment layer node is checked,
Check whether intermediate node tables of data has shortage of data.Wherein, due to pretreatment layer node in step 1 the row of having been carried out
Look into, therefore no longer checked in step 2.
In the present invention, check that the complete situation of intermediate node tables of data can include the complete feelings of subregion for checking tables of data
Condition, when the subregion of some intermediate node tables of data is imperfect, it is determined that the intermediate node is abnormal origin.
In step 2, it is to generate the abnormal number that abnormal data, which is less than the reason for a reference value explanation causes the abnormal data,
According to intermediate data table the incomplete situation of data be present, because only that in the case where having lacked related data, abnormal data
Corresponding a reference value can be just less than.Therefore, when some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin, its
In, abnormal origin may be multiple, after abnormal origin determines, return to the caller of the inventive method process, terminate the present invention's
Position fixing process.In the present invention, after incomplete abnormal origin tables of data is determined, export determination abnormal origin it is clear
Single, i.e. there is the inventory of the tables of data of shortage of data in output, more specifically, in the inventory of output, can list every number
According to the subregion of the corresponding missing of table.The inventory can feed back to corresponding director, be operated with carrying out follow-up history complement, repairing.
If step 2 checks that each intermediate node tables of data is complete, and abnormal data is greater than corresponding a reference value, then enters
The bore of the follow-up step three of row unanimously checks.In step 3, each intermediate node in addition to pretreatment layer node is checked
Whether the standard gauge of bore and abnormal data is consistent, if the bore of some intermediate node and the bore of abnormal data are inconsistent,
The intermediate node is then determined as abnormal origin and is returned.Wherein, because pretreatment layer node has been carried out investigating in step 1,
Therefore no longer checked in step 2.
Wherein, check whether the bore of intermediate node and the standard gauge of abnormal data unanimously specifically include：Basis first
Logical relation tree obtains the bore inventory of each intermediate node in addition to pretreatment layer node, and logical relation is listed in the inventory
The bore of data related to the abnormal data in every layer of each tables of data in tree.Then, the middle node in bore inventory is checked
Whether the bore of point and the bore of abnormal data are consistent.Wherein, bore refers to Statistical Criteria, in logical relation tree, root node
Bore is the set of its all descendant nodes bore, i.e. the standard gauge of abnormal data should be it under non-abnormal conditions and produce
The union set of the bore for the data table data being related in journey.
In the present invention, the standard mouth of the bore for each intermediate node in addition to pretreatment layer node and abnormal data
The whether consistent inspection in footpath, that is, check whether the bore of data related to abnormal data in each intermediate node tables of data belongs to
The standard gauge set of abnormal data, if belonging to, it is determined that the bore of the tables of data is consistent with the bore of abnormal data, otherwise really
It is set to inconsistent, it is abnormal origin to determine the tables of data, wherein, abnormal origin may be multiple, after abnormal origin determines, return
The caller of the inventive method process, terminate positioning.In the present invention, the inconsistent abnormal origin tables of data of bore is being determined
Afterwards, the tables of data inventory of the abnormal origin of determination is exported, so that each table director is repaired.
Above-mentioned steps one, step 2 and step 3 are according to the possibility to go wrong and the complexity of investigation, from relative
For possibility it is big and the problem of investigation is easy, the data source that may be present investigated successively in the logical relation tree is asked
Topic, the imperfect problem of tables of data and bore inconsistence problems, enter the reason for for causing abnormal data to occur under normal circumstances
Go and progressively investigated, so as to realize the positioning of the abnormal origin for abnormal data.
The method of the abnormal origin of positioning abnormal data provided by the invention is carried out more with reference to an instantiation
Detailed description.
In this example, the method for the abnormal origin of positioning abnormal data provided by the invention is used under location data warehouse
Swim the abnormal origin of the abnormal index data of service application generation.It is that logic corresponding to abnormal index data F is closed shown in Fig. 2
System tree, the logical relation tree is that the original service logical relation tree in the logical relation document to achievement data F actual developments enters
What row beta pruning obtained.In the logical relation tree, root node is the achievement data F of service application APP generations, and leaf node is index
Tables of data C1, C2, C3 ... C7 of data source corresponding to data F, intermediate node represent to cause the straight of this achievement data F exceptions
The tables of data where factor is connect, including：During abnormal data produces the index that is related to collect layer data Table A DM1 and
ADM2, universal model layer data table GDM1 and GDM2, middle temporary layer tables of data TMP1, data cleansing layer data table FDM1,
FDM2、FDM3……FDM7。
In the logical relation tree, the data of data source are passed through data cleansing layer, middle temporary layer in data warehouse, led to
Achievement data F is generated after collecting layer and service application APP processing with model layer, index.Wherein, index collects layer and is mainly used in
Store the various indexs of various dimensions calculated, index collect can further be carried out on layer data statistic analysis,
Excavation or various aminated polyepichlorohydrins；Universal model layer is according to the numerous and jumbled bottom data of some subject area in data warehouse, with reference to industry
Business, the model that can describe some subject area service conditions abstracted；Middle temporary layer is during mould processing
For interim storage data；Data cleansing layer is the pretreatment layer that the embodiment of the present invention is mentioned above, and data source data is passed through
Extract, be stored in data cleansing layer after cleaning, conversion and loading.
As shown in figure 3, when positioning abnormal index data F abnormal origin, abnormity point and direct sources table are inputted first
The information such as name, as shown in table 1, input abnormal index title (electric business field statistical indicator：Selling cost under line), abnormal index
Table name (adm_s10_spwms_invt_stock_sum), abnormal index are (different compared to the situation of standard index (i.e. a reference value)
Chang Zhibiao is high or low than normative reference, is high in this example) and abnormal index standard gauge (with the financial settlement time
Meter, all storehouse quotation summations for counting the spare part commodity sold under date part warehouse lines subtract what statistics date was returned goods
The storehouse quotation summation of spare part commodity) this four parameters.Abnormal index title is used to inform which index of system (for database
Be exactly which field) it is problematic, the table name of the abnormal index of input be used to informing system exception index be in which table, according to
The table name of the abnormal index of input is assured that the general tables of data scope that abnormal index directly relies on.Abnormal index is compared
Whether the situation of normative reference needs first to check the complete situation of subregion during subsequently positioning abnormal origin for auxiliary judgment.
According to the four of above-mentioned input parameter systems can the logical relation tree based on the abnormal index carry out self-service track problems positioning,
Processing, system, which can be introduced into, after the completion of input sentences duty module.
Table 1
By sentencing duty module by each data source data table (i.e. production system table) and the number of Data Warehouse cleaning layer
Comparing is carried out according to table, it is main to compare the relevant informations such as the data volume related to abnormal index F, data value (amount of money).Sentence duty
Module is found different based on abnormal index F as shown in Figure 4 with the mapping relations tree that Data Warehouse cleaning layer directly relies on
Data cleansing layer data table FDM1, FDM2, FDM3 ... FDM7 corresponding to Chang Zhibiao, and by the data of above-mentioned cleaning layer tables of data
Corresponding production system tables of data C1, C2, C3 ... C7 data are made comparisons.
If data cleansing layer and production system are inconsistent, system can export an inconsistent tables of data inventory to corresponding
Director, notify that it handle and feedback result.In actual application, the cleaning of data source data, conversion and
The possibility that the process of loading generally occurs within problem is extremely low, and problem is frequently experienced in bottom data source (i.e. production system problem),
Such as data source changes, and the logic of isolating of data cleansing layer does not make respective change, the data cleansing caused by
The data of layer and the data of production system are inconsistent, when sentencing duty module check to exporting corresponding data cleansing layer after the above situation
Tables of data inventory is to corresponding director, pending reparation of isolating.
Sentence duty module directly can quickly judge that abnormal origin is in data source or number by above-mentioned comparison procedure
According to store interior, if data warehouse data cleaning layer is consistent with production system, that is, data source problem is excluded, is transferred to next step
Data warehouse checks oneself processing, reduce problem scope to Data Warehouse application layer, index collect layer, universal model layer or
Middle temporary layer.
Checked oneself inside data warehouse, to abnormal index F in logical relation tree to the data cleansing layer (exception shown in Fig. 2
Index F to FDM) between all table all carry out examination.First, the situation that normative reference is compared based on abnormal index is judged, is counted
Whether checked oneself according to store interior needs first to check the complete situation of subregion.
If abnormal index is lower than normative reference, need further to check from abnormal index F to data cleansing layer it
Between all tables the complete situation of subregion, wherein, process is checked oneself inside data warehouse need not reexamine data cleansing layer, because can
It is transferred to data store interior and checks oneself process, just illustrates that data cleansing layer has no problem.
System successively checks the integrality of tables of data in logical relation tree, wherein preferably, from logical relation tree
The service application APP of data downstream starts successively to check to data upstream, until checking the universal model before data cleansing layer
Layer GDM or middle temporary layer TMP.Because the node of logical relation tree data downstream is less than upstream, if problem occurs
In data downstream, the various tables of data in upstream can be required no using this checks sequence and faster navigate to abnormal rise
Source, it is on the contrary then the problem of can just check downstream node after checking out substantial amounts of upstream node.
It is main to verify tables of data corresponding to abnormal index timing statisticses scope in the complete situation of the subregion of inspection tables of data
The situation of subregion, such as abnormal index are the order volumes for counting on July 1st, 2017, then when checking the complete situation of subregion, it is necessary to
What is checked is exactly that data table related subregion on July 1 whether there is, if the subregion lacks, it is determined that the tables of data subregion is endless
Whole, system can export an imperfect table inventory.For example, the timing statisticses scope of abnormal index is May, it is a certain when checking
The subregion in individual tables of data May has missing, then as shown in table 2, table name gdm_m10_afs_ser_ is contained in the inventory of output
The inventory is fed back to problem table director, so as to according to clear in sum, the zone time scope lacked, each by stages with CSV
It is single to have adjusted task to carry out history complement, repair automatically, and feedback result.
| Table name | Subregion scope |
| gdm_m10_afs_ser_sum | 2017-05-10,2017-05-13,2017-05-14 |
Table 2
If the subregion of the tables of data of each intermediate node in logical relation tree is complete, or abnormal index is higher than mark
Quasi- index, system is transferred to the bore problem for checking intermediate node tables of data automatically.
Similar to integrity checking, system successively checks the bore of tables of data in logical relation tree, wherein preferably, from
The service application APP of data downstream in logical relation tree starts successively to check to data upstream, until checking data cleansing
Universal model layer GDM or middle temporary layer TMP before layer.
Wherein, arrange and export and remove according to the field processing logic between each table in logical relation tree and the condition of subquery
The bore inventory of each intermediate node data outside data cleansing node layer.By the standard of bore inventory and the abnormal index of input
Bore is contrasted.The bore of each table related data should belong to set (such as the standard in table 1 of standard gauge in bore inventory
Bore：In terms of the financial settlement time, the storehouse quotation summation for the spare part commodity sold under all statistics date part warehouse lines subtracts
Go the storehouse quotation summation of the spare part commodity of the statistics date return of goods), if belonging to, both sides' bore is consistent, and business is fed back before
Abnormal index F no problems, and feed back inform related personnel, otherwise judge that both sides' bore is inconsistent, export a bore and differ
The table inventory of cause, and this part of inventory is imported into system, system can notify each table director to carry out script according to this inventory
Repair, after having repaired, fed back.
The method of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention, is related to original based on abnormal data
Service logic relational tree carry out beta pruning obtain its corresponding logical relation tree, so as to sort out cause the abnormal data generation it is different
Tables of data set where normal direct factor, then according to the possibility that goes wrong and the complexity of investigation, from relative
For possibility it is big and the problem of investigation is easy, the data source that may be present investigated successively in the logical relation tree is asked
Topic, the imperfect problem of tables of data and bore inconsistence problems, so as to navigate to the abnormal origin of abnormal data.Pass through the present invention
The above method is provided, related personnel can it is self-service, quickly navigate to abnormal origin, and provide abnormal related tables of data letter
Breath, in order to it is follow-up the problem of handle and repair, so as to shorten party in request's stand-by period, and can in time informing business handle into
Degree.Relative to existing localization method need to carry out it is pure be positioned manually abnormal with processing data, the inventive method can be advised effectively
Keep away human error, reduce to problem investigation processor requirement so that not can only developer could handle.
The embodiment of the present invention also provides a kind of device for the abnormal origin for positioning abnormal data, as shown in figure 5, the device
500 include：Sentence duty module 501, integrity check module 502 and bore check module 503.
In the present invention, abnormal data corresponding logical relationship tree, the logical relation tree are existed by the abnormal data of the generation
The set with level that involved tables of data forms according to the logical relation for generating the data during generation, logic
The root node of relational tree is the abnormal data, and leaf node is the tables of data of data source corresponding to the abnormal data, and intermediate node is
The intermediate data table that the abnormal data is related to during producing.Logical relation tree include field processing logic between each table and
The condition of subquery, logical relation tree can be obtained by the logical relation document for the business development that the abnormal data is related to.
In the present invention, service logic original corresponding to abnormal data is included in the logical relation document of business development
Relational tree, logical relation tree be in service logic relational tree original corresponding to the abnormal data with positioning abnormal origin without
The part of pass cuts off acquisition.
Sentence duty module 501 to be used to the data of leaf node make comparisons with the data of corresponding pretreatment layer node, pretreatment layer
Node is the intermediate node that the data source of corresponding leaf node generates after pretreatment, wherein, when some pretreatment layer node and its
Corresponding leaf node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned.Sentence duty module and pass through above-mentioned comparison
Process directly can quickly judge that abnormal origin is in data source or in data warehouse or data store internal, if data
Depot data cleaning layer is consistent with production system, that is, excludes data source problem, be transferred to follow-up inside and check oneself processing.
Integrity check module 502 is used to judge whether abnormal data is more than corresponding a reference value, when abnormal data is little
In a reference value, then check the complete situation of each intermediate node in addition to pretreatment layer node, wherein, when some intermediate node not
Completely, it is determined that the intermediate node is abnormal origin and returned.
Bore checks that module 503 is used for the bore and abnormal data for checking each intermediate node in addition to pretreatment layer node
Standard gauge it is whether consistent, if the bore of some intermediate node and the bore of abnormal data are inconsistent, it is determined that the middle node
Point is abnormal origin and returns.
The device of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention also includes output module, output module
For exporting the inventory of the abnormal origin determined.
Sentence duty module 501 be further used for logic-based relational tree obtain abnormal data with its produce during be related to it is pre-
The direct mapping relations tree of node layer is handled, the corresponding leaf of the data of the pretreatment layer node in direct mapping relations tree
The data of node are made comparisons.
Bore checks that module 503 is further used for obtaining each centre in addition to pretreatment layer node according to logical relation tree
The bore inventory of node, check bore inventory in intermediate node bore and abnormal data bore it is whether consistent.
The device of the abnormal origin of positioning abnormal data provided in an embodiment of the present invention, is related to original based on abnormal data
Service logic relational tree carry out beta pruning obtain its corresponding logical relation tree, so as to sort out cause the abnormal data generation it is different
Tables of data set where normal direct factor, then according to the possibility that goes wrong and the complexity of investigation, from relative
For possibility it is big and the problem of investigation is easy, the data source that may be present investigated successively in the logical relation tree is asked
Topic, the imperfect problem of tables of data and bore inconsistence problems, so as to navigate to the abnormal origin of abnormal data.Pass through the present invention
The above method is provided, related personnel can it is self-service, quickly navigate to abnormal origin, and provide abnormal related tables of data letter
Breath, in order to it is follow-up the problem of handle and repair, so as to shorten party in request's stand-by period, and can in time informing business handle into
Degree.Relative to existing localization method need to carry out it is pure be positioned manually abnormal with processing data, the inventive method can be advised effectively
Keep away human error, reduce to problem investigation processor requirement so that not can only developer could handle.
Below with reference to Fig. 6, it illustrates suitable for for realizing the computer system X00 of the electronic equipment of the embodiment of the present invention
Structural representation.Electronic equipment shown in Fig. 6 is only an example, to the function of the embodiment of the present invention and should not use model
Shroud carrys out any restrictions.
As shown in fig. 6, computer system X00 includes CPU (CPU) X01, it can be read-only according to being stored in
Program in memory (ROM) X02 or be loaded into program in random access storage device (RAM) X03 from storage part X08 and
Perform various appropriate actions and processing.In RAM X03, various programs and data needed for system X00 operations are also stored with.
CPU X01, ROM X02 and RAM X03 are connected with each other by bus X04.Input/output (I/O) interface X05 is also connected to always
Line X04.
I/O interfaces X05 is connected to lower component：Importation X06 including keyboard, mouse etc.；Penetrated including such as negative electrode
The output par, c X07 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage part X08 including hard disk etc.；
And the communications portion X09 of the NIC including LAN card, modem etc..Communications portion X09 via such as because
The network of spy's net performs communication process.Driver X10 is also according to needing to be connected to I/O interfaces X05.Detachable media X11, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., it is arranged on as needed on driver X10, in order to read from it
Computer program be mounted into as needed storage part X08.
Especially, according to embodiment disclosed by the invention, may be implemented as counting above with reference to the process of flow chart description
Calculation machine software program.For example, embodiment disclosed by the invention includes a kind of computer program product, it includes being carried on computer
Computer program on computer-readable recording medium, the computer program include the program code for being used for the method shown in execution flow chart.
In such embodiment, the computer program can be downloaded and installed by communications portion X09 from network, and/or from can
Dismounting medium X11 is mounted.When the computer program is performed by CPU (CPU) X01, system of the invention is performed
The above-mentioned function of middle restriction.
It should be noted that the computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer-readable recording medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, system, device or the device of infrared ray or semiconductor, or it is any more than combination.Meter
The more specifically example of calculation machine readable storage medium storing program for executing can include but is not limited to：Electrical connection with one or more wires, just
Take formula computer disk, hard disk, random access storage device (RAM), read-only storage (ROM), erasable type and may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only storage (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer-readable recording medium can any include or store journey
The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.And at this
In invention, computer-readable signal media can include in a base band or as carrier wave a part propagation data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium beyond storage medium is read, the computer-readable medium, which can send, propagates or transmit, to be used for
By instruction execution system, device either device use or program in connection.Included on computer-readable medium
Program code can be transmitted with any appropriate medium, be included but is not limited to：Wirelessly, electric wire, optical cable, RF etc., or it is above-mentioned
Any appropriate combination.
Flow chart and block diagram in accompanying drawing, it is illustrated that according to the system of various embodiments of the invention, method and computer journey
Architectural framework in the cards, function and the operation of sequence product.At this point, each square frame in flow chart or block diagram can generation
The part of one module of table, program segment or code, a part for above-mentioned module, program segment or code include one or more
For realizing the executable instruction of defined logic function.It should also be noted that some as replace realization in, institute in square frame
The function of mark can also be with different from the order marked in accompanying drawing generation.For example, two square frames succeedingly represented are actual
On can perform substantially in parallel, they can also be performed in the opposite order sometimes, and this is depending on involved function.Also
It is noted that the combination of each square frame and block diagram in block diagram or flow chart or the square frame in flow chart, can use and perform rule
Fixed function or the special hardware based system of operation are realized, or can use the group of specialized hardware and computer instruction
Close to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be set within a processor, for example, can be described as：A kind of processor bag
Include and sentence duty module, integrity check module and bore inspection module.Wherein, the title of these modules not structure under certain conditions
The paired restriction of the module in itself, it is also described as " being used for the data leaf node and corresponding pre- place for example, sentencing duty module
The module that the data of reason node layer are made comparisons ".
As on the other hand, present invention also offers a kind of computer-readable medium, the computer-readable medium can be
Included in equipment described in above-described embodiment；Can also be individualism, and without be incorporated the equipment in.Above-mentioned calculating
Machine computer-readable recording medium carries one or more program, when said one or multiple programs are performed by the equipment, makes
Obtaining the equipment includes：
Abnormal data corresponding logical relationship tree, the root node of the logical relation tree is the abnormal data, and leaf node is several
According to the tables of data in source, intermediate node is the intermediate data table being related to during the abnormal data produces,
The data of the leaf node are made comparisons with the data of corresponding pretreatment layer node, the pretreatment layer node is pair
The intermediate node for answering the data source of leaf node to generate after pretreatment, wherein, when the corresponding leaf of some pretreatment layer node
Node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned；
Judge whether the abnormal data is more than corresponding a reference value, when the abnormal data is not more than a reference value,
The complete situation of each intermediate node in addition to the pretreatment layer node is then checked, wherein, when some intermediate node is imperfect,
The intermediate node is then determined as abnormal origin and is returned；
Check the bore of each intermediate node in addition to the pretreatment layer node and the standard gauge of the abnormal data
It is whether consistent, if the bore of the bore of some intermediate node and the abnormal data is inconsistent, it is determined that the intermediate node is different
Often originate from and return.
Above-mentioned embodiment, does not form limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (11)
- A kind of 1. method for the abnormal origin for positioning abnormal data, it is characterised in that the abnormal data corresponding logical relationship tree, The root node of the logical relation tree is the abnormal data, and leaf node is the tables of data of data source, and intermediate node is the abnormal number According to the intermediate data table being related to during generation,Methods described includes：Step 1, the data of the leaf node are made comparisons with the data of corresponding pretreatment layer node, the pretreatment layer node It is the intermediate node that the data source of corresponding leaf node generates after pretreatment, wherein, when some pretreatment layer node is corresponding Leaf node it is inconsistent, it is determined that the pretreatment layer node is abnormal origin and to return, and otherwise performs step 2；Step 2, judges whether the abnormal data is more than corresponding a reference value, when the abnormal data is not more than the benchmark Value, then the complete situation of each intermediate node in addition to the pretreatment layer node is checked, otherwise performs step 3, wherein, when Some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin and returned, and otherwise performs step 3；Step 3, check the bore of each intermediate node and the standard mouth of the abnormal data in addition to the pretreatment layer node Whether footpath is consistent, if the bore of the bore of some intermediate node and the abnormal data is inconsistent, it is determined that the intermediate node is Abnormal origin simultaneously returns.
- 2. according to the method for claim 1, it is characterised in that the logical relation tree is corresponding to the abnormal data The part unrelated with positioning abnormal origin in original service logic relational tree cuts off acquisition.
- 3. according to the method for claim 1, it is characterised in that also include：Export the inventory of the abnormal origin of the determination.
- 4. according to the method for claim 1, it is characterised in that data and the corresponding pretreatment layer node of the leaf node Data make comparisons including：Based on the logical relation tree obtain the abnormal data with its produce during the pretreatment layer node that is related to Direct mapping relations tree；The data of the corresponding leaf node of the data of pretreatment layer node in the directly mapping relations tree are made ratio Compared with.
- 5. according to the method for claim 1, it is characterised in that in described each in addition to the pretreatment layer node of inspection Whether the standard gauge of the bore of intermediate node and the abnormal data unanimously includes：The bore inventory of each intermediate node in addition to the pretreatment layer node is obtained according to the logical relation tree；Check whether the bore of the intermediate node in the bore inventory is consistent with the bore of the abnormal data.
- A kind of 6. device for the abnormal origin for positioning abnormal data, it is characterised in that the abnormal data corresponding logical relationship tree, The root node of the logical relation tree is the abnormal data, and leaf node is the tables of data of data source, and intermediate node is the abnormal number According to the intermediate data table being related to during generation,Described device includes：Sentence duty module, for step 1, the data of the leaf node are made comparisons with the data of corresponding pretreatment layer node, it is described Pretreatment layer node is the intermediate node that the data source of corresponding leaf node generates after pretreatment, wherein, when some pretreatment layer The corresponding leaf node of node is inconsistent, it is determined that the pretreatment layer node is abnormal origin and returned, and otherwise performs step Two；Integrity check module, for step 2, judge whether the abnormal data is more than corresponding a reference value, when the exception Data are not more than a reference value, then check the complete situation of each intermediate node in addition to the pretreatment layer node, otherwise Step 3 is performed, wherein, when some intermediate node is imperfect, it is determined that the intermediate node is abnormal origin and returned, and is otherwise held Row step 3；Bore checks module, for step 3, checks bore and the institute of each intermediate node in addition to the pretreatment layer node Whether consistent state the standard gauge of abnormal data, if the bore of the bore of some intermediate node and the abnormal data is inconsistent, The intermediate node is then determined as abnormal origin and is returned.
- 7. device according to claim 6, it is characterised in that also include：Output module, the inventory of the abnormal origin for exporting the determination.
- 8. device according to claim 6, it is characterised in that it is described sentence duty module be further used for based on the logic close System tree obtain the abnormal data with its produce during the direct mapping relations tree of the pretreatment layer node that is related to, institute The data for stating the corresponding leaf node of data of the pretreatment layer node in direct mapping relations tree are made comparisons.
- 9. device according to claim 6, it is characterised in that the bore checks that module is further used for patrolling according to The bore inventory that relational tree obtains each intermediate node in addition to the pretreatment layer node is collected, is checked in the bore inventory Whether the bore of intermediate node is consistent with the bore of the abnormal data.
- A kind of 10. electronic equipment for judging data exception reason, it is characterised in that including：One or more processors；Storage device, for storing one or more programs,When one or more of programs are by one or more of computing devices so that one or more of processors are real The now method as described in any in claim 1-5.
- 11. a kind of computer-readable medium, is stored thereon with computer program, it is characterised in that described program is held by processor The method as described in any in claim 1-5 is realized during row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710722887.9A CN107643956B (en) | 2017-08-22 | 2017-08-22 | Method and apparatus for locating the origin of an anomaly in anomaly data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710722887.9A CN107643956B (en) | 2017-08-22 | 2017-08-22 | Method and apparatus for locating the origin of an anomaly in anomaly data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107643956A true CN107643956A (en) | 2018-01-30 |
| CN107643956B CN107643956B (en) | 2020-09-01 |
Family
ID=61110186
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710722887.9A Active CN107643956B (en) | 2017-08-22 | 2017-08-22 | Method and apparatus for locating the origin of an anomaly in anomaly data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107643956B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108429636A (en) * | 2018-02-01 | 2018-08-21 | 阿里巴巴集团控股有限公司 | Position the method and device and electronic equipment of pathological system |
| CN109144884A (en) * | 2018-09-29 | 2019-01-04 | 平安科技（深圳）有限公司 | Program error localization method, device and computer readable storage medium |
| CN109254986A (en) * | 2018-08-31 | 2019-01-22 | 阿里巴巴集团控股有限公司 | A kind of determination method and device of abnormal data |
| CN110471962A (en) * | 2019-07-05 | 2019-11-19 | 中国平安人寿保险股份有限公司 | The generation method and system of alive data report |
| CN111367775A (en) * | 2018-12-26 | 2020-07-03 | 北京嘀嘀无限科技发展有限公司 | Problem node positioning method, computer device and computer-readable storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101261602A (en) * | 2008-04-08 | 2008-09-10 | 杭州电子科技大学 | Program correctness verification method based on syntax tree |
| CN102117306A (en) * | 2010-01-04 | 2011-07-06 | 阿里巴巴集团控股有限公司 | Method and system for monitoring ETL (extract-transform-load) data processing process |
| US20120030165A1 (en) * | 2010-07-29 | 2012-02-02 | Oracle International Corporation | System and method for real-time transactional data obfuscation |
| CN102650992A (en) * | 2011-02-25 | 2012-08-29 | 国际商业机器公司 | Method and device for generating binary XML (extensible markup language) data and locating nodes of the binary XML data |
| US20140232725A1 (en) * | 2011-10-26 | 2014-08-21 | Fujifilm Corporation | Image processing apparatus, image processing method, and image processing program |
| CN105302657A (en) * | 2015-11-05 | 2016-02-03 | 网易宝有限公司 | Abnormal condition analysis method and apparatus |
| WO2016093937A1 (en) * | 2014-12-09 | 2016-06-16 | Hitachi Data Systems Corporation | Elastic metadata and multiple tray allocation |
| CN105760383A (en) * | 2014-12-16 | 2016-07-13 | 阿里巴巴集团控股有限公司 | Method and device for detecting index alteration in ETL (extract-transform-load) task |
| CN105897922A (en) * | 2016-05-30 | 2016-08-24 | 乐视控股（北京）有限公司 | Data transmission method and device |
| CN106709024A (en) * | 2016-12-28 | 2017-05-24 | 深圳市华傲数据技术有限公司 | Data table source-tracing method and device based on consanguinity analysis |
| CN106802931A (en) * | 2016-12-28 | 2017-06-06 | 深圳市华傲数据技术有限公司 | The method and device of data table search is carried out based on impact analysis |
| CN106951315A (en) * | 2017-03-17 | 2017-07-14 | 北京搜狐新媒体信息技术有限公司 | A kind of data task dispatching method and system based on ETL |
-
2017
- 2017-08-22 CN CN201710722887.9A patent/CN107643956B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101261602A (en) * | 2008-04-08 | 2008-09-10 | 杭州电子科技大学 | Program correctness verification method based on syntax tree |
| CN102117306A (en) * | 2010-01-04 | 2011-07-06 | 阿里巴巴集团控股有限公司 | Method and system for monitoring ETL (extract-transform-load) data processing process |
| US20120030165A1 (en) * | 2010-07-29 | 2012-02-02 | Oracle International Corporation | System and method for real-time transactional data obfuscation |
| CN102650992A (en) * | 2011-02-25 | 2012-08-29 | 国际商业机器公司 | Method and device for generating binary XML (extensible markup language) data and locating nodes of the binary XML data |
| US20140232725A1 (en) * | 2011-10-26 | 2014-08-21 | Fujifilm Corporation | Image processing apparatus, image processing method, and image processing program |
| WO2016093937A1 (en) * | 2014-12-09 | 2016-06-16 | Hitachi Data Systems Corporation | Elastic metadata and multiple tray allocation |
| CN105760383A (en) * | 2014-12-16 | 2016-07-13 | 阿里巴巴集团控股有限公司 | Method and device for detecting index alteration in ETL (extract-transform-load) task |
| CN105302657A (en) * | 2015-11-05 | 2016-02-03 | 网易宝有限公司 | Abnormal condition analysis method and apparatus |
| CN105897922A (en) * | 2016-05-30 | 2016-08-24 | 乐视控股（北京）有限公司 | Data transmission method and device |
| CN106709024A (en) * | 2016-12-28 | 2017-05-24 | 深圳市华傲数据技术有限公司 | Data table source-tracing method and device based on consanguinity analysis |
| CN106802931A (en) * | 2016-12-28 | 2017-06-06 | 深圳市华傲数据技术有限公司 | The method and device of data table search is carried out based on impact analysis |
| CN106951315A (en) * | 2017-03-17 | 2017-07-14 | 北京搜狐新媒体信息技术有限公司 | A kind of data task dispatching method and system based on ETL |
Non-Patent Citations (1)
| Title |
|---|
| 王丽珍等: "基于数据仓库的动态异常点检测研究", 《计算机研究与发展》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108429636A (en) * | 2018-02-01 | 2018-08-21 | 阿里巴巴集团控股有限公司 | Position the method and device and electronic equipment of pathological system |
| CN108429636B (en) * | 2018-02-01 | 2021-11-23 | 创新先进技术有限公司 | Method and device for positioning abnormal system and electronic equipment |
| CN109254986A (en) * | 2018-08-31 | 2019-01-22 | 阿里巴巴集团控股有限公司 | A kind of determination method and device of abnormal data |
| CN109144884A (en) * | 2018-09-29 | 2019-01-04 | 平安科技（深圳）有限公司 | Program error localization method, device and computer readable storage medium |
| CN111367775A (en) * | 2018-12-26 | 2020-07-03 | 北京嘀嘀无限科技发展有限公司 | Problem node positioning method, computer device and computer-readable storage medium |
| CN111367775B (en) * | 2018-12-26 | 2023-11-14 | 北京嘀嘀无限科技发展有限公司 | Problem node positioning method, computer device, and computer-readable storage medium |
| CN110471962A (en) * | 2019-07-05 | 2019-11-19 | 中国平安人寿保险股份有限公司 | The generation method and system of alive data report |
| CN110471962B (en) * | 2019-07-05 | 2023-11-03 | 中国平安人寿保险股份有限公司 | Method and system for generating active data report |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107643956B (en) | 2020-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107643956A (en) | The method and apparatus for positioning the abnormal origin of abnormal data | |
| WO2021052031A1 (en) | Statistical interquartile range-based commodity inventory risk early warning method and system, and computer readable storage medium | |
| WO2020220810A1 (en) | Data fusion method and apparatus | |
| JP6707564B2 (en) | Data quality analysis | |
| JP6066927B2 (en) | Generation of data pattern information | |
| WO2019212857A1 (en) | Systems and methods for enriching modeling tools and infrastructure with semantics | |
| CN110287316A (en) | A kind of Alarm Classification method, apparatus, electronic equipment and storage medium | |
| CN110990529B (en) | Industry detail dividing method and system for enterprises | |
| CN112348521A (en) | Intelligent risk quality inspection method and system based on business audit and electronic equipment | |
| US20150302420A1 (en) | Compliance framework for providing regulatory compliance check as a service | |
| CN112598513A (en) | Method and device for identifying shareholder risk transaction behavior | |
| CN111695979A (en) | Method, device and equipment for analyzing relation between raw material and finished product | |
| CN113111095B (en) | Intelligent information management method and system | |
| CN109947797B (en) | Data inspection device and method | |
| CN114372892A (en) | Payment data monitoring method, device, equipment and medium | |
| CN113450208A (en) | Loan risk change early warning and model training method and device | |
| CN113780986A (en) | Measurement method, system, equipment and medium for software development process | |
| CN107679096A (en) | The shared method and apparatus of index between Data Mart | |
| CN112053217A (en) | Financial valuation statement generation method and device | |
| CN111612302A (en) | Group-level data management method and equipment | |
| CN112825165A (en) | Project quality management method and device | |
| CN112561368B (en) | Visual performance calculation method and device for OA approval system | |
| US20230252008A1 (en) | Systems and methods for data verification | |
| CN115392805B (en) | Transaction type contract compliance risk diagnosis method and system | |
| US20230342281A1 (en) | Branching data monitoring watchpoints to enable continuous integration and continuous delivery of data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |