CN113760882B - Data abnormity alarm method and device, electronic equipment and computer readable medium - Google Patents

Data abnormity alarm method and device, electronic equipment and computer readable medium Download PDF

Info

Publication number
CN113760882B
CN113760882B CN202111323641.7A CN202111323641A CN113760882B CN 113760882 B CN113760882 B CN 113760882B CN 202111323641 A CN202111323641 A CN 202111323641A CN 113760882 B CN113760882 B CN 113760882B
Authority
CN
China
Prior art keywords
abnormal
information table
article information
data
consistency
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111323641.7A
Other languages
Chinese (zh)
Other versions
CN113760882A (en
Inventor
黄颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Foshan Huayue Intellectual Property Operation Co ltd
Original Assignee
Beijing Daily Vegetable Market Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Daily Vegetable Market Technology Co ltd filed Critical Beijing Daily Vegetable Market Technology Co ltd
Priority to CN202111323641.7A priority Critical patent/CN113760882B/en
Publication of CN113760882A publication Critical patent/CN113760882A/en
Application granted granted Critical
Publication of CN113760882B publication Critical patent/CN113760882B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2365Ensuring data consistency and integrity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/283Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Quality & Reliability (AREA)
  • Alarm Systems (AREA)

Abstract

The embodiment of the disclosure discloses a data abnormity alarm method, a data abnormity alarm device, electronic equipment and a computer readable medium. One embodiment of the method comprises: acquiring an article information table set; responding to the article information table meeting the data integrity abnormal condition in the article information table set, and generating a data integrity abnormal table; generating a data consistency abnormal table; generating a rate value exception table; respectively generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information; and sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a user terminal. According to the embodiment, the comprehensive monitoring on the article information table set is realized, and the integrity, consistency and accuracy of data in the article information table set are improved, so that the reliability of the data is improved.

Description

Data abnormity alarm method and device, electronic equipment and computer readable medium
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a data abnormity alarm method, a data abnormity alarm device, electronic equipment and a computer readable medium.
Background
With the development of internet technology, more and more fresh electric merchants for selling fresh products emerge. The order table, the logistics table and other related data of the fresh products are taken as the item information table to be periodically synchronized from the fresh electric supplier to the item information table set included in the data warehouse. In order to find out the abnormality in time, the item information table set included in the data warehouse needs to be monitored. The prior art does not have multi-orientation monitoring of a collection of item information tables included in a data warehouse.
However, when the prior art is adopted to monitor the item information table set included in the data warehouse, the following technical problems often exist:
firstly, multi-directional monitoring on each article information table cannot be realized, and the integrity, consistency and accuracy of data in the article information tables are poor, so that the reliability of the data is low;
secondly, when the data types of the data in the article information table inquired through different inquiry modes are inconsistent, the consistency of the data in the article information table is poor, and the reliability of the data is low.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose a data anomaly alarm method, apparatus, electronic device and computer readable medium to solve one or more of the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a data anomaly alarm method, including: acquiring an article information table set; responding to the article information table meeting the data integrity abnormal condition in the article information table set, and generating a data integrity abnormal table according to the article information table meeting the data integrity abnormal condition in the article information table set; responding to the article information table meeting the data consistency abnormal condition in the article information table set, and generating a data consistency abnormal table according to the article information table meeting the data consistency abnormal condition in the article information table set; responding to an article information table meeting a rate value abnormal condition in an article information table set, and generating a rate value abnormal table according to the article information table meeting the rate value abnormal condition in the article information table set; generating integrity abnormal alarm information, consistency abnormal alarm information and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate abnormal table; and sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to the associated user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
In a second aspect, some embodiments of the present disclosure provide a data anomaly alarm device, including: an acquisition unit configured to acquire a set of item information tables; a first generating unit configured to generate a data integrity abnormal table according to the item information table satisfying the data integrity abnormal condition in the item information table set in response to the item information table satisfying the data integrity abnormal condition existing in the item information table set; a second generation unit configured to generate a data consistency abnormal table from the item information tables satisfying the data consistency abnormal condition in the item information table set in response to the item information tables satisfying the data consistency abnormal condition existing in the item information table set; a third generating unit configured to generate a rate value exception table according to the item information table satisfying the rate value exception condition in the item information table set in response to the item information table satisfying the rate value exception condition existing in the item information table set; the fourth generation unit is configured to generate integrity abnormal alarm information, consistency abnormal alarm information and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate abnormal table respectively; and the sending unit is configured to send the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to the associated user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first aspect.
In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium on which a computer program is stored, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect.
The above embodiments of the present disclosure have the following advantages: according to the data abnormity alarm method of some embodiments of the disclosure, the integrity, consistency and accuracy of data in the article information table set are improved, and the reliability of the data is improved. Specifically, the reason why the integrity, consistency and accuracy of data in the item information table set cannot be improved is that: and multi-directional monitoring on the article information table set cannot be realized. Based on this, the data anomaly alarm method of some embodiments of the present disclosure first obtains an item information table set. And then, responding to the article information tables meeting the data integrity abnormal conditions in the article information table set, and generating a data integrity abnormal table according to the article information tables meeting the data integrity abnormal conditions in the article information table set. Therefore, the obtained data integrity abnormal table records the related information of the article information table meeting the data integrity abnormal condition in the article information table set. And secondly, in response to the article information tables meeting the data consistency abnormal condition in the article information table set, generating a data consistency abnormal table according to the article information tables meeting the data consistency abnormal condition in the article information table set. In this way, the obtained data consistency abnormality table records the related information of the article information table satisfying the data consistency abnormality condition in the article information table set. And then, responding to the article information tables meeting the abnormal condition of the rate value in the article information table set, and generating an abnormal table of the rate value according to the article information tables meeting the abnormal condition of the rate value in the article information table set. In this way, the obtained value abnormality table records information related to the article information table satisfying the value abnormality condition in the article information table set. And then, generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate value abnormal table. Therefore, integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information can be obtained, and the article information table of related workers is prompted to have abnormal data. And finally, sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a related user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information. Therefore, related workers can be informed of the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information. And the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information, so that related workers can know the article information table needing to be modified, and can modify data with integrity abnormality, consistency abnormality or rate value abnormality in the modified article information table. Therefore, the integrity, consistency and accuracy of the data in the article information table are improved, and the reliability of the data is improved.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of an application scenario of a data anomaly alerting method of some embodiments of the present disclosure;
FIG. 2 is a flow diagram of some embodiments of a data anomaly alerting method according to the present disclosure;
FIG. 3 is a flow diagram of further embodiments of a data anomaly alert method according to the present disclosure;
FIG. 4 is a schematic block diagram of some embodiments of a data anomaly alarm device according to the present disclosure;
FIG. 5 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic diagram of an application scenario of a data anomaly alerting method according to some embodiments of the present disclosure.
In the application scenario of fig. 1, first, the computing device 101 may obtain a set of item information tables 102. Then, in response to the existence of an item information table satisfying the data integrity exception condition in the item information table set 102, the computing device 101 may generate the data integrity exception table 103 from the item information tables satisfying the data integrity exception condition in the item information table set 102. Thereafter, the computing device 101 may generate the data consistency exception table 104 from the item information tables in the set of item information tables 102 that satisfy the data consistency exception condition in response to the presence of an item information table in the set of item information tables 102 that satisfies the data consistency exception condition. Next, the computing device 101 may generate a rate value abnormality table 105 from the item information tables in the item information table set 102 that satisfy the rate value abnormality condition in response to the presence of an item information table in the item information table set 102 that satisfies the rate value abnormality condition. Then, computing device 101 may generate integrity abnormality alert information 106, consistency abnormality alert information 107, and rate value abnormality alert information 108 from data integrity abnormality table 103, data consistency abnormality table 104, and rate value abnormality table 105, respectively. Finally, computing device 101 may send integrity abnormality alert information 106, consistency abnormality alert information 107, and rate value abnormality alert information 108 to an associated user terminal 109, such that user terminal 109 presents integrity abnormality alert information 106, consistency abnormality alert information 107, and rate value abnormality alert information 108.
The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster composed of multiple servers or terminal devices, or may be implemented as a single server or a single terminal device. When the computing device is embodied as software, it may be installed in the hardware devices enumerated above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.
It should be understood that the number of computing devices in FIG. 1 is merely illustrative. There may be any number of computing devices, as implementation needs dictate.
With continued reference to fig. 2, a flow 200 of some embodiments of a data anomaly alerting method according to the present disclosure is shown. The data abnormity warning method comprises the following steps:
step 201, an item information table set is obtained.
In some embodiments, the executing entity (e.g., the computing device 101 shown in fig. 1) of the data anomaly alarm method may obtain the item information table set from the terminal through a wired connection or a wireless connection. The item information table in the item information table set may be a list including item information. The item information table may include, but is not limited to: a table field name set, each table field value corresponding to each table field name in the table field name set, a miscellaneous dimension name, and each miscellaneous dimension value corresponding to the miscellaneous dimension name. The table name may be a table name of the article information table. The set of table field names may be a set of table field names of respective table fields included in the item information table. The miscellaneous dimension name may be a name of a miscellaneous dimension included in the article information table. The miscellaneous dimension may be a specific table field included in the item information table. The miscellaneous dimension values may be table field values corresponding to the miscellaneous dimension names. For example, the item information table may be an item flow table or an order table. The logistics table may be a table containing logistics information of the articles. The article logistics information may include, but is not limited to: item identification, item weight, item volume, shipping address. The item identifier may uniquely identify the item. The order table may be a table containing item order information. The item order information may include, but is not limited to: item identification, order number, item quantity, item unit value, and order total value. The order number may uniquely identify the order. The unit value of an item may be a value (e.g., unit price) paid to obtain a single item. The order total value may be a total value required to pay the order (e.g., an order total). It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
In some optional implementation manners of some embodiments, in a first step, the executing body may generate an item information virtual table based on each original item information table in a set of original item information tables stored in advance, so as to obtain a set of item information virtual tables. Wherein the item information virtual table includes metadata information of the original item information table. The metadata information may include, but is not limited to: table name, field name and data type of each table field, name of person responsible for table, dimension name of miscellaneous item, and dimension value of miscellaneous item corresponding to dimension of miscellaneous item. In practice, first, the execution body may create an item information virtual empty table. Then, the metadata information of the original article information table may be extracted from a terminal in which the original article information table set is stored in advance, and the metadata information may be written in the article information virtual empty table to obtain an article information virtual table. In the second step, the executing agent may obtain table state information of each item information virtual table in the item information virtual table set. The table state information may be information indicating a table state of the article information virtual table. The above table states may include, but are not limited to: a waiting-to-go state, a going-on state, a rejected state, an approval state and a going-off state. And thirdly, the executing body may use an original article information table corresponding to the article information virtual table in which the corresponding table state information representation is in an online state in the article information virtual table set as the article information table to obtain the article information table set.
Step 202, in response to the article information table meeting the data integrity abnormal condition existing in the article information table set, generating a data integrity abnormal table according to the article information table meeting the data integrity abnormal condition in the article information table set.
In some embodiments, the execution subject may generate the data integrity exception table according to the item information table satisfying the data integrity exception condition in the item information table set in response to the item information table satisfying the data integrity exception condition existing in the item information table set. The data integrity abnormal condition may be "a table field value in the article information table is 0". In practice, first, the execution main body may select an article information table satisfying the data integrity abnormality condition from the article information table set as an integrity candidate article information table, and obtain an integrity candidate article information table set. A data integrity exception empty table may then be created. Then, the table name, the table principal name, and the table field name with the table field value of 0 of each integrity candidate item information table in the integrity candidate item information table set may be written into the data integrity abnormal empty table to obtain the data integrity abnormal table. Therefore, the obtained data integrity abnormal table records the related information of the article information table meeting the data integrity abnormal condition in the article information table set.
In some optional implementation manners of some embodiments, in a first step, the execution main body may generate the first data integrity exception table according to an item information table satisfying the first data integrity exception condition in response to an item information table whose table field value satisfies the first data integrity exception condition exists in the item information table set. Wherein the first data integrity exception condition may characterize a null value for the table field value. It is understood that the table field value is a NULL value or a NULL value. In practice, first, the execution subject may select an article information table satisfying the first data integrity abnormality condition from the article information table set as a first candidate article information table, to obtain a first candidate article information table set. A first data integrity exception empty table may then be created. Then, the table name, the table principal name, and the table field name whose table field value is NULL value or NULL value of each first candidate item information table in the first candidate item information table set may be written into the first data integrity abnormality NULL table, so as to obtain a first data integrity abnormality table. The first data integrity abnormality table thus obtained records information related to each item information table having a table field value of NULL or NULL.
And secondly, the executive body can acquire the number of the table fields corresponding to each article information table in the article information table set. The number of the table fields may be the number of the table fields included in the article information table. In practice, the execution subject may obtain, by using the Presto engine, the number of table fields corresponding to each item information table in the item information table set.
And thirdly, the execution main body can respond to the article information table with the corresponding table field number meeting the second data integrity abnormal condition in the article information table set and generate a second data integrity abnormal table according to the article information table meeting the second data integrity abnormal condition. The second data integrity abnormal condition may be that the number of table fields is different from the number of table field names included in the item information virtual table corresponding to the item information table. In practice, first, the execution subject may select an item information table satisfying the second data integrity abnormality condition from the item information table set as a second candidate item information table, to obtain a second candidate item information table set. A second data integrity exception empty table may then be created. Then, the table name and the table principal name of each second candidate item information table in the second candidate item information table set may be written into the second data integrity abnormality empty table to obtain a second data integrity abnormality table. The second data integrity abnormality table thus obtained records information related to the article information table in which the number of field names is abnormal.
Fourthly, the execution main body can obtain a miscellaneous item dimension value set corresponding to each item information table in the item information table set. The miscellaneous dimension value may be a miscellaneous dimension value corresponding to the miscellaneous dimension. The miscellaneous dimension may be a specific field. In practice, the execution subject may obtain, by using the Presto engine, a miscellaneous dimension value set corresponding to each item information table in the item information table set.
And fifthly, the execution main body can respond to the article information table in which the corresponding de-duplicated miscellaneous dimension value set meets the third data integrity abnormal condition in the article information table set, and generate a third data integrity abnormal table according to the article information table meeting the third data integrity abnormal condition. The third data integrity abnormal condition may be that the miscellaneous item dimension value included in the duplicate-removed miscellaneous item dimension value set does not exist in the miscellaneous item dimension values included in the article information virtual table corresponding to the article information table. In practice, first, the execution subject may select an article information table satisfying the third data integrity abnormality condition from the article information table set as a third candidate article information table, to obtain a third candidate article information table set. A third data integrity exception empty table may then be created. Then, the table name, the table principal name, and the miscellaneous dimension name of each third candidate item information table in the third candidate item information table set may be written into the third data integrity abnormality empty table to obtain a third data integrity abnormality table. Therefore, the obtained third data integrity abnormal table records the related information of the article information table with abnormal miscellaneous dimension values.
Step 203, responding to the article information tables meeting the abnormal condition of data consistency in the article information table set, and generating a data consistency abnormal table according to the article information tables meeting the abnormal condition of data consistency in the article information table set.
In some embodiments, the execution subject may generate the data consistency exception table according to the item information table satisfying the data consistency exception condition in the item information table set in response to the item information table satisfying the data consistency exception condition in the item information table set. The data consistency exception condition may be that the same tuple exists in the item information table. The tuple may be a row included in the item information table. In practice, first, the execution main body may select an article information table satisfying the data consistency abnormality condition from the article information table set as a consistency candidate article information table, and obtain a consistency candidate article information table set. A data consistency exception null table may then be created. Then, the table name and the table principal name of each consistency candidate item information table in the consistency candidate item information table set may be written into the data consistency abnormality empty table to obtain a data consistency abnormality table. In this way, the obtained data consistency abnormality table records the related information of the article information table satisfying the data consistency abnormality condition in the article information table set.
In some optional implementation manners of some embodiments, in a first step, the execution subject may generate, in response to an item information table in which a data type of a table field included in the item information table set satisfies a first data consistency exception condition, a first data consistency exception table according to the item information table satisfying the first data consistency exception condition. The first data consistency abnormal condition may be that the data type of the table field is different from the data type of the target table field included in the item information virtual table corresponding to the item information table. The data types of the table fields may include, but are not limited to: int (integer), float (floating point). The object table field may be a table field in the virtual table of item information corresponding to a table field included in the item information table. In practice, first, the execution main body may select an article information table satisfying the first data consistency abnormality condition from the article information table set as a fourth candidate article information table, and obtain a fourth candidate article information table set. A first data consistency exception null table may then be created. Then, the table name, the table administrator name, and the table field name of the table field corresponding to the target table field included in each fourth candidate item information table in the fourth candidate item information table set may be written into the first data consistency abnormality empty table, so as to obtain a first data consistency abnormality table. In this way, the obtained first data consistency abnormality table records the related information of the item information table in which the data type of the field is different from the data type of the corresponding target table field included in the item information virtual table.
And secondly, the execution main body can respond to the article information tables meeting the prerequisite in the article information table set, and determine the article information tables meeting the prerequisite in the article information table set as the first article information table to obtain the first article information table set. The prerequisite may be that at least two item information tables of the same type exist in the item information table set and are stored in the same database. The above-mentioned type may be a type of an item information table. The above types may include, but are not limited to, an offline type and a real-time type. The offline type item information table may be an item information table of a historical time period. The real-time type item information table may be an item information table updated in real time. In this way, a first item information table set can be obtained such that, for any one of the first item information tables in the first item information table set, the same first item information table as the type and corresponding database of the any one of the first item information tables exists.
And thirdly, determining each first article information table meeting the preset index field condition in the first article information table set as a second article information table set to obtain a second article information table set. The preset index field condition is that the same index field exists in each first article information table, and each first article information table is related through a foreign key included in the first article information table. The index field may be a field whose data type is int (integer), binary, or decimal. Therefore, each second item information table included in each second item information table group included in the obtained second item information table group set contains the same index field.
And fourthly, for each second item information table set in the second item information table set, determining an aggregation value corresponding to the index field of the corresponding second item information table set included in each second item information table in the second item information table set to obtain an aggregation value set. In practice, the execution subject may determine a sum of table field values corresponding to the index field in each second item information table as an aggregate value, so as to obtain an aggregate value set corresponding to the index field. Thus, a set of aggregate values can be derived to provide data support for subsequent processing.
In the fifth step, the execution subject may determine, as an abnormal aggregate value group, an aggregate value group satisfying the second data consistency abnormality condition from among the obtained aggregate value groups, to obtain an abnormal aggregate value group set. Wherein the second data consistency exception condition is that the aggregate value is different from any of the set of aggregate values. Thus, an abnormal aggregate value group set characterizing the existence of an aggregate value anomaly may be obtained.
And sixthly, the execution main body can generate a second data consistency abnormal table according to each second article information table group corresponding to the abnormal aggregation value group set. In practice, first, the execution subject may select an article information table satisfying the second data consistency abnormality condition from the article information table set as a fifth candidate article information table, and obtain a fifth candidate article information table set. A second data consistency exception null table may then be created. Then, the table name of each fifth candidate item information table in the fifth candidate item information table set, the table leader name, and the table field name of the index field corresponding to the different aggregation value may be written into the second data consistency abnormality empty table, so as to obtain a second data consistency abnormality table. Thus, the second data consistency abnormality table can be obtained, so that the obtained second data consistency abnormality table records the related information of the article information table with the aggregation value abnormality.
And seventhly, responding to the article information table with the tuple meeting the third data consistency abnormal condition in the article information table set, and generating a third data consistency abnormal table according to the article information table meeting the third data consistency abnormal condition. And the third data consistency abnormal condition is that the same tuple exists in the article information table. In practice, first, the execution subject may select an article information table satisfying the third data consistency abnormality condition from the article information table set as a sixth candidate article information table, and obtain a sixth candidate article information table set. A third data consistency exception null table may then be created. Then, the table name and the table administrator name of each sixth candidate item information table in the sixth candidate item information table set may be written into the third data consistency abnormality empty table to obtain a third data consistency abnormality table. Thus, the third data consistency abnormality table can be obtained, so that the obtained third data consistency abnormality table records the related information of the article information table with tuple abnormality.
In some optional implementation manners of some embodiments, first, the executing entity may delete a white list field included in each item information table in the item information table set from the item information table, and determine the item information table after deletion processing as a target item information table, so as to obtain a target item information table set. The article information table included in the article information table set is an article information table containing a white list field. The white list field may be a field that characterizes no monitoring required. Then, in response to the existence of the target item information table satisfying the data integrity abnormal condition in the target item information table set, a data integrity abnormal table may be generated according to the target item information table satisfying the data integrity abnormal condition in the target item information table set. Therefore, the detection and monitoring of the white list field can be avoided, and the monitoring efficiency is improved.
And step 204, responding to the article information tables meeting the abnormal condition of the rate value in the article information table set, and generating an abnormal table of the rate value according to the article information tables meeting the abnormal condition of the rate value in the article information table set.
In some embodiments, the executing agent may generate the rate value exception table according to the item information table satisfying the rate value exception condition in the item information table set in response to the item information table satisfying the rate value exception condition existing in the item information table set. The above-mentioned rate value abnormal condition may be that a rate value greater than 1 exists in the article information table. The value may be a percentage of the value type in the item information table. In practice, the execution subject may select an article information table satisfying the above-mentioned abnormal condition of the probability value from the article information table set as a candidate article information table of the probability value, and obtain a candidate article information table set of the probability value. Then, a rate value exception empty table may be created. Then, the table name of each rate value candidate item information table in the rate value candidate item information table set, the table leader name and the table field name with the rate value larger than 1 may be written into the rate value abnormal empty table to obtain the rate value abnormal table. In this way, the obtained value abnormality table records information related to the article information table satisfying the value abnormality condition in the article information table set.
And step 205, generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate value abnormal table.
In some embodiments, the execution main body may generate integrity abnormal alarm information, consistency abnormal alarm information, and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table, and the rate abnormal table, respectively. In practice, the execution main body may store the data integrity exception table as a data integrity exception table image, and use the data integrity exception table image as integrity exception alarm information. The data consistency abnormality table may be saved as a data consistency abnormality table image, and the data consistency abnormality table image may be used as consistency abnormality alarm information. The execution body may store the rate value abnormality table as a rate value abnormality table image, and use the rate value abnormality table image as rate value abnormality warning information. Therefore, integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information can be obtained, and the article information table of related workers is prompted to have abnormal data.
In some optional implementation manners of some embodiments, in a first step, the execution main body may generate a data integrity exception table image and a data integrity exception message reminding number according to the data integrity exception table. In practice, first, the execution subject may save the data integrity exception table as a data integrity exception table image. Then, the number of tuples included in the data integrity exception table may be queried, and the number of tuples included in the data integrity exception table may be used as a data integrity exception message reminding number. And finally, classifying the data integrity abnormal message reminding number according to the name of the table leader to obtain at least one type of data integrity abnormal message reminding number, so that each type of data integrity abnormal message reminding number corresponds to one table leader name. Therefore, the data integrity exception form image and the data integrity exception message reminding number can be obtained so as to provide data support for subsequent processing.
And secondly, combining the data integrity abnormal form image, the data integrity abnormal message reminding number and the information of the responsible person associated with the data integrity abnormal form to obtain integrity abnormal alarm information. The responsible person information may include a list responsible person name and a name of a person on duty on the day of the data integrity exception table. In practice, the execution main body may combine the data integrity abnormal table image, the data integrity abnormal message reminding number of each type and the information of the responsible person corresponding to the data integrity abnormal message reminding number to obtain integrity abnormal alarm information. Therefore, the integrity abnormal alarm information can be obtained, and the content included by the integrity abnormal alarm information is more detailed.
And thirdly, generating a data consistency abnormal form image and a data consistency abnormal message reminding number according to the data consistency abnormal form. In practice, first, the execution subject may save the data consistency exception table as a data consistency exception table image. Then, the number of tuples included in the data consistency exception table may be queried, and the number of tuples included in the data consistency exception table may be used as a data consistency exception message reminding number. And finally, classifying the data consistency abnormal message reminding number according to the name of the table leader to obtain at least one type of data consistency abnormal message reminding number, so that each type of data consistency abnormal message reminding number corresponds to one table leader name. Therefore, the data consistency exception form image and the data consistency exception message reminding number can be obtained so as to provide data support for subsequent processing.
And fourthly, combining the data consistency abnormal form image, the data consistency abnormal message reminding number and the information of the responsible person associated with the data consistency abnormal form to obtain consistency abnormal alarm information. The person in charge information may include a name of a person in charge and a name of a person in charge on duty on the day of the data consistency exception table. In practice, the execution main body may combine the data consistency abnormal form image, the data consistency abnormal message reminding number of each type, and the information of the responsible person corresponding to the data consistency abnormal message reminding number to obtain the consistency abnormal alarm information. Therefore, consistency abnormal alarm information can be obtained, and the content included by the consistency abnormal alarm information is more detailed.
And fifthly, generating a rate value abnormal table image and a rate value abnormal message reminding number according to the rate value abnormal table. In practice, first, the execution subject may save the rate value exception table as a rate value exception table image. Then, the number of tuples included in the rate value exception table may be queried, and the number of tuples included in the rate value exception table may be used as a rate value exception message reminding number. Finally, the rate value abnormal message reminding number can be classified according to the name of the person in charge of the table to obtain at least one type of rate value abnormal message reminding number, so that each type of rate value abnormal message reminding number corresponds to one name of the person in charge of the table. Therefore, the rate value exception table image and the rate value exception message reminding number can be obtained so as to provide data support for subsequent processing.
And sixthly, combining the rate value abnormal table image, the rate value abnormal message reminding number and the information of the responsible person associated with the rate value abnormal table to obtain rate value abnormal alarm information. The person in charge information may include a table person in charge name and a person in charge name on duty on the day of the above-mentioned rate value abnormality table. In practice, the execution main body may combine the rate value abnormal table image, each type of rate value abnormal message reminding number, and the information of the responsible person corresponding to the rate value abnormal message reminding number to obtain the rate value abnormal alarm information. Therefore, the value abnormal alarm information can be obtained, and the content included by the value abnormal alarm information is more detailed.
And step 206, sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a related user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
In some embodiments, the execution main body may send the integrity abnormal alarm information, the consistency abnormal alarm information, and the rate value abnormal alarm information to an associated user terminal in a wired connection manner or a wireless connection manner, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information, and the rate value abnormal alarm information. The associated user terminal may be a user terminal capable of displaying the integrity abnormality alarm information, the consistency abnormality alarm information, and the rate abnormality alarm information. The user terminal can be a smart phone of a related worker. Therefore, related workers can be informed of the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
The above embodiments of the present disclosure have the following advantages: according to the data abnormity alarm method of some embodiments of the disclosure, the integrity, consistency and accuracy of data in the article information table set are improved, and the reliability of the data is improved. Specifically, the reason why the integrity, consistency and accuracy of data in the item information table set cannot be improved is that: and multi-directional monitoring on the article information table set cannot be realized. Based on this, the data anomaly alarm method of some embodiments of the present disclosure first obtains an item information table set. And then, responding to the article information tables meeting the data integrity abnormal conditions in the article information table set, and generating a data integrity abnormal table according to the article information tables meeting the data integrity abnormal conditions in the article information table set. Therefore, the obtained data integrity abnormal table records the related information of the article information table meeting the data integrity abnormal condition in the article information table set. And secondly, in response to the article information tables meeting the data consistency abnormal condition in the article information table set, generating a data consistency abnormal table according to the article information tables meeting the data consistency abnormal condition in the article information table set. In this way, the obtained data consistency abnormality table records the related information of the article information table satisfying the data consistency abnormality condition in the article information table set. And then, responding to the article information tables meeting the abnormal condition of the rate value in the article information table set, and generating an abnormal table of the rate value according to the article information tables meeting the abnormal condition of the rate value in the article information table set. In this way, the obtained value abnormality table records information related to the article information table satisfying the value abnormality condition in the article information table set. And then, generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate value abnormal table. Therefore, integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information can be obtained, and the article information table of related workers is prompted to have abnormal data. And finally, sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a related user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information. Therefore, related workers can be informed of the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information. And the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information, so that related workers can know the article information table needing to be modified, and can modify data with integrity abnormality, consistency abnormality or rate value abnormality in the modified article information table. Therefore, the integrity, consistency and accuracy of the data in the article information table are improved, and the reliability of the data is improved.
With further reference to FIG. 3, a flow 300 of further embodiments of a data anomaly alerting method is shown. The flow 300 of the data anomaly alarm method includes the following steps:
step 301, an item information table set is obtained.
Step 302, in response to the article information table meeting the data integrity abnormal condition existing in the article information table set, generating a data integrity abnormal table according to the article information table meeting the data integrity abnormal condition in the article information table set.
In some embodiments, the specific implementation of the steps 301 and 302 and the technical effects brought by the same can refer to the steps 201 and 202 in the embodiments corresponding to fig. 2, which are not described herein again.
Step 303, determining a data type set of each table field included in each item information table in the item information table set.
In some embodiments, an executing agent of the data anomaly alerting method (e.g., computing device 101 shown in FIG. 1) can determine a set of data types for each table field included in each item information table of the set of item information tables. In practice, the execution subject may determine, by using the Hive engine, a data type of each table field included in each item information table in the item information table set, and determine, by using the Click House engine, a data type of each table field included in each item information table in the item information table set, to obtain a data type set of each table field. Therefore, the data type set of each table field can be obtained, and data support is provided for subsequent processing.
And 304, responding to the article information table with the data type set of the table field meeting preset different conditions in the article information table set, and generating an attribute consistency abnormal table according to the article information table with the data type set of the table field meeting the preset different conditions.
In some embodiments, the execution subject may generate the attribute consistency exception table according to the item information table in which the data type set of the table field satisfies the preset different condition in response to the item information table in which the data type set of the table field satisfies the preset different condition existing in the item information table set. The preset dissimilarity condition may be that the data type is different from any data type in the data type set. In practice, first, the execution main body may select an article information table satisfying the preset different condition from the article information table set as a seventh candidate article information table, to obtain a seventh candidate article information table set. An attribute consistency exception null table may then be created. Then, the table name, the table administrator name, and the table field name having different data types of each seventh candidate item information table in the seventh candidate item information table set may be written into the attribute consistency abnormality empty table to obtain the attribute consistency abnormality table. Thus, the obtained attribute consistency exception table records fields in which the attributes do not agree.
And 305, generating attribute consistency abnormal alarm information according to the attribute consistency abnormal table.
In some embodiments, the execution subject may generate the attribute consistency abnormality warning information according to the attribute consistency abnormality table. In practice, the execution subject may save the attribute consistency abnormality table as an attribute consistency abnormality table image, and use the attribute consistency abnormality table image as attribute consistency abnormality alarm information. Therefore, the attribute consistency abnormal alarm information can be obtained, so that the related staff article information table is prompted to have attribute consistency abnormal data.
And step 306, sending the attribute consistency abnormal alarm information to the user terminal, so that the user terminal displays the attribute consistency abnormal alarm information.
In some embodiments, the execution main body may send the attribute consistency abnormality alarm information to the user terminal, so that the user terminal displays the attribute consistency abnormality alarm information. In practice, the execution main body may send the attribute consistency abnormality alarm information to the user terminal in a wired connection manner or a wireless connection manner, so that the user terminal displays the attribute consistency abnormality alarm information. Therefore, related workers can be informed of the attribute consistency abnormal alarm information.
Step 307, in response to the article information tables meeting the data consistency abnormal condition in the article information table set, generating a data consistency abnormal table according to the article information tables meeting the data consistency abnormal condition in the article information table set.
In some embodiments, the execution subject may generate the data consistency exception table according to the item information table satisfying the data consistency exception condition in the item information table set in response to the item information table satisfying the data consistency exception condition in the item information table set. The data consistency abnormal condition may be that the data type of the table field is different from the data type of the target table field included in the item information virtual table corresponding to the item information table. For example, the Presto engine may be utilized to determine the data type of the table fields included in the item information table. The data types of the table fields may include, but are not limited to: int (integer), and float (floating point). The object table field may be a table field in the virtual table of item information corresponding to a table field included in the item information table. In practice, first, the execution main body may select an article information table satisfying the data consistency abnormality condition from the article information table set as an eighth candidate article information table, and obtain an eighth candidate article information table set. A data consistency exception null table may then be created. Then, the table name, the table administrator name, and the table field name corresponding to the target table field of each eighth candidate item information table in the eighth candidate item information table set may be written in the data consistency abnormality empty table to obtain the data consistency abnormality table. In this way, the obtained data consistency abnormality table records the related information of the article information table satisfying the data consistency abnormality condition in the article information table set.
Step 303-. The factors that result in the inability to improve data consistency and reliability are as follows: and the data type of the data in the item information table which cannot be inquired in different inquiry modes. If the factors are solved, the effect of improving the consistency and the reliability of the data can be achieved. In order to achieve the effect, the data type set of each table field is obtained by using the Hive engine and the Click House engine, the attribute consistency abnormal table is generated, the attribute consistency abnormal alarm information is generated according to the attribute consistency abnormal table and is sent to the associated user terminal, and related workers can modify the table fields with the attribute consistency abnormality, so that the consistency and the reliability of the data are improved. And then whether the attribute of the table field is consistent with the attribute of the data recorded in the metadata is detected, so that related workers are prompted to modify the inconsistent condition, and the consistency and reliability of the data are further improved.
And 308, responding to the article information table meeting the abnormal condition of the rate value in the article information table set, and generating an abnormal table of the rate value according to the article information table meeting the abnormal condition of the rate value in the article information table set.
And 309, generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate value abnormal table.
And 310, sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a related user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
In some embodiments, the specific implementation of the steps 308-310 and the technical effects thereof can refer to the steps 204-206 in the embodiments corresponding to fig. 2, which are not described herein again.
As can be seen from fig. 3, compared with the description of some embodiments corresponding to fig. 2, the flow 300 of the data anomaly alarm method in some embodiments corresponding to fig. 3 embodies a step of expanding the data attribute consistency anomaly. Therefore, the solutions described in the embodiments can monitor whether the attributes of the data in the data warehouse are consistent, and prompt the relevant staff to modify the inconsistent situation, thereby improving the consistency and reliability of the data.
With further reference to fig. 4, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data anomaly warning device, which correspond to those of the method embodiments shown in fig. 2, and which can be applied in various electronic devices.
As shown in fig. 4, the data abnormality warning apparatus 400 of some embodiments includes: an acquisition unit 401, a first generation unit 402, a second generation unit 403, a third generation unit 404, a fourth generation unit 405, and a transmission unit 406. Wherein the obtaining unit 401 is configured to obtain a set of item information tables; the first generating unit 402 is configured to generate a data integrity exception table according to the item information tables satisfying the data integrity exception condition in the item information table set in response to the item information tables satisfying the data integrity exception condition existing in the item information table set; the second generating unit 403 is configured to generate a data consistency abnormal table according to the item information table satisfying the data consistency abnormal condition in the item information table set in response to the item information table satisfying the data consistency abnormal condition existing in the item information table set; the third generating unit 404 is configured to generate a rate value exception table according to the item information table satisfying the rate value exception condition in the item information table set in response to the item information table satisfying the rate value exception condition existing in the item information table set; the fourth generating unit 405 is configured to generate integrity abnormal alarm information, consistency abnormal alarm information, and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table, and the rate abnormal table, respectively; and the sending unit 406 is configured to send the integrity abnormality warning information, the consistency abnormality warning information, and the rate value abnormality warning information to the associated user terminal, so that the user terminal displays the integrity abnormality warning information, the consistency abnormality warning information, and the rate value abnormality warning information.
It will be understood that the elements described in the apparatus 400 correspond to various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 400 and the units included therein, and will not be described herein again.
Referring now to FIG. 5, a block diagram of an electronic device (such as computing device 101 shown in FIG. 1) 500 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, electronic device 500 may include a processing means (e.g., central processing unit, graphics processor, etc.) 501 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
Generally, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and a communication device 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 illustrates an electronic device 500 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 5 may represent one device or may represent multiple devices as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network via the communication means 509, or installed from the storage means 508, or installed from the ROM 502. The computer program, when executed by the processing device 501, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: and responding to the article information table meeting the data integrity abnormal condition in the article information table set, and generating a data integrity abnormal table according to the article information table meeting the data integrity abnormal condition in the article information table set. And generating a data consistency abnormal table according to the article information tables meeting the data consistency abnormal condition in the article information table set in response to the article information tables meeting the data consistency abnormal condition in the article information table set. And responding to the article information table meeting the abnormal condition of the rate value in the article information table set, and generating the abnormal table of the rate value according to the article information table meeting the abnormal condition of the rate value in the article information table set. And generating integrity abnormal alarm information, consistency abnormal alarm information and rate value abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate value abnormal table. And sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to the associated user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, a first generation unit, a second generation unit, a third generation unit, a fourth generation unit, and a transmission unit. Here, the names of these units do not constitute a limitation to the unit itself in some cases, and for example, the acquisition unit may also be described as "a unit that acquires a set of item information tables".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims (8)

1. A data abnormity warning method comprises the following steps:
acquiring an item information table set, wherein the acquiring the item information table set comprises:
generating an article information virtual table based on each original article information table in a prestored original article information table set to obtain an article information virtual table set, wherein the article information virtual table comprises metadata information of the original article information table;
acquiring table state information of each item information virtual table in the item information virtual table set;
taking an original article information table corresponding to the article information virtual table with the table state information representation online state corresponding to the article information virtual table in the article information virtual table set as an article information table to obtain an article information table set;
responding to the article information table meeting the data integrity abnormal condition in the article information table set, and generating a data integrity abnormal table according to the article information table meeting the data integrity abnormal condition in the article information table set, wherein the generating of the data integrity abnormal table comprises the following steps:
responding to an article information table with a table field value meeting a first data integrity abnormal condition in the article information table set, and generating a first data integrity abnormal table according to the article information table meeting the first data integrity abnormal condition, wherein the first data integrity abnormal condition is a table field value representation null value;
acquiring the number of table fields corresponding to each article information table in the article information table set;
in response to the existence of an article information table in the article information table set, wherein the corresponding table field number of the article information table set meets a second data integrity abnormal condition, a second data integrity abnormal table is generated according to the article information table meeting the second data integrity abnormal condition, wherein the second data integrity abnormal condition is that the table field number is different from the number of table field names included in an article information virtual table corresponding to the article information table;
acquiring a miscellaneous item dimension value set corresponding to each item information table in the item information table set;
responding to an article information table in which a corresponding de-duplicated miscellaneous item dimension value set meets a third data integrity abnormal condition in the article information table set, and generating a third data integrity abnormal table according to the article information table meeting the third data integrity abnormal condition, wherein the third data integrity abnormal condition is that a miscellaneous item dimension value included in the de-duplicated miscellaneous item dimension value set does not exist in miscellaneous item dimension values included in an article information virtual table corresponding to the article information table;
responding to the article information table meeting the data consistency abnormal condition in the article information table set, and generating a data consistency abnormal table according to the article information table meeting the data consistency abnormal condition in the article information table set;
responding to an article information table meeting a rate value abnormal condition in the article information table set, and generating a rate value abnormal table according to the article information table meeting the rate value abnormal condition in the article information table set;
generating integrity abnormal alarm information, consistency abnormal alarm information and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table and the rate abnormal table respectively;
and sending the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information to a related user terminal, so that the user terminal displays the integrity abnormal alarm information, the consistency abnormal alarm information and the rate value abnormal alarm information.
2. The method of claim 1, wherein the item information table included in the set of item information tables is an item information table comprising a white list field; and
the generating of the data integrity exception table includes:
deleting the white list field included in each article information table in the article information table set from the article information table, and determining the article information table subjected to deletion processing as a target article information table to obtain a target article information table set;
and responding to the target article information table meeting the data integrity abnormal condition in the target article information table set, and generating a data integrity abnormal table according to the target article information table meeting the data integrity abnormal condition in the target article information table set.
3. The method of claim 1, wherein prior to the generating a data consistency exception table, the method further comprises:
determining a set of data types of each table field included by each item information table in the set of item information tables;
responding to an article information table with a data type set of table fields meeting preset different conditions in the article information table set, and generating an attribute consistency abnormal table according to the article information table with the data type set of the table fields meeting the preset different conditions, wherein the preset different conditions are that the data type is different from any data type in the data type set;
generating attribute consistency abnormal alarm information according to the attribute consistency abnormal table;
and sending the attribute consistency abnormal alarm information to the user terminal, so that the user terminal displays the attribute consistency abnormal alarm information.
4. The method of claim 1, wherein the generating a data consistency exception table comprises:
responding to an article information table with the data type of a table field meeting a first data consistency abnormal condition in the article information table set, and generating a first data consistency abnormal table according to the article information table meeting the first data consistency abnormal condition, wherein the first data consistency abnormal condition is that the data type of the table field is different from the data type of a target table field included in an article information virtual table corresponding to the article information table;
in response to the existence of the item information table meeting the prerequisite condition in the item information table set, determining the item information table meeting the prerequisite condition in the item information table set as a first item information table to obtain a first item information table set;
determining each first article information table meeting preset index field conditions in the first article information table set as a second article information table group to obtain a second article information table set, wherein the preset index field conditions are that the same index field exists in each first article information table;
for each second item information table set in the second item information table set, determining an aggregation value corresponding to an index field corresponding to the second item information table set, which is included in each second item information table in the second item information table set, to obtain an aggregation value set;
determining an aggregation value group which meets a second data consistency abnormal condition in the obtained aggregation value groups as an abnormal aggregation value group to obtain an abnormal aggregation value group set, wherein the second data consistency abnormal condition is that the aggregation value is different from any aggregation value in the aggregation value groups;
generating a second data consistency abnormal table according to each second item information table group corresponding to the abnormal aggregation value group set;
and in response to the existence of an article information table with a tuple meeting a third data consistency abnormal condition in the article information table set, generating a third data consistency abnormal table according to the article information table meeting the third data consistency abnormal condition, wherein the third data consistency abnormal condition is that the same tuple exists in the article information table.
5. The method of claim 1, wherein the generating integrity anomaly alert information, consistency anomaly alert information, and rate value anomaly alert information comprises:
generating a data integrity abnormal table image and a data integrity abnormal message reminding number according to the data integrity abnormal table;
combining the data integrity abnormal form image, the data integrity abnormal message reminding number and the information of the responsible person associated with the data integrity abnormal form to obtain the integrity abnormal alarm information;
generating a data consistency abnormal form image and a data consistency abnormal message reminding number according to the data consistency abnormal form;
combining the data consistency abnormal form image, the data consistency abnormal message reminding number and the information of the responsible person associated with the data consistency abnormal form to obtain consistency abnormal alarm information;
generating a rate value abnormal table image and a rate value abnormal message reminding number according to the rate value abnormal table;
and combining the rate value abnormal table image, the rate value abnormal message reminding number and the information of the responsible person associated with the rate value abnormal table to obtain the rate value abnormal alarm information.
6. A data anomaly warning device comprising:
an obtaining unit configured to obtain a set of item information tables, wherein the obtaining of the set of item information tables includes:
generating an article information virtual table based on each original article information table in a prestored original article information table set to obtain an article information virtual table set, wherein the article information virtual table comprises metadata information of the original article information table;
acquiring table state information of each item information virtual table in the item information virtual table set;
taking an original article information table corresponding to the article information virtual table with the table state information representation online state corresponding to the article information virtual table in the article information virtual table set as an article information table to obtain an article information table set;
a first generating unit, configured to, in response to an item information table satisfying a data integrity exception condition existing in the item information table set, generate a data integrity exception table according to an item information table satisfying the data integrity exception condition in the item information table set, where the generating a data integrity exception table includes:
responding to an article information table with a table field value meeting a first data integrity abnormal condition in the article information table set, and generating a first data integrity abnormal table according to the article information table meeting the first data integrity abnormal condition, wherein the first data integrity abnormal condition is a table field value representation null value;
acquiring the number of table fields corresponding to each article information table in the article information table set;
in response to the existence of an article information table in the article information table set, wherein the corresponding table field number of the article information table set meets a second data integrity abnormal condition, a second data integrity abnormal table is generated according to the article information table meeting the second data integrity abnormal condition, wherein the second data integrity abnormal condition is that the table field number is different from the number of table field names included in an article information virtual table corresponding to the article information table;
acquiring a miscellaneous item dimension value set corresponding to each item information table in the item information table set;
responding to an article information table in which a corresponding de-duplicated miscellaneous item dimension value set meets a third data integrity abnormal condition in the article information table set, and generating a third data integrity abnormal table according to the article information table meeting the third data integrity abnormal condition, wherein the third data integrity abnormal condition is that a miscellaneous item dimension value included in the de-duplicated miscellaneous item dimension value set does not exist in miscellaneous item dimension values included in an article information virtual table corresponding to the article information table;
a second generating unit configured to generate a data consistency abnormal table according to the item information table satisfying the data consistency abnormal condition in the item information table set in response to the item information table satisfying the data consistency abnormal condition existing in the item information table set;
a third generating unit configured to generate a rate value exception table according to the item information tables satisfying the rate value exception condition in the item information table set in response to the item information tables satisfying the rate value exception condition existing in the item information table set;
a fourth generating unit configured to generate integrity abnormal alarm information, consistency abnormal alarm information, and rate abnormal alarm information according to the data integrity abnormal table, the data consistency abnormal table, and the rate abnormal table, respectively;
a sending unit configured to send the integrity abnormality alarm information, the consistency abnormality alarm information, and the rate value abnormality alarm information to an associated user terminal, so that the user terminal displays the integrity abnormality alarm information, the consistency abnormality alarm information, and the rate value abnormality alarm information.
7. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.
8. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-5.
CN202111323641.7A 2021-11-10 2021-11-10 Data abnormity alarm method and device, electronic equipment and computer readable medium Active CN113760882B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111323641.7A CN113760882B (en) 2021-11-10 2021-11-10 Data abnormity alarm method and device, electronic equipment and computer readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111323641.7A CN113760882B (en) 2021-11-10 2021-11-10 Data abnormity alarm method and device, electronic equipment and computer readable medium

Publications (2)

Publication Number Publication Date
CN113760882A CN113760882A (en) 2021-12-07
CN113760882B true CN113760882B (en) 2022-01-28

Family

ID=78784920

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111323641.7A Active CN113760882B (en) 2021-11-10 2021-11-10 Data abnormity alarm method and device, electronic equipment and computer readable medium

Country Status (1)

Country Link
CN (1) CN113760882B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107644077A (en) * 2017-09-19 2018-01-30 杭州贝购科技有限公司 Data consistency monitoring method, computer equipment and storage medium
CN111143623A (en) * 2019-12-31 2020-05-12 科技谷(厦门)信息技术有限公司 Data quality monitoring method in big data environment
CN111240936A (en) * 2020-01-13 2020-06-05 北京点众科技股份有限公司 Data integrity checking method and equipment
CN112416896A (en) * 2020-11-17 2021-02-26 珠海格力电器股份有限公司 Data abnormity warning method and device, storage medium and electronic device
CN112598342A (en) * 2021-03-05 2021-04-02 北京每日优鲜电子商务有限公司 Alarm control method, device, equipment and medium for data display equipment

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9298381B2 (en) * 2014-05-30 2016-03-29 International Business Machines Corporation Data integrity monitoring among sysplexes with a shared direct access storage device (DASD)
KR101922044B1 (en) * 2016-11-29 2019-02-14 주식회사 실크로드소프트 Recovery technique of data intergrity with non-stop database server redundancy
CN110928891B (en) * 2019-11-08 2023-01-03 中国工商银行股份有限公司 Data consistency detection method, device, computing equipment and medium
CN111372043B (en) * 2020-02-06 2021-05-11 浙江大华技术股份有限公司 Abnormity detection method and related equipment and device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107644077A (en) * 2017-09-19 2018-01-30 杭州贝购科技有限公司 Data consistency monitoring method, computer equipment and storage medium
CN111143623A (en) * 2019-12-31 2020-05-12 科技谷(厦门)信息技术有限公司 Data quality monitoring method in big data environment
CN111240936A (en) * 2020-01-13 2020-06-05 北京点众科技股份有限公司 Data integrity checking method and equipment
CN112416896A (en) * 2020-11-17 2021-02-26 珠海格力电器股份有限公司 Data abnormity warning method and device, storage medium and electronic device
CN112598342A (en) * 2021-03-05 2021-04-02 北京每日优鲜电子商务有限公司 Alarm control method, device, equipment and medium for data display equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
通信集中监测及告警技术研究;王欣;《铁路通信信号工程技术》;20191231;第16卷(第12期);第25-28 *

Also Published As

Publication number Publication date
CN113760882A (en) 2021-12-07

Similar Documents

Publication Publication Date Title
CN109299348B (en) Data query method and device, electronic equipment and storage medium
CN112765152B (en) Method and apparatus for merging data tables
US20210200782A1 (en) Creating and Performing Transforms for Indexed Data on a Continuous Basis
CN115293657B (en) Carbon emission index information generation method, apparatus, electronic device, and medium
CN110737655B (en) Method and device for reporting data
CN114003473A (en) Page operation behavior backtracking method and device and electronic equipment
CN113760882B (en) Data abnormity alarm method and device, electronic equipment and computer readable medium
CN113378346A (en) Method and device for model simulation
CN115187364A (en) Method and device for monitoring deposit risk under bank distributed scene
CN115277355A (en) Method, device, equipment and medium for processing state code data of monitoring system
CN114625763A (en) Information analysis method and device for database, electronic equipment and readable medium
CN114443437A (en) Alarm root cause output method, apparatus, device, medium, and program product
CN111552715B (en) User query method and device
CN111427878B (en) Data monitoring alarm method, device, server and storage medium
CN114116924A (en) Data query method based on map data, map data construction method and device
CN113282455A (en) Monitoring processing method and device
CN113722315A (en) Data generation method and device, electronic equipment and computer readable medium
CN116739789B (en) Virtual article return information sending method and device, electronic equipment and medium
CN114064494A (en) Data abnormity alarm method and device, electronic equipment and computer readable medium
CN110019352B (en) Method and apparatus for storing data
CN116663907A (en) Method and device for generating early warning information, computer equipment and storage medium
CN113568936A (en) Real-time streaming data storage method and device and terminal equipment
CN111930704A (en) Service alarm equipment control method, device, equipment and computer readable medium
CN113962782A (en) Order information processing method and device, electronic equipment and computer readable medium
CN115599627A (en) Stream data sequence quality detection method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20240122

Address after: Room 2103, F008, Building 1, No. 213 Funing Road, Zumiao Street, Chancheng District, Foshan City, Guangdong Province, 528000 (Residence Declaration)

Patentee after: Foshan Huayue Intellectual Property Operation Co.,Ltd.

Country or region after: China

Address before: Room 078, No. 1-305, 3rd floor, commercial building, No. 9 Wangjing street, Chaoyang District, Beijing 100101

Patentee before: Beijing daily vegetable market Technology Co.,Ltd.

Country or region before: China

TR01 Transfer of patent right