CN113535701B - Method, device, medium and product for inspecting quality of warehouse - Google Patents

Method, device, medium and product for inspecting quality of warehouse Download PDF

Info

Publication number
CN113535701B
CN113535701B CN202110819727.2A CN202110819727A CN113535701B CN 113535701 B CN113535701 B CN 113535701B CN 202110819727 A CN202110819727 A CN 202110819727A CN 113535701 B CN113535701 B CN 113535701B
Authority
CN
China
Prior art keywords
inspected
metadata
quality
measurement information
parameter measurement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110819727.2A
Other languages
Chinese (zh)
Other versions
CN113535701A (en
Inventor
王金东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202110819727.2A priority Critical patent/CN113535701B/en
Publication of CN113535701A publication Critical patent/CN113535701A/en
Application granted granted Critical
Publication of CN113535701B publication Critical patent/CN113535701B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/283Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The method obtains a rule set of a field to be inspected, wherein the rule set of the field to be inspected comprises evaluation parameters corresponding to a data warehouse to be inspected and parameter measurement information corresponding to the evaluation parameters, and the parameter measurement information comprises preset conditions which are required to be met by metadata of an object to be inspected, which belongs to a set object type; and obtaining a meta-rule model containing the search information and the preset conditions based on the parameter measurement information. The metadata warehouse stores the metadata of the object to be inspected, and the search information can search the metadata of the object to be inspected belonging to the set object type from the metadata warehouse, so that the association relation between the rule set of the field to be inspected and the metadata warehouse can be established in a mode of associating the parameter measurement information and the meta rule model, and the quality inspection model of the warehouse is obtained. Quality audit results may be obtained based on the several-bin quality audit model. Therefore, the purpose of quality detection of the data warehouse is achieved.

Description

Method, device, medium and product for inspecting quality of warehouse
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method, an apparatus, a medium, and a product for inspecting quality of a bin.
Background
The Data Warehouse (DW or DWH, abbreviated as "Data Warehouse") is a theme-Oriented (Subject organized), integrated (Integrated), relatively stable (Non-volume), and history change-reflecting (Time variance) Data collection used for supporting management decisions. The data warehouse may provide services to the enterprise, for example, to guide business process improvements.
If the quality of the data warehouse is poor, for example, metadata of data stored in the data warehouse is not standard and accurate, difficulty of mining analysis of the data, resource waste, decision errors, reduction of value density of the data warehouse, and the like may be caused, so that the service provided for the enterprise based on the data warehouse is inaccurate, or the service provided for the enterprise based on the data warehouse consumes a long time. Therefore, it is important to check the quality of the data warehouse.
Disclosure of Invention
The disclosure provides a method, a device, a medium and a product for inspecting quality of a data warehouse, which at least solve the problem that the quality of the data warehouse cannot be detected in the related art. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, a method for inspecting quality of a digital warehouse is provided, which is applied to a server, and includes: acquiring a to-be-inspected field rule set, wherein the to-be-inspected field rule set comprises evaluation parameters corresponding to a to-be-inspected data warehouse and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, the type of the set object corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; obtaining a meta-rule model containing search information and the preset conditions based on the parameter measurement information, wherein the search information is used for searching the metadata of the object to be inspected, which belongs to the set object type, from a metadata warehouse, and the metadata of the object to be inspected, which is stored in the data warehouse, is stored in the metadata warehouse; associating the parameter measurement information with the meta-rule model to obtain a multi-bin quality inspection model, wherein the multi-bin quality inspection model comprises the to-be-inspected field rule set and the meta-rule model associated with the parameter measurement information; and obtaining a quality inspection result representing whether the metadata of the object to be inspected meets the preset condition or not based on the warehouse quality inspection model.
With reference to the first aspect, in a first possible implementation manner, the step of obtaining a rule set of a domain to be audited includes: receiving a request for constructing the quality inspection model of the warehouse sent by the electronic equipment; sending a preset domain rule set to the electronic equipment, wherein the domain rule set to be checked is a subset of the domain rule set; and receiving the to-be-inspected field rule set fed back by the electronic equipment.
With reference to the first aspect, in a second possible implementation manner, the rule set of the field to be inspected further includes a field to be inspected, a field sub-item included in the field to be inspected, and a preset evaluation manner corresponding to the parameter measurement information, where the evaluation manner includes quality results corresponding to a plurality of evaluation parameter ranges, respectively, and the evaluation parameters are evaluation parameters corresponding to the field sub-item, and the step of obtaining the quality inspection result of the data warehouse based on the multi-bin quality inspection model includes: searching the metadata of the object to be inspected which belongs to the set object type from the metadata warehouse according to the searching information; detecting whether the metadata of the object to be inspected meets the preset conditions contained in the meta-rule model or not to obtain a grading parameter; obtaining a first quality result corresponding to an evaluation parameter range containing the scoring parameter from the evaluation mode; calculating to obtain a second quality result corresponding to the evaluation parameter according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information; calculating to obtain a third quality result corresponding to the field sub-item according to the second quality result corresponding to the evaluation parameter and a second weight corresponding to the evaluation parameter; calculating to obtain a fourth quality result corresponding to the field to be inspected according to the third quality result corresponding to the field sub-item and the third weight corresponding to the field sub-item; and calculating to obtain quality evaluation parameters corresponding to the data warehouse according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, wherein the quality inspection result comprises the quality evaluation parameters.
With reference to the first aspect, in a third possible implementation manner, after the step of detecting whether the metadata of the object to be inspected satisfies the preset condition included in the meta rule model, the method further includes: obtaining target metadata of the target object to be inspected which does not meet the preset condition; acquiring target parameter measurement information corresponding to the target metadata; acquiring a target evaluation parameter corresponding to the target parameter measurement information; wherein the quality inspection result further comprises target metadata modification guidance information, and the target metadata modification guidance information comprises: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
With reference to the first aspect, in a fourth possible implementation manner, the set of domain rules to be audited further includes: communication information of the administration personnel; the warehouse quality inspection method further comprises the following steps: and sending the quality inspection result to the electronic equipment with the communication information of the administration personnel. Receiving correction content sent by the electronic equipment, wherein the correction content comprises correction information corresponding to the target metadata, and the target metadata modification guidance information is a basis for a user to modify the target metadata to obtain the correction content.
According to a second aspect of the embodiments of the present disclosure, there is provided a method for inspecting quality of a digital warehouse, applied to an electronic device, including: sending a request for constructing a warehouse quality inspection model of a data warehouse to a server; receiving a domain rule set fed back by the server, wherein the domain rule set comprises a plurality of candidate evaluation parameters and candidate parameter weighing information corresponding to each candidate evaluation parameter, the candidate parameter weighing information comprises preset conditions which need to be met by metadata of an object to be inspected, which belongs to a set object type, the set object type corresponds to the evaluation parameters, and the object to be inspected is stored in the data warehouse; displaying the set of domain rules; responding to the selection operation aiming at the domain rule set to obtain a domain rule set to be inspected, wherein the domain rule set to be inspected comprises evaluation parameters selected from the domain rule set and parameter measurement information corresponding to the evaluation parameters selected from the domain rule set; sending the rule set of the field to be inspected to the server; the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset conditions, the search information is used for searching metadata of an object to be audited which belongs to a set object type from a metadata warehouse, the metadata of the object to be audited stored in the metadata warehouse is stored, and the quality audit model comprises the field rule set to be audited and the meta-rule model associated with the parameter measurement information.
With reference to the second aspect, in a first possible implementation manner, the method further includes: receiving the quality inspection result sent by the server, wherein the quality inspection result comprises target metadata modification guidance information and quality evaluation parameters corresponding to the data warehouse, the quality evaluation parameters represent whether metadata of an object to be inspected in the data warehouse meets preset conditions or not, the target metadata modification guidance information comprises at least one of target metadata, target parameter measurement information and target evaluation parameters corresponding to the target parameter measurement information, and the target metadata does not meet the preset conditions contained in the target parameter measurement information; and displaying the quality inspection result on a display interface.
According to a third aspect of the embodiments of the present disclosure, there is provided a device for inspecting quality of a data warehouse, applied to a server, including:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is configured to acquire a field rule set to be inspected, the field rule set to be inspected comprises evaluation parameters corresponding to a data warehouse to be inspected and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, which belongs to a set object type, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; a second obtaining module, configured to obtain a meta-rule model including search information and the preset condition based on the parameter measurement information, where the search information is used to search for metadata of the object to be inspected belonging to the set object type from a metadata repository, and the metadata of the object to be inspected stored in the metadata repository is stored in the metadata repository; the correlation module is configured to correlate the parameter measurement information with the meta-rule model to obtain a multi-bin quality inspection model, and the multi-bin quality inspection model comprises the to-be-inspected field rule set and the meta-rule model correlated with the parameter measurement information; and the third acquisition module is configured to acquire a quality inspection result representing whether the metadata of the object to be inspected meets the preset condition or not based on the warehouse quality inspection model.
With reference to the third aspect, in a first possible implementation manner, the first obtaining module is specifically configured to: a first receiving unit configured to receive a request sent by an electronic device to construct the warehouse quality inspection model; the sending unit is configured to send a preset domain rule set to the electronic equipment, wherein the domain rule set to be audited is a subset of the domain rule set; the second receiving unit is configured to receive the to-be-audited domain rule set fed back by the electronic equipment.
With reference to the third aspect, in a second possible implementation manner, the rule set of the field to be inspected further includes a field to be inspected, a field sub-item included in the field to be inspected, and a preset evaluation manner corresponding to the parameter measurement information, where the evaluation manner includes quality results corresponding to a plurality of evaluation parameter ranges, and the evaluation parameter is an evaluation parameter corresponding to the field sub-item, and the third obtaining module is specifically configured to: the searching unit is configured to search the metadata of the object to be checked, which belongs to the set object type, from the metadata warehouse according to the searching information; the first acquisition unit is configured to detect whether the metadata of the object to be inspected meets the preset conditions contained in the meta-rule model or not to obtain a grading parameter; the second acquisition unit is configured to acquire a first quality result corresponding to an evaluation parameter range containing the scoring parameter from the evaluation mode; the first calculating unit is configured to calculate a second quality result corresponding to the evaluation parameter according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information; the second calculating unit is configured to calculate a third quality result corresponding to the field sub-item according to the second quality result corresponding to the evaluation parameter and a second weight corresponding to the evaluation parameter; the third calculating unit is configured to calculate a fourth quality result corresponding to the to-be-inspected field according to the third quality result corresponding to the field sub-item and a third weight corresponding to the field sub-item; and the fourth calculating unit is configured to calculate and obtain the quality assessment parameters corresponding to the data warehouse according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, wherein the quality inspection result comprises the quality assessment parameters.
With reference to the third aspect, in a third possible implementation manner, the method further includes: the fourth acquisition module is configured to acquire target metadata of the target object to be inspected, which does not meet the preset condition; a fifth obtaining module, configured to obtain target parameter measurement information corresponding to the target metadata; a sixth obtaining module, configured to obtain a target evaluation parameter corresponding to the target parameter measurement information; wherein the quality audit result further comprises target metadata modification guidance information, and the target metadata modification guidance information comprises: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
With reference to the third aspect, in a fourth possible implementation manner, the set of domain rules to be audited further includes: communication information of the administration personnel; further comprising: and the sending result module is configured to send the quality inspection result to the electronic equipment with the communication information of the administration personnel. A content receiving module configured to receive corrected content sent by the electronic device, where the corrected content includes correction information corresponding to the target metadata, and the target metadata modification guidance information is a basis for a user to modify the target metadata to obtain the corrected content.
According to a fourth aspect of the embodiments of the present disclosure, there is provided a digital warehouse quality inspection device, applied to an electronic device, including: a first sending module configured to send a request to build a warehouse quality audit model of a data warehouse to a server; a first receiving module, configured to receive a domain rule set fed back by the server, where the domain rule set includes a plurality of candidate evaluation parameters and candidate parameter measurement information corresponding to each of the candidate evaluation parameters, the candidate parameter measurement information includes a preset condition that metadata of an object to be inspected that belongs to a set object type needs to be satisfied, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; a first display module configured to display the set of domain rules; the system comprises an acquisition set module, a domain rule set selection module and a domain rule set checking module, wherein the acquisition set module is configured to respond to selection operation aiming at the domain rule set and acquire a domain rule set to be checked, and the domain rule set to be checked comprises evaluation parameters selected from the domain rule set and parameter measurement information corresponding to the evaluation parameters selected from the domain rule set; the second sending module is configured to send the rule set of the field to be audited to the server; the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset conditions, the search information is used for searching metadata of an object to be audited which belongs to a set object type from a metadata warehouse, the metadata of the object to be audited stored in the metadata warehouse is stored, and the quality audit model comprises the field rule set to be audited and the meta-rule model associated with the parameter measurement information.
With reference to the fourth aspect, in a first possible implementation manner, the method further includes: a second receiving module, configured to receive the quality audit result sent by the server, where the quality audit result includes target metadata modification guidance information and quality evaluation parameters corresponding to the data warehouse, where the quality evaluation parameters characterize whether metadata of an object to be audited in the data warehouse meets a preset condition, the target metadata modification guidance information includes at least one of target metadata, target parameter measurement information, and target evaluation parameters corresponding to the target parameter measurement information, and the target metadata does not meet the preset condition included in the target parameter measurement information; a second display module configured to display the quality audit result on a display interface.
According to a fifth aspect of embodiments of the present disclosure, there is provided a server including: a first processor; a first memory for storing the first processor-executable instructions; wherein the first processor is configured to execute the instructions to implement the bin quality audit method of the first aspect.
According to a sixth aspect of embodiments of the present disclosure, there is provided an electronic apparatus including: a second processor; a second memory for storing the second processor-executable instructions; wherein the second processor is configured to execute the instructions to implement the bin quality audit method of aspect.
According to an eighth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions, when executed by a first processor of a server, enable the server to perform the bin quality audit method according to the first aspect; or, when executed by a second processor of an electronic device, enable the electronic device to perform the bin quality audit method of the second aspect.
According to a ninth aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a first processor, implement the bin quality audit method of the first aspect; alternatively, the computer program/instructions when executed by the second processor implement the method for quality audit of a bin of the second aspect.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
in the method for inspecting quality of the data warehouse, a rule set of a field to be inspected is obtained, wherein the rule set of the field to be inspected comprises evaluation parameters corresponding to the data warehouse to be inspected and parameter measurement information corresponding to the evaluation parameters, and the parameter measurement information comprises preset conditions which are required to be met by metadata of an object to be inspected, which belongs to a set object type, namely the rule set of the field to be inspected represents an inspection mode aiming at the data warehouse; and obtaining a meta-rule model containing the search information and the preset conditions based on the parameter measurement information. The metadata warehouse stores metadata of the objects to be inspected, and the search information can search the metadata of the objects to be inspected which belong to the set object type from the metadata warehouse, so that the association relation between the rule set of the fields to be inspected and the metadata warehouse can be established by associating the parameter measurement information and the meta rule model, and the multi-bin quality inspection model comprising the rule set of the fields to be inspected and the meta rule model can be obtained. Therefore, the metadata of the object to be inspected, which belongs to the set object type, can be obtained from the metadata warehouse based on the meta-rule model in the quality inspection model of the several bins; and then, judging whether the metadata of the object to be inspected meets the preset conditions or not based on the preset conditions contained in the meta-rule model so as to obtain a quality inspection result. Therefore, the purpose of quality detection of the data warehouse is achieved. Compared with the process that the association relationship between the rule set of the field to be inspected and the metadata warehouse is manually established by the user, the method provided by the embodiment of the disclosure is more flexible and efficient.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a block diagram illustrating hardware involved in an application scenario of an embodiment of the present disclosure, according to an exemplary embodiment;
FIG. 2 is a flow chart illustrating interaction of an electronic device with a server in accordance with an exemplary embodiment;
FIG. 3 is a block diagram illustrating a candidate domain to be audited and relationships between candidate domain sub-items, according to an example embodiment;
FIG. 4 is a diagram illustrating an electronic device presenting a user interface containing a set of domain rules in accordance with an exemplary embodiment;
FIGS. 5 a-5 d are schematic diagrams illustrating modification of parametric measurement information according to an exemplary embodiment;
6 a-6 d are schematic diagrams illustrating one implementation of hierarchical structure information in accordance with an illustrative embodiment;
FIG. 7 is a flow diagram illustrating a method for bin quality audit as applied to a server in accordance with an exemplary embodiment;
FIG. 8 is a block diagram illustrating a bin quality audit device applied to a server in accordance with an exemplary embodiment;
FIG. 9 is a block diagram illustrating a bin quality audit device applied to an electronic device in accordance with an exemplary embodiment;
FIG. 10 is a block diagram illustrating an apparatus 100 for a server in accordance with an exemplary embodiment;
FIG. 11 is a block diagram illustrating an apparatus 110 for an electronic device in accordance with an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in other sequences than those illustrated or described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The embodiment of the disclosure provides a method and a device for inspecting quality of a plurality of bins, a server, electronic equipment, a medium and a product.
Fig. 1 is a block diagram illustrating hardware involved in an application scenario of an embodiment of the present disclosure according to an exemplary embodiment. The first application scenario relates to hardware comprising: a server 11, at least one electronic device 12, at least one metadata repository 13, and at least one electronic device 14.
For example, different metadata warehouses 13 may belong to the same enterprise or may belong to different enterprises.
Illustratively, the metadata repository 13 is used to store metadata of objects to be audited stored by the data repository.
The differences and connections between the data warehouse and the metadata warehouse are described below by way of example.
The data warehouse stores objects to be inspected, and the objects to be inspected can be any one of data, tables, products and tasks. The following description will be given by taking table 1 as an example, and the information is shown in table 1.
TABLE 1
User name Number learning Achievement
Zhang San 123456 60
Li Si 123457 90
The data warehouse stores a table 1 containing user names, school numbers, and scores. Illustratively, the objects to be audited stored in the data warehouse may be table 1, or, fields (user name, scholarly number, score, etc.), or, data (zhang san, lie san, etc.).
Metadata (Metadata), also called intermediary data or relay data, is data (data about data) describing data, which is data about a data warehouse, and is mainly information describing data attributes, and is used to support functions such as indicating storage locations, historical data, resource lookup, file records, and the like, and related key data about data source definitions, target definitions, conversion rules, and the like generated during the construction of the data warehouse.
The metadata warehouse with the association relation with the data warehouse is used for storing the metadata of the objects to be checked in the data warehouse.
In the following, taking table 1 as an example to illustrate the metadata, if the object to be inspected is table 1, the metadata of the object to be inspected may be: at least one of the name of table 1, the size of table 1, the storage format of table 1 and the fields contained in table 1, if the object to be inspected is Zhang III, the metadata of the object to be inspected is the fields in which Zhang III is located.
For example, the metadata stored in the metadata repository may be as shown in table 2, where table 2 is merely an example and does not limit the format of the metadata stored in the metadata repository.
Table 2 metadata stored by metadata warehouse
Figure BDA0003171447040000061
Figure BDA0003171447040000071
As can be seen from Table 2, the data stored in the metadata repository is metadata of the object to be inspected.
As can be seen from tables 1 and 2, the data warehouse is used for storing objects to be inspected. And the metadata warehouse is used for storing the metadata of the object to be checked.
The objects to be inspected are only examples and are not limited to the types of the objects to be inspected, for example, the objects to be inspected may also be tasks or products or tables. For example, the metadata of the object to be inspected may be a name of the task or table or product, a storage format of the task or table or product, a size of the task or table or product, a number of fields included in the task or table or product, and a specific field included in the task or table or product.
Illustratively, one data warehouse corresponds to one or more metadata warehouses, that is, one or more metadata warehouses store metadata of objects to be inspected stored in one data warehouse. For example, the data warehouse stores tables, tasks, and products, and assuming that metadata of the tables is stored by the metadata warehouse 1, metadata of the tasks is stored by the metadata warehouse 2, and metadata of the products is stored by the metadata warehouse 3, the data warehouse corresponds to 3 metadata warehouses, and the metadata warehouses are respectively: metadata repository 1, metadata repository 2, metadata repository 3.
It is understood that the data warehouse and the metadata warehouse have an association relationship, and if the metadata corresponding to the object to be inspected stored in the data warehouse changes (for example, the storage format of table 1 of the object to be inspected stored in the data warehouse changes), the metadata of the object to be inspected stored in the metadata warehouse also changes correspondingly, for example, the field value of the storage format corresponding to table 1 in table 2 changes, and vice versa.
The server 11 may be, for example, one server, a server cluster composed of a plurality of servers, or a cloud computing service center. The server 11 may include a processor, memory, and a network interface, among others.
Illustratively, the server 11 stores a collection of web pages related to a specific content, which are created on the internet according to a certain rule using a tool such as HTML (standard universal markup language). I.e. the server 11 is the carrier of the web site.
For example, electronic device 12 and electronic device 14 may be the same electronic device; or electronic device 12 and electronic device 14 may be different electronic devices.
For example, the electronic device 12 may be any electronic product capable of interacting with a user through one or more modes of a keyboard, a touch pad, a touch screen, a remote controller, voice interaction, or a handwriting device, for example, a mobile phone, a tablet computer, a palm computer, a personal computer, a wearable device, a smart television, and the like.
For example, the electronic device 14 may be any electronic product capable of interacting with a user through one or more modes of a keyboard, a touch pad, a touch screen, a remote controller, voice interaction, or a handwriting device, for example, a mobile phone, a tablet computer, a palm computer, a personal computer, a wearable device, a smart television, and the like.
The embodiments of the present application can be applied to various application scenarios, and the present application provides, but is not limited to, the following two application scenarios.
In a first application scenario, electronic device 12 may be an electronic device belonging to a business, and electronic device 12 and electronic device 14 may be the same electronic device. The staff of the enterprise has the requirement of constructing the quality inspection model of the warehouse, and can access the server 11 through the electronic equipment 12 to construct the quality inspection model of the warehouse on line in real time.
In this case, the electronic device 12 establishes connection and communication with the server 11 through the wireless network; the server 11 establishes a connection and communicates with the metadata repository 13 through a wireless network.
In a second application scenario, the electronic device 12 may be an electronic device belonging to the server 11 side, and the electronic device 14 is an electronic device belonging to the enterprise side, that is, if an enterprise operator has a requirement for constructing the quality inspection model of the warehouse, the operator at the server 11 side may be notified, and the operator constructs the quality inspection model of the warehouse through the electronic device 12 and the server 11, and then sends the quality inspection model of the warehouse or obtains the quality inspection result based on the quality inspection model of the warehouse to the electronic device 14 at the enterprise side.
In this case, the electronic device 12 and the server 11 may establish connection and communication through a wired network or a wireless network; the server 11 establishes a connection and communicates with the metadata repository 13 through a wireless network.
Fig. 1 is only an example, and the number of the servers 11, the electronic devices 12, the metadata repository 13, and the electronic devices 14 may be set according to actual needs. In fig. 1, a server 11, an electronic device 12, a metadata repository 13 and an electronic device 14 are shown. For example, the electronic device 12 and the electronic device 14 may be the same electronic device or different electronic devices, and may be specifically determined based on a specific application scenario (in the first application scenario, the electronic device 12 and the electronic device 14 are the same electronic device, and in the second application scenario, the electronic device 12 and the electronic device 14 are different electronic devices).
For example, the constructed multi-bin quality inspection model can be repeatedly used, for example, a data warehouse is inspected based on the multi-bin quality inspection model every preset time, quality inspection results are obtained, so that quality inspection results at all times are obtained, and whether the treatment of the metadata corresponding to the object to be inspected, which is stored in the data warehouse, is effective or not can be obtained through the quality inspection results at different times, so that the purpose of circularly evaluating the treatment is realized, and the continuous optimization of the treatment work is reflected.
For example, after a period of time, if the quality audit model does not meet the requirements, a new quality audit model may be generated again through the interaction between the electronic device 12 and the server 11, reflecting the continuous optimization of the treatment level.
Illustratively, the server 11 may send the quality audit result to the electronic device 14. The electronic device 14 can display the quality inspection result and perform corresponding operations.
In an alternative implementation, the corresponding operation may be to correct for erroneous metadata. The metadata warehouse and the data warehouse have an association relation, and after error metadata is corrected to obtain corrected content, the metadata corresponding to the corresponding object to be inspected stored in the data warehouse and the metadata stored in the metadata warehouse can be corrected through the corrected content, so that the error metadata stored in the metadata warehouse can be continuously corrected, the metadata corresponding to the object to be inspected stored in the data warehouse can be continuously corrected, and the accuracy of the metadata corresponding to the object to be inspected stored in the data warehouse can be improved.
Illustratively, the accuracy of the object to be inspected stored in the data warehouse is represented by the accuracy of the metadata corresponding to the object to be inspected, for example, the range of the data to which the object to be inspected belongs is more accurate, the names of the fields corresponding to the same object to be inspected are the same, and the type of the value to which the object to be inspected belongs is more accurate, thereby reducing the difficulty of mining and analyzing the data in the data warehouse.
In an alternative implementation, the corresponding operation may be to perform deduplication on repeated metadata, thereby reducing the data amount of redundant data of the metadata. After the duplicate metadata is deduplicated, the corresponding duplicate metadata stored in the data warehouse is deduplicated, and the duplicate metadata stored in the metadata warehouse is deduplicated, so that the data volume of redundant data of the metadata of the object to be audited stored in the data warehouse is reduced.
Illustratively, after the corresponding duplicate metadata stored by the data warehouse is deduplicated, the objects to be audited associated with the metadata are also deduplicated.
After the metadata is corrected and deduplicated, the metadata corresponding to the object to be checked stored in the data warehouse is more accurate and has no redundant data. For example, the storage format of the table, the size of the table, the data range to which the object to be inspected stored in the table belongs, the fields in the table, and the data type to which the data in the table belongs are more accurate. In the process of accessing the data warehouse, the corresponding object to be inspected cannot be accessed due to errors of the storage format; the association relation of each table cannot be obtained due to different field names of the fields which are supposed to be the same in different tables; the metadata corresponding to the object to be inspected stored in the data warehouse is accurate and has no redundant data, so that the access frequency of the object to be inspected stored in the data warehouse is increased, and the value density of the data stored in the data warehouse is increased.
The value density is explained below.
If a large portion of the objects to be inspected stored in the data warehouse are used when the data warehouse is used to provide services for an enterprise, the value density of the data warehouse is very high. If most of the objects to be inspected stored in the data warehouse are not used, the value density of the data warehouse is low.
Those skilled in the art will appreciate that the above described electronic devices and servers are merely examples, and that other existing or future electronic devices or servers, as may be suitable for use with the present disclosure, are also included within the scope of the present disclosure and are hereby incorporated by reference.
The following describes an interaction flow between the server 11 and the electronic device 12 with reference to the application scenario.
FIG. 2 is a flow chart illustrating an interaction of an electronic device with a server according to an example embodiment.
In step S21, if the electronic device 12 detects an operation of constructing the quality audit model for the data warehouse, it sends a request for constructing the quality audit model to the server 11.
In step S22, the server 11 sends a preset domain rule set to the electronic device 12 in response to the request in step S21.
In step S23, the electronic device 12 presents the domain rule set.
Illustratively, the server 11 is a carrier of a website, and then a browser may be run in the electronic device 12, and the electronic device 12 accesses the server 11 through the browser. And interacts with the web site in the server 11 through the browser.
Illustratively, the electronic device 12 may run a client corresponding to the server 11, and the electronic device 12 interacts with the server 11 through the client. The client may be a web version client or an application client.
In step S24, the electronic device 12 responds to the selection operation on the domain rule set to obtain a domain rule set to be audited, where the domain rule set to be audited includes the evaluation parameters selected from the domain rule set and the parameter measurement information corresponding to the evaluation parameters selected from the domain rule set.
In step S25, the electronic device 12 sends the set of domain rules to be checked to the server 11.
In step S26, the server 11 obtains a bin quality inspection model based on the domain rule set to be inspected and the metadata repository.
The set of domain rules is explained below.
The set of domain rules includes a plurality of domain levels, each domain level including: the method comprises the steps of selecting a candidate field to be inspected, selecting a candidate field subitem contained in the candidate field to be inspected, selecting a candidate evaluation parameter corresponding to the candidate field subitem, selecting candidate parameter measurement information corresponding to the candidate evaluation parameter, and selecting a candidate evaluation mode corresponding to the candidate parameter measurement information.
For example, each domain level may include one or more candidate domains to be inspected, each candidate domain to be inspected includes one or more candidate domain sub-items, each candidate domain sub-item may correspond to one or more candidate evaluation parameters, each candidate evaluation parameter may correspond to one or more candidate parameter metrics, and each candidate parameter metric corresponds to one or more candidate evaluation modes.
In the embodiment of the application, the preset fields to be inspected, field sub-items, evaluation parameters, parameter measurement information and evaluation modes are called candidate fields to be inspected, candidate field sub-items, candidate evaluation parameters, candidate parameter measurement information and candidate evaluation modes. In the embodiment of the application, the information selected by the user from the candidate to-be-inspected field, the candidate field subitems, the candidate evaluation parameters, the candidate parameter measurement information and the candidate evaluation mode is called the to-be-inspected field, the field subitems, the evaluation parameters, the parameter measurement information and the evaluation mode. Both are essentially identical.
Next, a description will be given of "a candidate to-be-inspected domain and a candidate domain sub-item included in the candidate to-be-inspected domain" in the domain hierarchy.
The candidate fields to be inspected can comprise one or more first-level fields, each first-level field comprises one or more second-level fields, each second-level field comprises one or more third-level fields, \8230, each ith-level field comprises one or more (i + 1) th-level fields, and i is an integer greater than or equal to 0.
The containment relationships between the respective hierarchical domains are determined based on the containment relationships between the domains.
If i =0, the 0 th level domain is the candidate to-be-inspected domain, that is, the first level domain included in the candidate to-be-inspected domain is the candidate domain sub-item.
If i = M, the M + 1-th level domain is a candidate domain sub-item, and M is a positive integer greater than or equal to 1. That is, in the embodiments of the present application, the bottom domain is referred to as a "candidate domain sub-item", and the following description is given by way of example.
Illustratively, if the relationship between the candidate to-be-inspected domain and the candidate domain sub-item is regarded as a structure tree, then the candidate to-be-inspected domain is the root node, and the candidate domain sub-item is the leaf node. Illustratively, FIG. 3 is a block diagram illustrating relationships between candidate domains to be audited and candidate domain sub-items, according to an exemplary embodiment.
As can be seen from fig. 3, the candidate areas to be inspected include 3 first level areas, which are: a first hierarchical domain 1, a first hierarchical domain 2 and a first hierarchical domain 3, wherein the first hierarchical domain 1 and the first hierarchical domain 3 are non-leaf nodes, the first hierarchical domain 2 is a leaf node, that is, the first hierarchical domain 2 is a candidate domain subitem; the first hierarchical domain 1 comprises two second hierarchical domains, namely a second hierarchical domain 11 and a second hierarchical domain 12, wherein the second hierarchical domain 11 and the second hierarchical domain 12 are leaf nodes, namely the second hierarchical domain 11 and the second hierarchical domain 12 are candidate domain sub-items; the first hierarchical domain 3 includes 1 second hierarchical domain 31; the second hierarchical domain 31 is a non-leaf node, the second hierarchical domain 31 includes 1 third hierarchical domain 311, and the third hierarchical domain 311 is a leaf node, so the third hierarchical domain 311 is a candidate domain sub-item.
In an optional implementation manner, the manner of dividing the technical fields is different, and the obtained candidate fields to be inspected are different. There are various dividing ways, and the embodiments of the present disclosure provide, but are not limited to, the following two.
The first division mode can be divided based on the data development life cycle, and the obtained multiple candidate fields to be inspected are respectively as follows: development and production, asset management, resource use, quality assurance, security management, application service and data destruction.
The second division mode can be divided based on the data life cycle process, and the obtained multiple candidate fields to be inspected are respectively: data specification, data design, data quality, data security, data applications, and data resources.
The candidate fields to be inspected obtained by the first division mode or the second division mode almost cover all fields needing to be evaluated by the data warehouse, and enterprises researching any technology can obtain the required candidate fields to be inspected from the candidate fields to be inspected, namely the method has universal applicability.
The storage format of the domain rule set stored in the server 11 may be any one of a table, a structure, and a linked list. The domain rule set is explained below by specific examples.
The following description takes 6 candidate to-be-inspected fields obtained by the second division manner as an example, and the data specification refers to: development specifications and management specifications of data construction; the data design means that: rationality, versatility, complexity of data table design; the data quality refers to: guarantee condition, timeliness and accuracy of data; the data security means that: safety audit and safety risk of data; the data application means: convenience, ease of use of data services; the data resources refer to: utilization and efficiency of data resources.
The following description takes the field rule set stored by the server 11 in a table as an example, and as shown in table 3, the first hierarchical field included in the candidate field to be inspected is taken as an example of a candidate field sub-entry in table 3.
TABLE 3
Figure BDA0003171447040000111
The following describes candidate evaluation parameters in the domain level.
The candidate evaluation parameter (evaluation index) of a candidate domain sub-item is a specific specification of the candidate domain sub-item for which evaluation is required. Is the materialization of the abstract candidate domain sub-item. As shown in Table 3, the candidate domain sub-item "production Specification" may be evaluated with the candidate evaluation parameter "number of non-Specification production tasks" and the evaluation parameter "number of non-Specification production tables".
The following describes candidate parameter metrics in the domain level.
The candidate parameter measurement information comprises preset conditions which need to be met by metadata of the object to be inspected, belonging to the set object type. The candidate parameter measurement information is information for measuring a candidate evaluation parameter. The candidate parameter measurement information is described below by way of example.
For example, the set object type may be any one of a table, a task, and a product.
TABLE 4
Figure BDA0003171447040000121
As can be seen from table 4, the candidate parameter measurement information "the task running time of the common model table should be less than 2h" and "the task running time of the application layer model table should be less than 1h" may be used to measure the "number of time-consuming tasks" of the candidate evaluation parameter.
As can be seen from table 4, one candidate parameter measurement information corresponds to one or more objects to be inspected, and one object to be inspected corresponds to one or more metadata, for example, the candidate parameter measurement information "task should be downstream and the heat of task should be greater than 0 in 30 days" corresponds to one "task" of object to be inspected, and the "task" of object to be inspected corresponds to two metadata, which are "downstream number of tasks" and "heat of task in 30 days" respectively.
Illustratively, each metadata corresponding to the object to be inspected corresponds to a preset condition, for example, the preset condition of "the number of downstream tasks" of the "task" of the object to be inspected is "greater than or equal to 1", and the preset condition of "the heat of the task of the" task of the object to be inspected is "greater than 0".
Illustratively, the table 4 is to illustrate preset conditions that the candidate parameter measurement information includes the object to be inspected, the metadata of the object to be inspected, and the metadata of the object to be inspected need to satisfy, so that the object to be inspected, the metadata of the object to be inspected, and the preset conditions that the object to be inspected need to satisfy are respectively displayed as fields in the table 4. In practical applications, the electronic device 12 may display the object to be inspected, the metadata of the object to be inspected, and the preset condition that the metadata of the object to be inspected needs to satisfy as fields, or may not display these three fields.
The following describes a candidate evaluation method included in the domain rule set by way of example.
Illustratively, the candidate evaluation mode includes quality results corresponding to a plurality of evaluation parameter ranges, respectively.
For example, the evaluation parameter range has various meanings, and the disclosed embodiment provides, but is not limited to, the following four.
The first method comprises the following steps: the evaluation parameter range refers to the number of objects to be inspected, the metadata of which meet preset conditions; and the second method comprises the following steps: the evaluation parameter range refers to the number of objects to be checked, the metadata of which do not meet the preset conditions; and the third is that: the evaluation parameter range refers to the ratio of the number of the objects to be inspected, the metadata of which meet the preset conditions, to the total number of the objects to be inspected; and fourthly: the evaluation parameter range refers to the ratio of the number of the objects to be inspected, the metadata of which do not meet the preset condition, to the total number of the objects to be inspected.
The meaning of the second evaluation parameter range is described below by taking the candidate evaluation parameter in table 4 as the time-consuming task number as an example (similar to the first case, which is not described herein again), as shown in table 5.
TABLE 5 candidate evaluation mode for time-consuming task number
Figure BDA0003171447040000131
Figure BDA0003171447040000141
Illustratively, the time-consuming task number corresponds to two candidate parameter measurement information, and each candidate parameter measurement information corresponds to one quality result, so that the quality results corresponding to the time-consuming task number can be obtained based on the quality results corresponding to the two candidate parameter measurement information corresponding to the time-consuming task number.
The meaning of the third evaluation parameter range is described below by taking the candidate evaluation parameter in table 4 as the storage format yield as an example, and the candidate evaluation mode of the storage format yield is shown in table 6.
Table 6 evaluation method of storage format yield
Figure BDA0003171447040000142
Illustratively, the storage format yield corresponds to one candidate parameter measurement information, and the quality result of the storage format yield is the quality result of the corresponding candidate parameter measurement information.
The meaning of the fourth evaluation parameter range is described below by taking the evaluation candidate evaluation parameter in table 4 as the redundant data proportion, and the candidate evaluation manner of the redundant data proportion is shown in table 7.
TABLE 7 candidate evaluation method of redundant data ratio
Figure BDA0003171447040000143
Figure BDA0003171447040000151
For example, the electronic device 12 may receive and display the domain rule set stored in the server 11, so that the user may select the information required by the user to obtain the domain rule set to be audited.
It can be understood that, under the condition that the user knows each candidate field to be inspected, each candidate field subitem, each candidate evaluation parameter, each candidate parameter measurement information and the meaning of each candidate evaluation mode, the user can select the required information to obtain the rule set of the field to be inspected, so that the rule set of the field to be inspected can be quickly and conveniently obtained.
Illustratively, the server 11 obtains the domain rule set to be audited through the electronic device 12, and generates the warehouse quality audit model based on the domain rule set to be audited and the metadata warehouse.
For example, the set of domain rules to be audited may be a subset of the set of domain rules.
There are various operations for detecting and constructing the warehouse quality inspection model in step S21, and the embodiment of the present disclosure provides, but is not limited to, the following five implementation manners.
The first implementation manner of detecting the operation of constructing the warehouse quality inspection model is as follows: and if the first preset gesture is detected, determining that the operation of constructing the warehouse quality inspection model is detected.
Illustratively, the first preset gesture may be any gesture, such as, "√" or "x," or the like.
The second implementation manner of detecting the operation of constructing the warehouse quality inspection model is as follows: and if the first preset voice is detected, determining that the operation of constructing the warehouse quality inspection model is detected.
Illustratively, the first predetermined speech includes a sound of "construct a bin quality audit model".
The third implementation manner of detecting the operation of constructing the warehouse quality inspection model is as follows: and if the first preset touch track is detected, determining that the operation of constructing the warehouse quality inspection model is detected.
Illustratively, the first preset touch down trajectory may be any trajectory, for example, any one of an upward swipe, a downward swipe, a leftward swipe, a rightward swipe, and a circular trajectory.
The fourth implementation manner of detecting the operation of constructing the warehouse quality inspection model is as follows: and if the operation of logging in the server is detected, determining that the operation of constructing the multi-bin quality inspection model is detected.
For example, if the server is a carrier of a website, a browser may be run in the electronic device 12, and the electronic device 12 logs in the website through the browser.
For example, the electronic device 12 may run a client corresponding to the server 11, and then the electronic device 12 may log in the server 11 through the client.
The fifth implementation manner of detecting the operation of constructing the warehouse quality inspection model is as follows: and if the first preset key is detected to be touched and pressed, determining that the operation of constructing the warehouse quality inspection model is detected.
For example, the first preset key may be a virtual key or a physical key.
Illustratively, the domain rule set displayed by the electronic device may be displayed in the form of a table, for example, as shown in table 3, or in the form of a structure tree, as shown in fig. 4.
FIG. 4 is a diagram illustrating an electronic device presenting a user interface containing a set of domain rules, according to an example embodiment. The objects in the selected state are characterized in fig. 4 by filled dark boxes.
Illustratively, the user interface is shown with all candidate domains to be inspected, for example, 6 candidate domains to be inspected in table 3.
For example, there are various ways for the electronic device to display the domain rule set, and the embodiments of the present disclosure provide, but are not limited to, the following two ways.
The first method for displaying the domain rule set comprises the following steps: the electronic device 12 completely displays the domain rule set for the user to select before the user performs the selection operation.
The second method for presenting a domain rule set includes steps a11 to a14. The sub-nodes of the candidate to-be-inspected domain do not include the non-candidate domain sub-items, i.e. all are the candidate domain sub-items.
In step a11, if a selection operation for a certain candidate to-be-inspected field is detected, a candidate field sub-item of the to-be-inspected field selected by the user (in the embodiment of the present application, the candidate to-be-inspected field selected by the user is referred to as the to-be-inspected field) is displayed.
For example, if the user does not select a candidate domain to be inspected, a candidate domain sub-item of the candidate domain to be inspected, such as the data specification 41 in fig. 4, may not be displayed, and the candidate domain sub-item of the data specification 41 is not shown in fig. 4 because the user does not select the data specification 41.
Since the user selects the candidate to-be-audited domain data design 42, FIG. 4 shows candidate domain sub-items of the data design 42, such as data integrity 421 and data commonality 422.
In step a12, if a selection operation for a candidate domain sub-item is detected, candidate evaluation parameters corresponding to the domain sub-item selected by the user (in the embodiment of the present application, the candidate domain sub-item selected by the user is referred to as a domain sub-item) are displayed.
For example, if the user selects the candidate domain sub-item of the data design 42, namely the data complete 421, and does not select the candidate domain sub-item of the data design, "data general" 422, then fig. 4 shows the candidate evaluation parameters of the domain sub-item, namely the data complete 421, but does not display the candidate evaluation parameters of the data general 422.
In step a13, if a selection operation for a candidate evaluation parameter is detected, candidate parameter measurement information corresponding to the evaluation parameter selected by the user (in the embodiment of the present application, the candidate evaluation parameter selected by the user is referred to as an evaluation parameter) is displayed.
For example, if the user selects the evaluation parameter "number of time-consuming tasks" 431 of the candidate field "data resource" 43 to be audited, fig. 4 shows the candidate parameter measurement information corresponding to the number of time-consuming tasks 431, for example, two candidate parameter measurement information, that is, the running time of the common model table should be less than 2h and the running time of the application layer model table should be less than 1 h.
In step a14, if a selection operation for the candidate parameter measurement information is detected, a candidate scoring manner corresponding to the parameter measurement information selected by the user (in the embodiment of the present application, the candidate parameter measurement information selected by the user is referred to as parameter measurement information) is displayed.
For example, if the user selects the evaluation parameter "the running duration of the application layer model table should be less than 1h", then fig. 4 shows the candidate scoring manner corresponding to "the running duration of the application layer model table should be less than 1h", for the user to select. Since the user does not select that the running time of the common model table should be less than 2h, fig. 4 does not show the candidate scoring mode corresponding to that the running time of the common model table should be less than 2 h.
Illustratively, the user interface displayed by the electronic device and including the domain rule set can also display the domain range of the candidate domain to be inspected (i.e. the description of the candidate domain to be inspected). The field-wide methods provided by the embodiments of the present disclosure include, but are not limited to, the following two ways.
The first implementation mode comprises the following steps: and if the cursor moves to the position of the candidate field to be inspected, displaying the field range of the candidate field to be inspected.
The representation of the cursor may be any graphic, and the cursor is represented in fig. 4 in a small hand shape, as shown in fig. 4, the cursor is located on the data resource 43, so the user interface shows the domain scope of the data resource 43.
The second implementation mode comprises the following steps: and the electronic equipment always displays the field range of each candidate field to be inspected in the process of displaying the field rule set.
In the method for inspecting quality of several warehouses provided by the embodiment of the disclosure, the server 11 stores the field rule set, and the user can browse the field rule set through the electronic device 12, and the user does not need to know the data stored in the data warehouse or the metadata stored in the metadata warehouse, that is, the user does not need to know the bottom data. After determining which field quality of the data warehouse needs to be inspected, the user can select from the field rule set to obtain a field rule set to be inspected, and after receiving the field rule set to be inspected, which is selected by the user through the electronic device 12, the server 11 can automatically establish association with the metadata warehouse through each parameter measurement information contained in the field rule set to be inspected, so as to obtain a quality inspection model of the data warehouse. Compared with the method that a user needs to know data at the bottom layer of a data warehouse and manually establishes the association between the rule set of the field to be inspected and the metadata warehouse, the method provided by the embodiment of the disclosure is more flexible and efficient.
For users of different enterprises, if the fields of the data warehouse to be inspected are different, different candidate fields to be inspected, candidate field subitems, candidate evaluation parameters, candidate parameter measurement information and candidate evaluation modes can be selected from the field rule set, so that different quality inspection models of the multi-bin can be obtained, and the rapid landing of the different quality inspection models of the multi-bin is realized.
In an alternative implementation manner, there are various implementation manners of step S24, and the embodiment of the present disclosure provides, but is not limited to, the following two implementation manners.
The first implementation of step S24 includes steps B11 to B15.
In step B11, a field to be inspected is obtained from a plurality of candidate fields to be inspected.
Exemplarily, a plurality of candidate fields to be inspected are divided according to a data development life cycle or a data life cycle process, so that the plurality of candidate fields to be inspected provided by the embodiment of the present disclosure cover all directions of quality inspection of a data warehouse, and therefore, the warehouse quality inspection method provided by the embodiment of the present disclosure can be applied to enterprises in different fields, and has general applicability.
Illustratively, the number of the areas to be inspected obtained in step B11 may be one or more.
In step B12, a domain sub-item corresponding to the to-be-inspected domain is obtained from the candidate domain sub-item corresponding to the to-be-inspected domain.
Illustratively, the number of the domain sub-items obtained in step B12 may be one or more.
In step B13, the evaluation parameters corresponding to the domain sub-items are obtained from the candidate evaluation parameters corresponding to the domain sub-items.
For example, the number of the evaluation parameters obtained in step B13 may be one or more.
In step B14, parameter measurement information corresponding to the evaluation parameter is obtained from the candidate parameter measurement information corresponding to the evaluation parameter.
For example, the number of parameter measurement information obtained in step B14 may be one or more.
For example, there are various implementations of step B14, and the embodiments of the present disclosure provide, but are not limited to, two implementations.
The first implementation manner of step B14 includes: and selecting the parameter measurement information corresponding to the evaluation parameter from the candidate parameter measurement information corresponding to the evaluation parameter.
That is, the parameter measurement information is any one of a plurality of candidate parameter measurement information stored in advance by the server 11.
The second implementation of step B14 includes steps B141 to B142.
It can be understood that the technology in the enterprise is continuously updated, and if the technology developed by the enterprise does not reach the preset condition included in the candidate parameter measurement information preset by the server 11, for example, the candidate parameter measurement information "the operation time of the public model table is less than 2h" preset by the server 11, the current technology of the enterprise may not reach "the operation time of the public model table is less than 2h", the current technology of the enterprise may reach that the operation time of the public model table is less than 3h, so the preset condition in the candidate parameter measurement information may need to be changed. With the lapse of time, the technology of the enterprise is continuously updated, for example, after the technology of the enterprise is continuously updated, the operation time of the public model table can be less than 2h, and at this time, the enterprise can change the corresponding preset condition in the candidate parameter measurement information again, so that the candidate parameter measurement information conforms to the actual condition of the technology of the enterprise. A second implementation of step B14 is proposed on the basis of this.
In step B141, for any candidate parameter measurement information corresponding to the evaluation parameter, if a change operation performed on the candidate parameter measurement information is detected, an edit box including the candidate parameter measurement information is displayed.
In step B142, the modified candidate parameter measurement information obtained through the edit box is determined as the parameter measurement information corresponding to the evaluation parameter.
Fig. 5a to 5a are schematic diagrams illustrating a modification operation of candidate parameter measurement information according to an exemplary embodiment.
Fig. 5a shows candidate parameter measurement information "the running time of the public model table should be less than 2h" preset by the server 11, fig. 5b shows an edit box 51, and the edit box 51 shows candidate parameter measurement information "the running time of the public model table should be less than 2h" to be changed. Fig. 5c is a schematic diagram of changing the candidate parameter measurement information "the running time of the public model table should be less than 2h" shown in the edit box 51 to "the running time of the public model table should be less than 3h", and the diagram is shown in fig. 5d after the change is completed.
As shown in FIG. 5d, after the change is completed, the "running time of the common model table should be less than 3h" is in the selected state.
In conclusion, due to the fact that the candidate parameter measurement information can be modified, the candidate parameter measurement information can be modified based on the current technical situation of an enterprise in the process of constructing the multi-bin quality inspection model so as to adapt to the current technology of the enterprise, and the candidate parameter measurement information can be correspondingly modified along with the continuous development of the enterprise technology, so that the characteristic of long-term performance is embodied.
In an optional implementation manner, if one candidate parameter measurement information corresponds to one candidate evaluation manner, the evaluation manner of the candidate parameter measurement information selected by the user may be defined as the corresponding candidate evaluation manner, and for example, if one candidate parameter measurement information corresponds to one candidate evaluation manner, the candidate evaluation manner may also be displayed to allow the user to select; if one candidate parameter measurement information corresponds to a plurality of candidate evaluation manners, and the plurality of candidate evaluation manners need to be presented for the user to select, then, for example, step B15 may also be included.
In step B15, the evaluation manner corresponding to the parameter measurement information is obtained from the candidate evaluation manner corresponding to the parameter measurement information.
For example, there are multiple implementations of step B15, and the embodiments of the present disclosure provide but are not limited to two implementations.
The first step B15 implementation manner includes: and selecting the evaluation mode corresponding to the parameter measurement information from the candidate evaluation modes corresponding to the parameter measurement information.
That is, each evaluation method is any one of a plurality of candidate evaluation methods stored in advance by the server 11.
The second step B15 implementation includes step B151 to step B152.
It is understood that if the candidate evaluation method preset by the server 11 does not meet the actual conditions of the enterprise, the user may change the candidate evaluation method. A second step B15 implementation is proposed based on this.
In step B151, an edit box including the candidate evaluation method is displayed for the candidate evaluation method corresponding to the parameter measurement information if a change operation performed on the candidate evaluation method is detected.
The process of displaying the edit box including the candidate evaluation mode can be referred to as the process of displaying the edit box including the candidate parameter measurement information shown in fig. 5a to 5d, and details are not repeated here.
For example, the candidate evaluation manner may include a lot of content, if the user needs to modify all the content of the candidate evaluation manner, the edit box may include all the content of the candidate evaluation manner, and if the user only needs to modify the local content of the candidate evaluation manner, the edit box may include the local content.
For example, the evaluation mode will be described below by taking the candidate parameter measurement information "the lifetime of the A1-level table needs to be set permanently" of the candidate evaluation parameter "the lifetime management pass rate" as an example, as shown in table 8.
TABLE 8
Figure BDA0003171447040000191
Illustratively, if the user needs to change the evaluation parameter range "(0.95, 1)", then the evaluation parameter range "(0.95, 1)", is double-clicked, then the evaluation parameter range "(0.95, 1)", is included in the edit box, illustratively, if the user needs to change the evaluation parameter range (0.7, 0.8) ", then the evaluation parameter range" (0.7, 0.8) ", is double-clicked, then the evaluation parameter range" (0.7, 0.8) ", is included in the edit box.
In step B152, the modified candidate evaluation method obtained through the edit box is determined as the evaluation method corresponding to the parameter measurement information.
In summary, the rule set of the domain to be inspected can be obtained through the steps B11 to B15.
The second implementation of step S24 includes steps C11 to C12.
It can be understood that if any object of the domain to be inspected, the domain subitems, the evaluation parameters, the parameter measurement information, and the evaluation mode, which is required by the user, is absent in the domain rule set displayed by the electronic device 12, the user is required to add a corresponding object. It can be understood that, in the adding process, if a user adds a field sub-item, the user needs to add an evaluation parameter of the field sub-item, if the evaluation parameter is added, the parameter measurement information corresponding to the evaluation parameter needs to be added, and if the parameter measurement information is added, the user needs to add an evaluation mode corresponding to the parameter measurement information, so that the user needs to be guided to increase step by step. Based on this, the embodiment of the present disclosure provides a second implementation manner of step S24, which involves an adding operation of an object. The adding operation of the object includes step C11 to step C12.
In step C11, if an operation of adding a target object implemented in the user interface is detected, object addition guidance information is obtained, where the object addition guidance information represents an object addition sequence from the target object to an evaluation manner corresponding to the target object, and the target object is any one of a field to be audited, a field sub-item, an evaluation parameter, parameter measurement information, and an evaluation manner.
In step C12, an object set is obtained by adding guidance information to the object, the object set includes objects in an evaluation manner corresponding to the target object from the target object, and the domain rule set to be audited includes the object set.
For example, if the target object is a field to be inspected, the object adds guiding information including, but not limited to: the field to be inspected → field sub-item → evaluation parameter → parameter measurement information → evaluation mode. The object set obtained in step C12 includes: the fields to be inspected, field sub-items, evaluation parameters, parameter measurement information and evaluation modes.
For example, if the target object is a domain sub-item, the object adding guidance information includes, but is not limited to: field sub-item → evaluation parameter → parameter measurement information → evaluation mode. The object set obtained in step C12 includes: the method comprises the following steps of field sub-items, evaluation parameters, parameter measurement information and an evaluation mode.
For example, if the target object is an evaluation parameter, the object adding guidance information includes, but is not limited to: evaluation parameters → parameter measurement information → evaluation mode. The object set obtained in step C12 includes: evaluation parameters, parameter measurement information and an evaluation mode.
For example, if the target object is parameter measurement information, the object adding guidance information includes, but is not limited to: parameter measurement information → evaluation mode. The object set obtained in step C12 includes: the parameters measure information and the evaluation mode.
For example, if the target object is in an evaluation mode, the object adding guidance information includes, but is not limited to: the manner of evaluation. The object set obtained in step C12 includes: the manner of evaluation.
For example, the object addition guide information may include a blank table or at least one window.
The following describes the object adding guidance information by using a specific example.
Assuming that the target object is a domain sub-item, the object is added with the guidance information as shown in table 9.
TABLE 9
Figure BDA0003171447040000201
Figure BDA0003171447040000211
In table 6, the area to be inspected is referred to as "data resource", and the content that the user needs to add is represented in the form of table.
As shown in table 6, after the user adds a field sub-item, a "must fill" text is displayed at the position of the evaluation parameter corresponding to the field sub-item to guide the user to add a corresponding object. If the user adds an evaluation parameter, a 'must fill' character is displayed at the position of the parameter measurement information corresponding to the evaluation parameter so as to guide the user to add a corresponding object. And the subsequent analogy is omitted for brevity.
Fig. 6 a-6 d are schematic diagrams illustrating one implementation of hierarchical structure information according to an exemplary embodiment.
The user can add objects of each level based on fig. 6a to 6 d.
Fig. 6a to 6d illustrate the target object as a field sub-item, and illustrate the field sub-item added under the "data resource" of the field to be inspected.
Suppose that the user adds a domain sub-item as a computing resource. Illustratively, each time the user clicks the add sub-node 61 button shown in FIG. 6a, an edit box may be added to facilitate the user entering a field sub-item. For example, if the user needs to delete a certain domain sub-item added by the user, the user may select the domain sub-item and click the delete sub-node 62 button.
After the user has added the field sub-item, the user can click the next item 63 button and then can enter the window shown in fig. 6 b. Illustratively, each time the user clicks the add sub-node 64 button shown in FIG. 6b, an edit box may be added to provide the user with an input of the evaluation parameter. For example, if the user needs to delete some evaluation parameter added by the user, the user may select the evaluation parameter and click the delete child node 65 button.
After the user has added the evaluation parameters, he can click on the next item 66 and can then proceed to the window shown in fig. 6 c. Illustratively, each time the user clicks the add sub-node 67 button shown in FIG. 6c, an edit box may be added to allow the user to enter a parameter measurement. For example, if the user needs to delete some parameter measurement information added by the user, the user may select the parameter measurement information and click the delete sub-node 68 button.
After the user has added the parameter measurement information, the user can click the next item 69 and then enter the window shown in fig. 6 d. Illustratively, each time the user clicks the add sub-node 70 button as shown in FIG. 6d, an edit box may be added to facilitate the user entering an evaluation mode. For example, if the user needs to delete a certain evaluation mode added by the user, the user may select the evaluation mode and click the delete child node 71 button. If the user has added the evaluation mode, the user can click the finish 72 button.
In summary, the user may add the guidance information based on the objects shown in table 6 or fig. 6a to 6d to obtain the object set.
For example, in the implementation manner of the second step S24, the rule set of the domain to be inspected may be obtained by adding a target object; for example, in the implementation manner of the second step S24, the set of domain rules to be inspected may be obtained by adding the target object and by the implementation manner of the first step S24.
The warehouse quality inspection method implemented by the server 11 is described with reference to the application scenario and the interaction flow between the server 11 and the electronic device 12.
Fig. 7 is a flowchart illustrating a quality audit method of a warehouse applied to a server according to an exemplary embodiment, which includes the following steps S71 to S74.
In step S71, a set of domain rules to be checked is obtained.
The rule set of the field to be inspected comprises the field to be inspected, field subitems contained in the field to be inspected, evaluation parameters corresponding to the field subitems, parameter measurement information corresponding to the evaluation parameters and an evaluation mode corresponding to the parameter measurement information.
For the descriptions of the to-be-inspected field, the field subitems, the evaluation parameters, the parameter measurement information, and the evaluation mode, the descriptions of the candidate to-be-inspected field, the candidate field subitems, the candidate evaluation parameters, the candidate parameter measurement information, and the candidate evaluation mode may be referred to, and are not repeated here.
Illustratively, the set of domain to be audited rules includes one or more domains to be audited. Each field to be inspected comprises one or more field sub-items, each field sub-item corresponds to one or more evaluation parameters, each evaluation parameter corresponds to one or more parameter weighing information, and each parameter weighing information corresponds to one evaluation mode.
In step S72, based on the parameter measurement information, a meta rule model including search information and the preset condition is obtained, where the search information is used to search for the metadata of the object to be inspected belonging to the set object type from a metadata repository, and the metadata of the object to be inspected stored in the metadata repository is stored in the metadata repository.
Illustratively, each parameter metric corresponds to a meta-rule model.
In step S73, the parameter measurement information and the meta-rule model are associated to obtain a multi-bin quality inspection model, where the multi-bin quality inspection model includes the to-be-inspected domain rule set and the meta-rule model associated with the parameter measurement information.
In step S74, based on the warehouse quality inspection model, a quality inspection result indicating whether the metadata of the object to be inspected satisfies the preset condition is obtained.
For example, the domain rule set to be checked may be sent to the server 11 by the electronic device 12.
Illustratively, each parameter metric corresponds to a meta-rule model. The meta-rule model is explained below by specific examples.
Assuming that the evaluation parameter is the number of non-standard production tables, and the parameter measurement information corresponding to the evaluation parameter is that "the table name needs to start with dwd or dws or ads", the meta-rule model corresponding to the parameter measurement information is table. Assuming that the parameter measurement information is "the annotation of the table needs to meet the requirement that the annotation specification cannot be null", the meta rule model corresponding to the parameter measurement information is table.
In summary, the meta rule model may obtain the metadata of the object to be inspected from the metadata repository through the search information, and determine whether the metadata of the object to be inspected satisfies the preset condition based on the preset condition included in the meta rule model.
In summary, step S72 is to convert the parameter measurement information that cannot be identified by the computer into the query judgment statement that can be identified by the computer.
In step S73, the association relationship between the parameter measurement information and the meta rule model is automatically established, and since the meta rule model can obtain the metadata of the object to be inspected belonging to the set object type from the metadata repository, after the association relationship between the parameter measurement information and the meta rule model is established, the association relationship between the rule set in the field to be inspected and the metadata repository is established.
It can be understood that the meta-rule model based on the bin quality inspection model can obtain the metadata of the object to be inspected, which belongs to the set object type, from the metadata warehouse; and then, judging whether the metadata of the object to be inspected meets the preset conditions or not based on the preset conditions contained in the meta-rule model so as to obtain a quality inspection result.
In the method for inspecting quality of several warehouses provided by the embodiment of the disclosure, a rule set of a field to be inspected is obtained, the rule set of the field to be inspected comprises evaluation parameters corresponding to a data warehouse to be inspected and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected belonging to a set object type, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected: and obtaining a meta-rule model containing the search information and the preset conditions based on the parameter measurement information. The metadata warehouse stores the metadata of the object to be inspected, and the search information can search the metadata of the object to be inspected belonging to the set object type from the metadata warehouse, so that the association relation between the rule set of the field to be inspected and the metadata warehouse can be established in a mode of associating the parameter measurement information and the meta rule model, and the quality inspection model of the multi-bin comprising the rule set of the field to be inspected and the meta rule model can be obtained. Therefore, the metadata of the object to be inspected, which belongs to the set object type, can be obtained from the metadata warehouse based on the meta-rule model in the multi-bin quality inspection model; and then, judging whether the metadata of the object to be inspected meets the preset conditions or not based on the preset conditions contained in the meta-rule model so as to obtain a quality inspection result. Therefore, the purpose of quality detection of the data warehouse is achieved. Compared with the method that the user needs to manually establish the association relationship between the domain rule set to be inspected and the metadata warehouse, the method provided by the embodiment of the invention is more flexible and efficient.
In an alternative implementation manner, there are various implementation manners of step S74, and the disclosed embodiment provides, but is not limited to, the following manner, which includes the following steps D11 to D13.
In step D11, the metadata of the object to be inspected belonging to the set object type is searched from the metadata repository according to the search information included in the meta rule model.
Step D11 is described below by way of example, assuming that the evaluation parameter is the lifecycle management yield, the parameter measurement information corresponding to the evaluation parameter is that the lifecycle of the A1-level table needs to be set permanently, the type of the set object to which the object to be audited belongs is the A1-level table, and the metadata stored in the metadata repository is stored in the form of a table, as shown in table 10. A table with a table name meta.
As shown in table 10, meta.dwd _ table refers to the name of a table storing metadata of the table, "param _ table" refers to a variable pointing to a field in the meta.dwd _ table, and [ param _ table ] table _ asseses level refers to a table _ asseses level field in the meta.dwd _ table, wherein table _ asseses level represents a table asset level,
table 10 metadata stored in metadata repository "meta.
Figure BDA0003171447040000231
Figure BDA0003171447040000241
The set object type to which the object to be inspected belongs is meta, dwd _ table [ param _ table ]. Table _ assetslevels = 'A1', the metadata corresponding to the object to be inspected is meta, dwd _ table [ param _ table ]. Table _ life cycle, the preset condition corresponding to the parameter measurement information to be inspected is table _ life cycle =9999, and exemplarily, a maximum value, for example 9999, is used for representing permanence. Then, the meta-rule model is: meta.dwd _ table [ param _ table ]. Table _ associations level = 'A1' and meta.dwd _ table [ param _ table ]. Table _ life cycle =999999.
Exemplarily, the lookup information in the meta rule model is meta.dwd _ table [ param _ table ]. Table _ associations level = 'A1' and meta.dwd _ table [ param _ table ]. Table _ life cycle, and the life cycle value of the table belonging to the A1-level table can be obtained through the lookup information.
In step D12, it is detected whether the metadata of the object to be inspected obtained in step D11 meets the preset condition included in the meta-rule model, so as to obtain a scoring parameter.
It is understood that the metadata repository may store one or more metadata of the objects to be audited, for example, the A1-level table in table 11 corresponds to a plurality of metadata, which are a table life cycle value and a data date.
In step D13, a first quality result corresponding to an evaluation parameter range including the score parameter is obtained from the evaluation means.
For example, assuming that the parameter measurement information is that the operation duration of the common model table should be less than 2h, if the evaluation manner is shown in table 5, and assuming that the number of the common model tables with the operation duration of greater than or equal to 2h obtained in step D12 is 9, the obtained score parameter is 9. As can be seen from table 5, the evaluation parameter range including the score parameter 9 is (8, 10), and the corresponding quality result is 70, and the first quality result is 70.
In step D14, a second quality result corresponding to the evaluation parameter is calculated according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information.
In step D15, a third quality result corresponding to the field sub-item is calculated according to the second quality result corresponding to the evaluation parameter and the second weight corresponding to the evaluation parameter.
In step D16, a fourth quality result corresponding to the to-be-inspected field is calculated and obtained according to the third quality result corresponding to the field sub-item and the third weight corresponding to the field sub-item.
In step D17, a quality assessment parameter corresponding to the data warehouse is calculated and obtained according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, where the quality inspection result includes the quality assessment parameter.
Illustratively, the rule set of the domains to be inspected further comprises the weight of each domain to be inspected, the weight of each domain sub-item, the weight of each evaluation parameter and the weight of each parameter measurement information.
For example, the weight may be preset; illustratively, the user may modify the preset weights.
It is assumed that the set of domain rules to be audited, which the user determines from the set of domain rules, is shown in table 11. It is assumed that each parameter measurement information in table 11 corresponds to an evaluation mode, and an evaluation mode corresponding to the parameter measurement information preset by the server is selected by default, so the evaluation mode is not shown in table 11.
TABLE 11
Figure BDA0003171447040000242
Figure BDA0003171447040000251
The steps D14 to D17 will be described with reference to table 11.
Assume that the first quality result of the parameter measurement information in table 11 obtained in step D13 is as follows: the first quality score of 'the public model running time should be less than 2 h' is 60, the first quality score of 'the application layer model running time should be less than 1 h' is 80, the first quality score of 'the mission should be downstream and the heat of the mission should be more than 0 in nearly 30 days' is 40, 'the first quality score of' the A1-level model lifecycle should be set to be 60, 'the first quality score of' the A2-level model lifecycle should not be more than 3 years 'is 60,' the first quality score of 'the A3-level model lifecycle should not be more than 1 year' is 90, 'the first quality score of' avoiding textFile, RCFile format storage 'is 80,' the first quality score of 'the model should be more than 0 in nearly 30 days' is 60, 'and the first quality score of' the model table with the same avoiding particle size is 70.
Then the second quality result of the evaluation parameter "time consuming task to ratio" =60 × 0.8+80 × 0.2=64; the second quality result of "redundant duty ratio" =40 × 1=40. Then, the third quality result of the field sub item "computing resources" =64 × 0.6+40 × 0.4=54.4. Similarly, if the third quality result of the "storage resource" of the domain sub-item is 71.6, then the fourth quality result =54.4 × 0.4+71.6 × 0.6=64.72 of the domain to be inspected.
If the rule set of the domain to be inspected includes a plurality of domains to be inspected, the sum of the fourth weights corresponding to the plurality of domains to be inspected is 1, and if the rule set of the domain to be inspected includes one domain to be inspected, the fourth weight corresponding to the domain to be inspected is 1, as shown in table 11. Then the quality assessment parameter =64.72 x 1=64.72.
In an alternative implementation, step S74 further includes the following steps F11 to F13, and exemplarily, step F11 to F13 are subsequent to step D12.
In step F11, target metadata of the target object to be inspected that does not satisfy the preset condition is obtained.
In the embodiment of the application, the metadata of the object to be inspected, which does not meet the preset condition, is called as target metadata of the target object to be inspected.
In step F12, target parameter measurement information corresponding to the target metadata is obtained.
In step F13, a target evaluation parameter corresponding to the target parameter measurement information is obtained.
The quality inspection result may include target metadata modification guidance information in addition to the quality evaluation parameters, where the target metadata modification guidance information includes: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
Assuming that the parameter measurement information is that "the life cycle setting of the A3-level table cannot be greater than 1 year", the object to be audited is table _ a, the preset condition is less than or equal to 1 year, and assuming that the metadata of table _ a is that "the life cycle is set to 999 days", then the metadata of table _ a does not satisfy the preset condition. Then, the quality inspection results include: target object to be inspected: table _ a; target metadata: the life cycle is set to 999 days; target parameter measurement information: the life cycle setting of the A3-level table cannot be more than 1 year; target evaluation parameters: the life cycle management yield.
In an optional implementation manner, the set of domain rules to be audited further includes: and (4) communication information of the administration personnel.
Illustratively, the abatement person's communication may be user input via electronic device 12; for example, the server 11 may set beforehand the communication information of the harness person so that the user selects the communication information of the corresponding harness person from the communication information of the plurality of harness persons through the electronic device 12.
Illustratively, the warehouse quality inspection method applied to the server further comprises the following steps: and sending the quality inspection result to the electronic equipment with the communication information of the administration personnel.
Illustratively, the electronic device having the abatement person's communication may be electronic device 14 or electronic device 12.
It can be understood that, in the embodiment of the present disclosure, after the quality of the data warehouse is detected, the quality inspection result is sent to the administrative staff, so that the administrative staff can correct the problematic target metadata in time.
In an optional implementation manner, the method for inspecting the quality of the bins applied to the electronic device 14 further includes: receiving a quality inspection result which is sent by the server and obtained based on the warehouse inspection model; and displaying the quality inspection result.
In an optional implementation manner, the warehouse quality inspection method applied to the electronic device 14 further includes steps G11 to G14.
In step G11, if a change operation of changing the target metadata included in the quality audit result is detected, a request for changing the target metadata is sent to the server, where the request carries the target parameter measurement information.
In step G12, a target data governance control corresponding to the target parameter measurement information fed back by the server is received.
In the embodiment of the disclosure, the data governance control corresponding to the target parameter measurement information is referred to as a target data governance control.
For example, the data governance control may be a client or a plug-in.
For example, the corresponding relationship between the parameter measurement information and the data governance control may be preset.
In an optional implementation manner, the warehouse quality inspection method applied to the server further includes steps H11 to H13.
In step H11, a request for changing the target metadata sent by the electronic device is received, where the request carries the target parameter measurement information.
In step H12, a target data governance control corresponding to the target parameter measurement information is searched from a preset correspondence between a plurality of parameter measurement information and data governance controls.
Illustratively, the data governance control corresponding to the parameter measurement information has a combined function of performing one or more of string operation, structured data operation, data operation and logic operation on the corresponding object to be inspected.
In step H13, the target data governance control is sent to the electronic device.
In step G13, the target metadata is modified by the target data governance control to obtain corrected content.
Illustratively, the target metadata modification guidance information is a basis for the user to modify the target metadata to obtain the corrected content.
That is, the user can determine how to modify the target metadata by viewing the target metadata modification guidance information, so that the target metadata can meet the preset condition.
For example, after obtaining the correction content, the server may correct the target metadata corresponding to the object to be checked stored in the data warehouse, and the target metadata stored in the metadata warehouse.
Assuming that the server obtains the corrected content as "adding a comment to a table with empty comment", the server calls an API (Application Programming Interface) Interface of a mysql table, and the mysql table stores comments corresponding to a plurality of tables respectively; and if the annotations of the table stored in the mysql table are empty, adding corresponding annotations so as to ensure that the annotations corresponding to the tables stored in the mysql table are not empty. After the annotations of the table stored by the mysql table are changed, the annotations of the table are synchronized into the metadata warehouse and the data warehouse.
Illustratively, different data governance controls may be capable of performing different types of operations.
Exemplary types of operations that may be performed to modify target metadata via the target data governance control include, but are not limited to: at least one of a string operation, a structured data operation, a data operation, and a logical operation.
Exemplary string operations include, but are not limited to: at least one of find, replace, intercept string, splice string, delete string, reverse string, copy string, compare string, and remove duplicate values. Wherein the replacement string includes at least one of a fixed value replacement, a format conversion replacement, and a Hash replacement.
For example, the operator corresponding to the string operation may be: FND (lookup), REP _ FIX (fixed value replacement), REP _ FMT (format replacement), REP _ HAS (Hash replacement), SPLT (truncated string), CCAT (concatenated string), SDEL (deleted string), SREV (reversed string), SCPY (copied string), SCMP (compared string), DST (removed duplicate value).
Exemplary, structured data operations include, but are not limited to: at least one of a column join, a table join, a row operation, a column operation, and a sort, wherein a column join may be a dataset join, a table join may be a data table join, a row operation may be a screening row operation, a column operation may be a screening column operation, and a sort may be an ascending or descending operation.
For example, the operator corresponding to the structuring operation may be: join (dataset join), join (data table join), FLTR (filter row), FLTC (filter column), ODR (ascending, descending).
Illustratively, the data operation may be a basic operation, a basic function operation, a clustering function operation, a statistical function operation, or the like, wherein the basic operation may include at least one of addition, subtraction, multiplication, division, exponentiation, logarithmization, exponentiation of the returned number, and remainder, the basic function may include at least one of sine, cosine, tangent, absolute value, rounding, and rounding down, the clustering function may include at least one of mean, number, sum, maximum, minimum, and variance, the statistical function may include a data perspective table function, or the like.
For example, the operators corresponding to the basic operation may be ADD (ADD), MNS (subtract), MPL (multiply), DVD (divide), EXP (exponentiation), LOG (LOG), POW (power returned), and MOD (remainder), the operators corresponding to the basic function may be SIN (sine), COS (cosine), TAN (tangent), ABS (absolute value), RND (round-robin), and FLR (round-down), the operators corresponding to the clustering function may be AVG (mean), CNT (number), SUM (SUM), MAX (maximum), MIN (minimum), and VAR (variance), and the operator applied to the statistical function may be PTAB (data perspective table function).
For example, the logical operation may include at least one of equal to, not equal to, less than or equal to, and greater than or equal to, and the operator corresponding to the logical operation may be: =! =, <, > =.
In step G14, the corrected content is sent to the server.
The method is described in detail in the embodiments of the present disclosure, and the method of the embodiments of the present disclosure can be implemented by using various types of apparatuses, so that various apparatuses are provided in the embodiments of the present disclosure, and specific embodiments are described in detail below.
FIG. 8 is a block diagram illustrating a quality audit device applied to a server for a warehouse according to an example embodiment. The apparatus comprises a first obtaining module 81, a second obtaining module 82, an associating module 83 and a third obtaining module 84.
A first obtaining module 81 configured to obtain a to-be-inspected field rule set, where the to-be-inspected field rule set includes evaluation parameters corresponding to a data warehouse to be inspected and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information includes preset conditions that metadata of an object to be inspected, which belongs to a set object type, needs to satisfy, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; a second obtaining module 82, configured to obtain, based on the parameter measurement information, a meta-rule model including search information and the preset condition, where the search information is used to search for metadata of the object to be inspected, which belongs to the set object type, from a metadata repository, where the metadata of the object to be inspected stored in the data repository is stored; the association module 83 is configured to associate the parameter measurement information with the meta-rule model to obtain a multi-bin quality audit model, where the multi-bin quality audit model includes the to-be-audited field rule set and the meta-rule model associated with the parameter measurement information; a third obtaining module 84 configured to obtain a quality inspection result indicating whether the metadata of the object to be inspected satisfies the preset condition based on the warehouse quality inspection model.
In an optional implementation manner, the first obtaining module is specifically configured to: the first receiving unit is configured to receive a request for constructing the warehouse quality inspection model, which is sent by the electronic equipment; the sending unit is configured to send a preset domain rule set to the electronic equipment, wherein the domain rule set to be audited is a subset of the domain rule set; a second receiving unit configured to receive the set of domain rules to be audited fed back by the electronic device.
In an optional implementation manner, the rule set of the field to be inspected further includes a field to be inspected, a field sub-item included in the field to be inspected, and a preset evaluation manner corresponding to the parameter measurement information, where the evaluation manner includes quality results corresponding to a plurality of evaluation parameter ranges, and the evaluation parameter is an evaluation parameter corresponding to the field sub-item, and the third obtaining module is specifically configured to: the searching unit is configured to search the metadata of the object to be checked, which belongs to the set object type, from the metadata warehouse according to the searching information; the first acquisition unit is configured to detect whether the metadata of the object to be inspected meets the preset conditions contained in the meta-rule model or not to obtain a grading parameter; the second acquisition unit is configured to acquire a first quality result corresponding to an evaluation parameter range containing the scoring parameter from the evaluation mode; the first calculating unit is configured to calculate a second quality result corresponding to the evaluation parameter according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information; the second calculating unit is configured to calculate a third quality result corresponding to the field sub-item according to the second quality result corresponding to the evaluation parameter and a second weight corresponding to the evaluation parameter; the third calculating unit is configured to calculate a fourth quality result corresponding to the to-be-inspected field according to the third quality result corresponding to the field sub-item and a third weight corresponding to the field sub-item; and the fourth calculating unit is configured to calculate and obtain the quality assessment parameters corresponding to the data warehouse according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, wherein the quality inspection result comprises the quality assessment parameters.
In an optional implementation manner, the method further includes: the fourth acquisition module is configured to acquire target metadata of the target object to be inspected, which does not meet the preset condition; a fifth obtaining module, configured to obtain target parameter measurement information corresponding to the target metadata; a sixth obtaining module, configured to obtain a target evaluation parameter corresponding to the target parameter measurement information; wherein the quality inspection result further comprises target metadata modification guidance information, and the target metadata modification guidance information comprises: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
In an optional implementation manner, the set of domain rules to be audited further includes: communication information of the administration personnel; further comprising: and the sending result module is configured to send the quality inspection result to the electronic equipment with the communication information of the administration personnel. A content receiving module configured to receive the corrected content sent by the electronic device, where the corrected content includes correction information corresponding to the target metadata, and the target metadata modification guidance information is a basis for a user to modify the target metadata to obtain the corrected content.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 9 is a block diagram illustrating a bin quality audit device applied to an electronic device according to an example embodiment. The device includes: a first sending module 91, a first receiving module 92, a first displaying module 93, an acquiring set module 94 and a second sending module 95.
A first sending module 91 configured to send a request to build a bin quality audit model of a data warehouse to a server; a first receiving module 92, configured to receive a domain rule set fed back by the server, where the domain rule set includes multiple candidate evaluation parameters and candidate parameter measurement information corresponding to each candidate evaluation parameter, where the candidate parameter measurement information includes a preset condition that metadata of an object to be inspected that belongs to a set object type needs to be satisfied, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; a first display module 93 configured to display the set of domain rules; an acquiring set module 94 configured to respond to a selection operation for the domain rule set to obtain a domain rule set to be audited, where the domain rule set to be audited includes evaluation parameters selected from the domain rule set and parameter measurement information corresponding to the evaluation parameters selected from the domain rule set; a second sending module 95 configured to send the set of domain rules to be audited to the server; the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset conditions, the search information is used for searching metadata of an object to be audited which belongs to a set object type from a metadata warehouse, the metadata of the object to be audited stored in the metadata warehouse is stored, and the quality audit model comprises the field rule set to be audited and the meta-rule model associated with the parameter measurement information.
In an optional implementation manner, the method further includes: a second receiving module, configured to receive the quality audit result sent by the server, where the quality audit result includes target metadata modification guidance information and quality evaluation parameters corresponding to the data warehouse, where the quality evaluation parameters characterize whether metadata of an object to be audited in the data warehouse meets a preset condition, the target metadata modification guidance information includes at least one of target metadata, target parameter measurement information, and target evaluation parameters corresponding to the target parameter measurement information, and the target metadata does not meet the preset condition included in the target parameter measurement information; a second display module configured to display the quality inspection result on a display interface.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 10 is a block diagram illustrating an apparatus 100 for a server according to an example embodiment.
The server 11 may include one or more of the following components: a first processor 101, a first memory 102, an audio component 103, an input unit 104 and a communication component 105.
Those skilled in the art will appreciate that the configuration shown in fig. 10 is merely an example of an implementation and does not constitute a limitation on the server 11, and that the server 11 may include more or less components than those shown, or combine certain components, or a different arrangement of components.
The following describes each component of the server 11 in detail with reference to fig. 10:
alternatively, the first processor 101 is a control center of the server 11, connects various parts of the entire server 11 by using various interfaces and lines, and performs various functions of the server 11 and processes data by running or executing software programs and/or modules stored in the first memory 102 and calling data stored in the first memory 102, thereby performing overall monitoring of the server 11. Optionally, the first processor 101 may include one or more processing units; preferably, the first processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, and the like, and the modem processor mainly processes wireless communication. It will be appreciated that the above-described modem processor may not be integrated into the first processor 101.
Illustratively, the first memory 102 is configured to store various types of data to support operations at the server 11. Examples of such data include instructions for any application or method operating on the server 11, tasks and task states corresponding to the tasks, and so forth. The first memory 102 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Illustratively, the audio component 103 is configured to output and/or input audio signals. For example, the audio component 103 includes a Microphone (MIC) configured to receive external audio signals when the server 11 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the first memory 102 or transmitted via the communication component 105. In some embodiments, audio component 103 also includes a speaker for outputting audio signals, such as when previewing a video stream.
Illustratively, the input unit 104 may be used to receive information or character information input by a user and to generate key signal inputs related to user settings and function control of the server 11. Alternatively, the input unit 104 may include a touch panel 1042 and other input devices 1041. The touch panel 1042, also called a touch screen, can collect a touch operation performed by a user on or near the touch panel 1042 (e.g., an operation performed by the user on the touch panel 1042 or near the touch panel 1042 by using a finger, a stylus, or any other suitable object or accessory), and drive the corresponding connection device according to a predetermined program. Alternatively, the touch panel 1042 may include two parts, i.e., a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the first processor 101, and can receive and execute commands sent by the first processor 101. In addition, the touch panel 1042 can be implemented by various types, such as resistive, capacitive, infrared, and surface acoustic wave. The input unit 104 may include other input devices 1041 in addition to the touch panel 1042. In particular, other input devices 1041 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The communication component 105 is configured to facilitate wired or wireless communication between the server 11 and other devices. The server 11 may have access to a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 105 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 105 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the server 11 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In the embodiment of the present disclosure, the first processor 101 included in the server 11 may be a central processing unit CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement the embodiment of the present invention.
The first processor 101 has the following functions: acquiring a to-be-inspected field rule set, wherein the to-be-inspected field rule set comprises evaluation parameters corresponding to a to-be-inspected data warehouse and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, the type of the set object corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected; obtaining a meta-rule model containing search information and the preset condition based on the parameter measurement information, wherein the search information is used for searching the metadata of the object to be inspected, which belongs to the set object type, from a metadata warehouse, and the metadata of the object to be inspected, which is stored in the data warehouse, is stored in the metadata warehouse; associating the parameter measurement information with the meta-rule model to obtain a multi-bin quality inspection model, wherein the multi-bin quality inspection model comprises the to-be-inspected field rule set and the meta-rule model associated with the parameter measurement information; and obtaining a quality inspection result representing whether the metadata of the object to be inspected meets the preset condition or not based on the warehouse quality inspection model.
Fig. 11 is a block diagram illustrating an apparatus 110 for an electronic device, according to an example embodiment.
The electronic device may include one or more of the following components: a second processor 111, a second memory 112, a display unit 113, an audio component 114, an input unit 115, and a communication component 116.
Those skilled in the art will appreciate that the configuration shown in fig. 11 is merely an example of an implementation and does not constitute a limitation on electronic devices that may include more or fewer components than those shown, or that certain components may be combined, or that a different arrangement of components may be used.
The following describes each component of the electronic device in detail with reference to fig. 11:
optionally, the second processor 111 is a control center of the electronic device, connects various parts of the whole electronic device by using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the second memory 112 and calling data stored in the second memory 112, thereby performing overall monitoring of the electronic device. Optionally, the second processor 111 may comprise one or more processing units; preferably, the second processor 111 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the second processor 111.
Illustratively, the second memory 112 is configured to store various types of data to support operations at the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, tasks and task states corresponding to the tasks, and so forth. The second memory 112 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Alternatively, the display unit 113 may be used to display information input by the user or information provided to the user (e.g., a set of domain rules). The display unit 113 may include a display panel 1131, and optionally, the display panel 1131 may be configured in the form of an LCD (Liquid crystal display), an OLED (Organic Light-Emitting Diode), or the like.
Illustratively, the audio component 114 is configured to output and/or input audio signals. For example, the audio component 114 includes a Microphone (MIC) configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the second memory 112 or transmitted via the communication component 116. In some embodiments, audio component 114 also includes a speaker for outputting audio signals, such as when previewing a video stream.
For example, the input unit 115 may be used to receive information or character information input by a user and generate key signal inputs related to user settings and function control of the electronic device. Alternatively, the input unit 115 may include a touch panel 1152 and other input devices 1151. Touch panel 1152, also referred to as a touch screen, may collect touch operations by a user on or near the touch panel 1152 (e.g., operations by a user on or near touch panel 1152 using any suitable object or accessory such as a finger, a stylus, etc.) and drive the corresponding connected device according to a predetermined program. Alternatively, the touch panel 1152 may include two portions of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the second processor 111, and can receive and execute commands sent by the second processor 111. In addition, the touch panel 1152 may be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit 115 may include other input devices 1151 in addition to the touch panel 1152. In particular, other input devices 1151 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
The communication component 116 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 116 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 116 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In the embodiment of the present disclosure, the second processor 111 included in the electronic device may be a central processing unit CPU, or an Application Specific Integrated Circuit ASIC (Application Specific Integrated Circuit), or one or more Integrated circuits configured to implement the embodiment of the present invention.
The second processor 111 has the following functions: sending a request for constructing a warehouse quality inspection model of a data warehouse to a server; receiving a domain rule set fed back by the server, wherein the domain rule set comprises a plurality of candidate evaluation parameters and candidate parameter measurement information corresponding to each candidate evaluation parameter, the candidate parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, which belongs to a set object type, the set object type corresponds to the evaluation parameters, and the object to be inspected is stored in the data warehouse; displaying the set of domain rules; responding to the selection operation aiming at the domain rule set to obtain a domain rule set to be inspected, wherein the domain rule set to be inspected comprises evaluation parameters selected from the domain rule set and parameter measurement information corresponding to the evaluation parameters selected from the domain rule set; sending the rule set of the field to be inspected to the server; the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset condition, the search information is used for searching metadata of an object to be inspected, which belongs to a set object type, from a meta-data warehouse, the meta-data warehouse stores the metadata of the object to be inspected, which is stored in the data warehouse, and the multi-bin quality inspection model comprises the field rule set to be inspected and the meta-rule model associated with the parameter measurement information.
In an exemplary embodiment, the disclosed embodiment also provides a storage medium comprising instructions, for example, the first memory 102 comprising instructions, which are executable by the first processor 101 of the server 11 to perform the above method. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, the disclosed embodiment also provides a storage medium comprising instructions, for example, the second memory 112 comprising instructions, which are executable by the second processor 112 of the electronic device to perform the above method. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, the disclosed embodiments also provide a computer program product comprising one or more instructions that may be executed by the second processor 112 of the electronic device or the first processor 101 of the server to perform the above-described method.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (17)

1. A warehouse quality inspection method is applied to a server and comprises the following steps:
acquiring a to-be-inspected field rule set, wherein the to-be-inspected field rule set comprises evaluation parameters corresponding to a to-be-inspected data warehouse and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, the type of the set object corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected;
obtaining a meta-rule model containing search information and the preset condition based on the parameter measurement information, wherein the search information is used for searching the metadata of the object to be inspected, which belongs to the set object type, from a metadata warehouse, and the metadata of the object to be inspected, which is stored in the data warehouse, is stored in the metadata warehouse;
associating the parameter measurement information with the meta-rule model to obtain a multi-bin quality inspection model, wherein the multi-bin quality inspection model comprises the rule set of the field to be inspected and the meta-rule model associated with the parameter measurement information;
and obtaining a quality inspection result representing whether the metadata of the object to be inspected meets the preset condition or not based on the warehouse quality inspection model.
2. The method of claim 1, wherein the step of obtaining the set of domain rules to be audited comprises:
receiving a request for constructing the quality inspection model of the warehouse sent by the electronic equipment;
sending a preset domain rule set to the electronic equipment, wherein the domain rule set to be inspected is a subset of the domain rule set;
and receiving the rule set of the field to be audited fed back by the electronic equipment.
3. The warehouse quality inspection method according to claim 1 or 2, wherein the rule set of the domain to be inspected further includes a domain to be inspected, a domain sub-item included in the domain to be inspected, and a preset evaluation manner corresponding to the parameter measurement information, the evaluation manner includes quality results corresponding to a plurality of evaluation parameter ranges, respectively, the evaluation parameters are evaluation parameters corresponding to the domain sub-item, and the step of obtaining the quality inspection result of the data warehouse based on the warehouse quality inspection model includes:
searching the metadata of the object to be inspected, which belongs to the set object type, from the metadata warehouse according to the searching information;
detecting whether the metadata of the object to be inspected meets the preset conditions contained in the meta-rule model or not to obtain a grading parameter;
obtaining a first quality result corresponding to an evaluation parameter range containing the scoring parameter from the evaluation mode;
calculating to obtain a second quality result corresponding to the evaluation parameter according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information;
calculating to obtain a third quality result corresponding to the field sub-item according to the second quality result corresponding to the evaluation parameter and a second weight corresponding to the evaluation parameter;
calculating to obtain a fourth quality result corresponding to the field to be inspected according to the third quality result corresponding to the field sub-item and the third weight corresponding to the field sub-item;
and calculating to obtain quality evaluation parameters corresponding to the data warehouse according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, wherein the quality inspection result comprises the quality evaluation parameters.
4. The warehouse quality inspection method according to claim 3, further comprising, after the step of detecting whether the metadata of the object to be inspected satisfies the preset condition included in the meta-rule model, the steps of:
obtaining target metadata of the target object to be inspected which does not meet the preset condition;
acquiring target parameter measurement information corresponding to the target metadata;
acquiring a target evaluation parameter corresponding to the target parameter measurement information;
wherein the quality audit result further comprises target metadata modification guidance information, and the target metadata modification guidance information comprises: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
5. The bin quality audit method of claim 4 wherein the set of domain rules to be audited further comprises: communication information of the administration personnel; the warehouse quality inspection method further comprises the following steps:
sending the quality inspection result to the electronic equipment with the communication information of the administration personnel;
receiving correction content sent by the electronic equipment, wherein the correction content comprises correction information corresponding to the target metadata, and the target metadata modification guidance information is a basis for a user to modify the target metadata to obtain the correction content.
6. A warehouse quality inspection method is applied to electronic equipment and comprises the following steps:
sending a request for constructing a warehouse quality inspection model of a data warehouse to a server;
receiving a domain rule set fed back by the server, wherein the domain rule set comprises a plurality of candidate evaluation parameters and candidate parameter measurement information corresponding to each candidate evaluation parameter, the candidate parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, which belongs to a set object type, the set object type corresponds to the evaluation parameters, and the object to be inspected is stored in the data warehouse;
displaying the set of domain rules;
responding to the selection operation aiming at the field rule set to obtain a field rule set to be inspected, wherein the field rule set to be inspected comprises evaluation parameters selected from the field rule set and parameter measurement information corresponding to the evaluation parameters selected from the field rule set;
sending the domain rule set to be checked to the server;
the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset conditions, the search information is used for searching metadata of an object to be audited which belongs to a set object type from a metadata warehouse, the metadata of the object to be audited stored in the metadata warehouse is stored, and the quality audit model comprises the field rule set to be audited and the meta-rule model associated with the parameter measurement information.
7. The method of claim 6, further comprising:
receiving a quality inspection result sent by the server, wherein the quality inspection result comprises target metadata modification guidance information and quality evaluation parameters corresponding to the data warehouse, the quality evaluation parameters represent whether metadata of an object to be inspected in the data warehouse meets preset conditions or not, the target metadata modification guidance information comprises at least one of target metadata, target parameter measurement information and target evaluation parameters corresponding to the target parameter measurement information, and the target metadata does not meet the preset conditions contained in the target parameter measurement information;
and displaying the quality inspection result on a display interface.
8. A kind of quality inspection device of the number storehouse, characterized by, apply to the server, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is configured to acquire a to-be-inspected field rule set, the to-be-inspected field rule set comprises evaluation parameters corresponding to a to-be-inspected data warehouse and parameter measurement information corresponding to the evaluation parameters, the parameter measurement information comprises preset conditions which need to be met by metadata of an object to be inspected, the type of the set object corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected;
a second obtaining module, configured to obtain a meta-rule model including search information and the preset condition based on the parameter measurement information, where the search information is used to search for metadata of the object to be inspected, which belongs to the set object type, from a metadata repository, and the metadata of the object to be inspected, which is stored in the data repository, is stored in the metadata repository;
the correlation module is configured to correlate the parameter measurement information with the meta-rule model to obtain a multi-bin quality inspection model, and the multi-bin quality inspection model comprises the to-be-inspected field rule set and the meta-rule model correlated with the parameter measurement information;
and the third acquisition module is configured to acquire a quality inspection result representing whether the metadata of the object to be inspected meets the preset conditions or not based on the warehouse quality inspection model.
9. The bin quality auditing device of claim 8 where the first acquisition module is specifically configured to:
the first receiving unit is configured to receive a request for constructing the warehouse quality inspection model, which is sent by the electronic equipment;
the sending unit is configured to send a preset domain rule set to the electronic equipment, and the domain rule set to be checked is a subset of the domain rule set;
the second receiving unit is configured to receive the to-be-audited domain rule set fed back by the electronic equipment.
10. The device according to claim 8 or 9, wherein the rule set of domains to be inspected further includes domains to be inspected, domain sub-items included in the domains to be inspected, and preset evaluation manners corresponding to the parameter measurement information, the evaluation manners include quality results corresponding to a plurality of evaluation parameter ranges, the evaluation parameters are evaluation parameters corresponding to the domain sub-items, and the third obtaining module is specifically configured to:
the searching unit is configured to search the metadata of the object to be checked, which belongs to the set object type, from the metadata warehouse according to the searching information;
the first acquisition unit is configured to detect whether the metadata of the object to be inspected meets the preset conditions contained in the meta-rule model or not to obtain a grading parameter;
the second acquisition unit is configured to acquire a first quality result corresponding to an evaluation parameter range containing the scoring parameter from the evaluation mode;
the first calculating unit is configured to calculate a second quality result corresponding to the evaluation parameter according to the first quality result corresponding to the parameter measurement information and the first weight corresponding to the parameter measurement information;
the second calculating unit is configured to calculate a third quality result corresponding to the field sub-item according to the second quality result corresponding to the evaluation parameter and a second weight corresponding to the evaluation parameter;
the third calculating unit is configured to calculate a fourth quality result corresponding to the to-be-inspected field according to the third quality result corresponding to the field sub-item and a third weight corresponding to the field sub-item;
and the fourth calculating unit is configured to calculate and obtain the quality assessment parameters corresponding to the data warehouse according to the fourth quality result corresponding to the field to be inspected and the fourth weight corresponding to the field to be inspected, wherein the quality inspection result comprises the quality assessment parameters.
11. The warehouse quality audit device of claim 10, further comprising:
the fourth acquisition module is configured to acquire target metadata of the target object to be inspected which does not meet the preset conditions;
a fifth obtaining module, configured to obtain target parameter measurement information corresponding to the target metadata;
a sixth obtaining module, configured to obtain a target evaluation parameter corresponding to the target parameter measurement information;
wherein the quality audit result further comprises target metadata modification guidance information, and the target metadata modification guidance information comprises: at least one of the target object to be inspected, the target metadata, the target parameter measurement information and the target evaluation parameter.
12. The bin quality audit device according to claim 11 wherein the set of domain rules to be audited further comprises: communication information of the administration personnel; further comprising:
a sending result module configured to send the quality inspection result to an electronic device having communication information of the administration staff;
a content receiving module configured to receive corrected content sent by the electronic device, where the corrected content includes correction information corresponding to the target metadata, and the target metadata modification guidance information is a basis for a user to modify the target metadata to obtain the corrected content.
13. The utility model provides a storehouse quality inspection device which characterized in that, is applied to electronic equipment, includes:
a first sending module configured to send a request to build a bin quality audit model of a data warehouse to a server;
a first receiving module, configured to receive a domain rule set fed back by the server, where the domain rule set includes multiple candidate evaluation parameters and candidate parameter measurement information corresponding to each of the candidate evaluation parameters, where the candidate parameter measurement information includes a preset condition that metadata of an object to be inspected that belongs to a set object type needs to be satisfied, the set object type corresponds to the evaluation parameters, and the data warehouse stores the object to be inspected;
a first display module configured to display the set of domain rules;
the system comprises an acquisition set module, a check module and a check module, wherein the acquisition set module is configured to respond to selection operation aiming at the field rule set and acquire a field rule set to be checked, and the field rule set to be checked comprises evaluation parameters selected from the field rule set and parameter measurement information corresponding to the evaluation parameters selected from the field rule set;
the second sending module is configured to send the rule set of the field to be audited to the server;
the parameter measurement information is a basis for obtaining a meta-rule model, the meta-rule model comprises search information and the preset condition, the search information is used for searching metadata of an object to be inspected, which belongs to a set object type, from a meta-data warehouse, the meta-data warehouse stores the metadata of the object to be inspected, which is stored in the data warehouse, and the multi-bin quality inspection model comprises the field rule set to be inspected and the meta-rule model associated with the parameter measurement information.
14. The warehouse quality audit device of claim 13, further comprising:
a second receiving module, configured to receive a quality audit result sent by the server, where the quality audit result includes target metadata modification guidance information and quality evaluation parameters corresponding to the data warehouse, where the quality evaluation parameters characterize whether metadata of an object to be audited in the data warehouse meets preset conditions, the target metadata modification guidance information includes at least one of target metadata, target parameter measurement information, and target evaluation parameters corresponding to the target parameter measurement information, and the target metadata does not meet the preset conditions included in the target parameter measurement information;
a second display module configured to display the quality inspection result on a display interface.
15. A server, comprising:
a first processor;
a first memory to store the first processor-executable instructions;
wherein the first processor is configured to execute the instructions to implement the bin quality audit method of any of claims 1-5.
16. An electronic device, comprising:
a second processor;
a second memory for storing the second processor-executable instructions;
wherein the second processor is configured to execute the instructions to implement the bin quality audit method of any of claims 6 to 7.
17. A computer readable storage medium, instructions in the storage medium, when executed by a first processor of a server, enable the server to perform a method of quality audit of a bin according to any of claims 1 to 5; or, when executed by a second processor of an electronic device, enable the electronic device to perform the bin quality audit method of any of claims 6 to 7.
CN202110819727.2A 2021-07-20 2021-07-20 Method, device, medium and product for inspecting quality of warehouse Active CN113535701B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110819727.2A CN113535701B (en) 2021-07-20 2021-07-20 Method, device, medium and product for inspecting quality of warehouse

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110819727.2A CN113535701B (en) 2021-07-20 2021-07-20 Method, device, medium and product for inspecting quality of warehouse

Publications (2)

Publication Number Publication Date
CN113535701A CN113535701A (en) 2021-10-22
CN113535701B true CN113535701B (en) 2022-11-04

Family

ID=78128968

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110819727.2A Active CN113535701B (en) 2021-07-20 2021-07-20 Method, device, medium and product for inspecting quality of warehouse

Country Status (1)

Country Link
CN (1) CN113535701B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103473672A (en) * 2013-09-30 2013-12-25 国家电网公司 System, method and platform for auditing metadata quality of enterprise-level data center
CN110851539A (en) * 2019-10-25 2020-02-28 东软集团股份有限公司 Metadata verification method and device, readable storage medium and electronic equipment
CN112395325A (en) * 2020-11-27 2021-02-23 广州光点信息科技有限公司 Data management method, system, terminal equipment and storage medium
CN112667619A (en) * 2020-12-30 2021-04-16 平安证券股份有限公司 Method and device for auxiliary checking data, terminal equipment and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10318501B2 (en) * 2016-10-25 2019-06-11 Mastercard International Incorporated Systems and methods for assessing data quality

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103473672A (en) * 2013-09-30 2013-12-25 国家电网公司 System, method and platform for auditing metadata quality of enterprise-level data center
CN110851539A (en) * 2019-10-25 2020-02-28 东软集团股份有限公司 Metadata verification method and device, readable storage medium and electronic equipment
CN112395325A (en) * 2020-11-27 2021-02-23 广州光点信息科技有限公司 Data management method, system, terminal equipment and storage medium
CN112667619A (en) * 2020-12-30 2021-04-16 平安证券股份有限公司 Method and device for auxiliary checking data, terminal equipment and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
元数据管理系统在电力企业的研究与实践;衡星辰等;《自动化与仪器仪表》;20170430;全文 *

Also Published As

Publication number Publication date
CN113535701A (en) 2021-10-22

Similar Documents

Publication Publication Date Title
US20210224693A1 (en) Systems and Methods for Predictive Coding
US9582556B2 (en) Automatic generation of an extract, transform, load (ETL) job
US10042911B2 (en) Discovery of related entities in a master data management system
US11640583B2 (en) Generation of user profile from source code
EP3566191A1 (en) Master view of tasks
US9639353B1 (en) Computing quality metrics of source code developers
CN110990445A (en) Data processing method, device, equipment and medium
CN112015912A (en) Intelligent index visualization method and device based on knowledge graph
Sun et al. Mining software repositories for automatic interface recommendation
CN113535701B (en) Method, device, medium and product for inspecting quality of warehouse
CN117829657A (en) Spacecraft system evaluation system and evaluation method
Lerman et al. Language, demographics, emotions, and the structure of online social networks
US11294558B2 (en) Interactive user interface for regression systems that process distorted or erroneous data obtained from an environment
CN104750609B (en) Determine the method and device of interface layout compatibility
CN106649880B (en) Power statistics management system and method
Hu et al. Large-scale location prediction for web pages
Liu et al. A Novel Approach Based on Kano Model, Interval 2‐Tuple Linguistic Representation Model, and Prospect Theory for Apperceiving Key Customer Requirements
Koçi et al. PatternLens: Inferring evolutive patterns from web API usage logs
AU2014101413B4 (en) Information displaying method and apparatus
CN114547231A (en) Data tracing method and system
US11455639B2 (en) Unsupervised universal anomaly detection for situation handling
Domínguez-Mayo et al. Quality issues on model-driven web engineering methodologies
CN113590442B (en) Host performance detection method and device, electronic equipment and storage medium
CN116010790A (en) Feature management method, device, computer equipment, storage medium and product
US20230359606A1 (en) Methods and systems for connecting data with non-standardized schemas in connected graph data exchanges

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant