CN116069770A

CN116069770A - Method, apparatus and computer-readable storage medium for checking object

Info

Publication number: CN116069770A
Application number: CN202310046309.3A
Authority: CN
Inventors: 苍洁; 黄健; 张淼; 陶元旭; 王桃桃; 谭静
Original assignee: China International Financial Ltd By Share Ltd
Current assignee: China International Financial Ltd By Share Ltd
Priority date: 2023-01-13
Filing date: 2023-01-13
Publication date: 2023-05-05

Abstract

The invention discloses a checking method for checking objects, which comprises the following steps: acquiring a checking task aiming at a checking object; acquiring a plurality of attribute data of a checking task; calculating the execution priority weight of the checking task based on at least part of attribute data in the plurality of attribute data; and adding the checking tasks into a checking task sequence according to the execution priority weights, and sequencing the checking tasks by the checking task sequence according to the execution priority weights of the checking tasks so as to execute the checking tasks according to the sequence of the checking task sequence. Corresponding apparatus and computer readable storage medium, etc. are also disclosed.

Description

Method, apparatus and computer-readable storage medium for checking object

Technical Field

The present disclosure relates to the field of data, and in particular, to a method, apparatus, and computer-readable storage medium for checking objects.

Background

In recent years, with the advent of the data age, data information has been developed in a large amount and in a variety. Data anomalies may result in serious losses or accidents. Along with the rapid development of enterprise business and the requirement of comprehensive digital transformation, the development of aggressive and effective data management work has become a common consensus of various industries. The data quality management work occupies the core position of the data management work, is an important basis for whether the data management work can exert service value, and the data quality problem is becoming a cost factor of enterprises or even causes related service interruption.

In some enterprise business applications, data quality rules are utilized to periodically check against a check object to check for degradation in data quality or changes in data quality over time of the check object, and if certain minimally required Key Performance Indicators (KPIs) of data quality are no longer met, pre-warning is performed to take appropriate action.

Inefficiency in the inspection of inspected objects can affect the efficiency of business applications for enterprises, particularly for the financial industry, which is sensitive to data quality, which can cause immeasurable losses to the enterprises.

Disclosure of Invention

In view of the above, the present disclosure provides methods, apparatus, and computer-readable storage media for inspection of inspected objects that may alleviate, mitigate, or even eliminate at least the above-mentioned problems.

According to a first aspect of the present disclosure, there is provided a checking method for checking an object, including the steps of: acquiring a checking task aiming at a checking object; acquiring a plurality of attribute data of a checking task; calculating the execution priority weight of the checking task based on at least part of attribute data in the plurality of attribute data; and adding the checking tasks into a checking task sequence according to the execution priority weights, and sequencing the checking tasks by the checking task sequence according to the execution priority weights of the checking tasks so as to execute the checking tasks according to the sequence of the checking task sequence.

In one embodiment, at least a portion of the attribute data includes a first indicator value indicating how frequently the quality problem occurs for a first specified time period of the inspected object within a first time period and a second indicator value indicating how frequently the inspected object is invoked by the inspected object consumer for a second specified time period of the inspected object within a second time period, the first time period being no less than the first specified time period, the second time period being no less than the second specified time period, and both the first time period and the second time period including the current time.

In one embodiment, at least some of the attribute data further includes a latency of the checkmark task.

In one embodiment, the priority weight calculation is performed as:

T _np ＝αT _nw +βQP _f +γUO _f ，

wherein T is _np Is the execution priority weight of the checking task, T _nw Is the wait time of the check task, QP _f Is a first indication value, UO _f Is a second indicator value, α is the weight of the waiting time of the checking task, β is the weight of the first indicator value, and γ is the weight of the second indicator value.

In one embodiment, when the number of times the quality problem occurs for the inspected object over the first specified time period is above a first threshold, or is continuously increasing, or is increasing in speed above a second threshold, the weight of the first indicator value is increased to affect the ordering of the inspected task in the inspected task sequence, wherein the period of time includes a plurality of first specified time periods.

In one embodiment, when the number of times the inspected object is invoked by the inspected object consumer over a second specified time period is above a third threshold, or is continuously increasing, or is increasing in speed above a fourth threshold, the weight of the second indicator value is increased to affect the ordering of the inspected tasks in the inspected task sequence, wherein the period of time includes a plurality of second specified time periods.

In one embodiment, the first indicator value and the second indicator value take values of dimensionless values determined relative to corresponding statistics of a period of time, the period of time being a period of time between when the inspected object is online and a current time.

In one embodiment, the first indicator value is determined as follows:

determining a first time number, wherein the first time number is the number of times that quality problems occur in a first appointed time period of a checked object in a first time period;

determining a second number of times, wherein the second number of times is the average number of times that the quality problem occurs in a first designated time period in a time period from the time when the checked object comes on line to the current time; and

the first number of times and the second number of times are compared, and a value of the first indication value is assigned based on a result of the comparison.

In one embodiment, assigning the first indication value further comprises: determining to assign a first indicator value in a first value interval in response to the first number being greater than the second number, determining to assign a first indicator value in a second value interval in response to the first number being equal to the second number, and determining to assign a first indicator value in a third value interval in response to the first number being less than the second number, wherein the value in the first value interval > the value in the second value interval > the value in the third value interval.

In one embodiment, the second indicator value is determined as follows: determining a third time, wherein the third time is the time of the inspected object being called by the inspected object consumption end in a second designated time period in a second time period; determining a fourth number, wherein the fourth number is the average number of times that the check object is called by the check object consumption end in a second designated time period in a time period from the time when the check object is on line to the current time; and comparing the third times with the fourth times, and assigning a value to the second indicator value based on the result of the comparison.

In one embodiment, assigning the second indicator value further comprises: determining to assign the second indicator value in the fourth numerical interval in response to the third number being greater than the fourth number, determining to assign the second indicator value in the fifth numerical interval in response to the third number being equal to the fourth number, and determining to assign the second indicator value in the sixth numerical interval in response to the third number being less than the fourth number, wherein: values in the fourth value interval > values in the fifth value interval > values in the sixth value interval.

In one embodiment, assigning further comprises: and in the determined numerical value interval, performing assignment further based on the importance of the checked object.

In one embodiment, the plurality of attribute data includes status information of the inspection task indicating whether the inspection task has an execution condition, the method further comprising: the following steps are triggered only when the check task has an execution condition: calculating the execution priority weight of the checking task; and adding the checking task into the checking task sequence according to the execution priority weight.

In one embodiment, the first specified time period and the second specified time period comprise: one working day.

In one embodiment, the checktasks include a plurality of checksubtasks, the method further comprising: and distributing a plurality of threads to the checking tasks according to the number of the checking sub-tasks so as to execute the checking sub-tasks in parallel.

In one embodiment, the method further comprises: and allocating an accompanying thread, wherein the accompanying thread is used for recording an execution result log of the checking task, and the accompanying thread is started synchronously with the number of threads.

In one embodiment, the plurality of attribute data includes: the preparation time of the checking task, which indicates the start time of the checking task with the execution condition, wherein the waiting time of the checking task is calculated as: the ready time for the checking task is subtracted from the current time.

In one embodiment, the method further comprises: setting a triggering condition in time to trigger polling for collecting an execution result of the checking task; determining whether the acquisition of the execution result is currently being executed or not in response to the trigger condition being satisfied; in response to determining that there is no collection of execution results currently being executed, collection of execution results of the inspection task is performed.

According to a second aspect of the present disclosure, there is provided a checking apparatus for checking an object, comprising: one or more processors; and one or more memories configured to have stored thereon computer-executable instructions that, when executed in one or more processors, cause implementation of a method according to the above aspects of the present disclosure.

According to a third aspect of the present disclosure there is provided a computer readable storage medium having instructions stored therein which, when run on one or more computers, cause the computers to implement a method according to the above aspects of the present disclosure.

In the whole, by setting and calculating the execution priority weights based on some attribute data, the attribute data can influence the execution priority of the checking task, so that the execution sequence of the checking task is more scientific and reasonable, and the execution sequence can be customized based on specific attribute data. Further advantages with respect to embodiments of the present disclosure will be further described in the detailed description section.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.

Fig. 1 illustrates an exemplary implementation environment diagram to which the inspection for the inspected object can be applied provided by the embodiments of the present disclosure.

Fig. 2 illustrates an overall flowchart of a method of checking for a check object according to an embodiment of the present disclosure.

Fig. 3 illustrates a further flowchart of a method of checking for a check object, according to an embodiment of the present disclosure.

Fig. 4a illustrates yet a further flowchart of a method of checking for a check object according to an embodiment of the present disclosure.

Fig. 4b illustrates yet a further flowchart of a method of checking for a check object according to an embodiment of the present disclosure.

Fig. 5 illustrates yet a further flowchart of a method of checking for a check object, according to an embodiment of the present disclosure.

Fig. 6 illustrates a hardware environment diagram related to a checkup for a checkup object according to an embodiment of the present disclosure.

Detailed Description

For the purposes of clarity, technical solutions and advantages of the present disclosure, the following further details the embodiments of the present disclosure with reference to the accompanying drawings.

It will be understood that the terms "first," "second," and the like, as used in this disclosure, may be used herein to describe various concepts, but are not limited by these terms unless otherwise specified. These terms are only used to distinguish one concept from another. For example, the first indicator value can be referred to as a second indicator value, or the second indicator value can be referred to as the first indicator value, without departing from the scope of the present disclosure. The terms "first," "second," and the like, do not denote any order or importance, nor the order of steps.

The terms "each," "plurality," and the like, as used in this disclosure, include two or more, each referring to each of the corresponding plurality.

The term "checking" as used in this disclosure includes the meaning of checking, verifying, detecting.

The term "database" as used in this disclosure is a data storage system used to provide data storage for various transaction management. In a broad sense, all entities that can provide data information can be considered a database.

Fig. 1 is a schematic diagram of an implementation environment 100 to which a method for checking a checking object provided by an embodiment of the present disclosure can be applied, and referring to fig. 1, the implementation environment may include a management end 110 operated by an administrator 150, a checking object end 120, and a data quality management platform 130, and may further include a consumption end 160 of the checking object operated by a user 170. The management terminal 110 and the check object terminal 120 are respectively connected with the data quality management platform 130 through a wireless network or a wired network 140. The consumer end 160 of the check object may be connected to the check object end 120 and the data quality management platform 130 through a wireless network or a wired network 140, respectively.

The management end 110 may be a terminal device such as a smart phone, a tablet computer, a portable computer, a special purpose computer, etc. The management end 110 may be a server-side device, and may be an independent physical system, a system cluster or a distributed system formed by a plurality of physical systems, or a cloud system that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and an artificial intelligence platform.

The management side 110 installs and runs an application program that can operate or send management instructions to the data quality management platform 130. The administrator 110 is illustratively a device used by an administrator in which an administrator account is logged into an application running. Note that "administrator" herein is to be understood in a broad sense, and may be a person, or any functional entity capable of operating an application, for example, a functional entity that triggers an operation of an application program based on an analysis of the operation behavior of the person. Those skilled in the art will appreciate that the number of management terminals 110 described above in an implementation environment may be greater or lesser. For example, the management end 110 may be only one, or several, etc. Embodiments of the present disclosure are not limited to the number of management ends 110 and the type of equipment.

The user can perform parameter configuration, data source configuration, detail configuration of checking results and the like of the data quality management platform 130 through the management end, and auxiliary support necessary for daily operation of the data quality management platform 130 is provided.

In one embodiment, the parameter configuration includes operations for operation and maintenance management such as adding, deleting, freezing, modifying, version managing, etc. the parameter types.

In one embodiment, the data source configuration includes a configuration of data source parameters and a configuration of maximum connection numbers. Configuration of the data source parameters includes, for example, providing data source link parameters including database drive names, link addresses, authorized usernames, passwords, and the like. Providing a configuration of the maximum number of connections of the data source may avoid operational disturbances to the examination object.

In one embodiment, the detailed configuration of the inspection results supports customized configuration of the inspection results for different inspection objects and categories of different inspection objects, including, for example: and configuring message header attribute aliases, data display column widths, data display sequences, data visibility and the like of the checking results. In general, the message header of the checking result contains attribute information of the checking result, and the message header information has large difference for different checking objects and different categories of the checking objects.

The check object 120 may be a server-side device, and may be an independent physical system, or may be a system cluster or a distributed system formed by a plurality of physical systems, or may be a cloud system that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and an artificial intelligence platform. The check object 120 may also be a terminal device such as a smart phone, a tablet computer, a portable computer, or a special purpose computer.

The check object side 120 is a functional entity that generates or stores a check object, and the check object side 120 may include a database, a business application of an enterprise, and the like. Those skilled in the art will appreciate that the number of check object sides 120 described above in the implementation environment 100 may be greater or lesser. For example, the number of the check object terminals 120 may be only one, or several, or the like. Embodiments of the present disclosure do not limit the number of check object sides 120 and the type of device.

The data quality management platform 130 is generally a server-side device, and may be an independent physical system, a system cluster or a distributed system formed by a plurality of physical systems, or a cloud system providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Of course, the possibility that the data quality management platform 130 is a terminal device cannot be completely excluded.

In one embodiment, the implementation environment 100 also includes a consumer end 160 of the check object, which is operated by a user 170. The consumer 160 may be a terminal device such as a smart phone, tablet, portable computer, special purpose computer, etc. The consumer 160 may be a server-side device, and may be an independent physical system, a system cluster or a distributed system formed by a plurality of physical systems, or a cloud system that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a distribution network (Content Delivery Network, CDN), and basic cloud computing services such as big data and an artificial intelligence platform.

The consumer end 160 installs and runs an application that can access the check object end 120. Illustratively, the consumer 160 is a device used by a user in which a user account is logged into a running application. Those skilled in the art will appreciate that the number of consumer ends 160 implementing the above described check objects in environment 100 may be greater or lesser. For example, the consumer end 160 of the check object may be only one, or a few, or the like. Embodiments of the present disclosure do not limit the number and device types of consumer ends 160 of the check object.

In one embodiment, the management side 110 may be in the same device as the data quality management platform 130. In one embodiment, even the management side 110 is in the same device as the data quality management platform 130, the check object side 120. The management end 110 and the data quality management platform 130, and the check object end 120 and the data quality management platform 130 may be directly or indirectly connected through wired or wireless communication, which is not limited in this disclosure.

The cloud computing (cloud computing) mentioned above is a computing mode that distributes computing tasks over a resource pool of a large number of computers, enabling various application systems to acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the cloud are infinitely expandable in the sense of users, and can be acquired at any time, used as needed, expanded at any time and paid for use as needed. The cloud computing resource pool mainly comprises computing equipment (as a virtualized machine, comprising an operating system), storage equipment and network equipment, wherein PaaS (Platform as a Service ) layers can be deployed on the IaS (Infrastructure as a Service ) layers according to logic function division, saaS (Software as a Service ) layers can be deployed on the PaaS layers, saaS can be directly deployed on the IaS as a software running platform, such as a database, a web container and the like.

Network 140 may include, but is not limited to, a wide area network, a local area network, a wired network, a wireless network, or any combination thereof.

In some embodiments of the present disclosure, the inspection method for inspecting an object may be implemented by the management side 110 and the data quality management platform 130 together as an execution body. In other embodiments of the present disclosure, the inspection method for inspecting the object may be implemented solely by the data quality management platform 130.

Fig. 2 illustrates an overall flowchart of a method of checking for a check object according to an embodiment of the present disclosure. The method may be applied to a data system such as the implementation environment described above in connection with fig. 1, which includes a management end 110, a check object end 120, and an information platform 130, and may further include a consumption end 160 of the check object. An overall flowchart of an inspection method for inspecting an object in this implementation environment is described below in conjunction with fig. 2.

The overall flow diagram may include four phases: defining the inspection task related data 210, scheduling the execution of the inspection task 220, collecting the inspection result 230, and analyzing the inspection result 240.

First, in step 210, check task related data is defined.

In one embodiment, defining the inspection task related data includes categorizing the inspection object. The classification of the categories is based on industry data, business rule characteristics, etc., and table 1 below gives some examples of several classified major categories and specific problems below the major categories, although the specific problems in the following table are merely exemplary in nature and are not intended to limit the disclosure in any way.

TABLE 1 class of check objects

In one embodiment, defining the inspection task related data includes partitioning dimensions of the quality of the inspection object. For example, the dimension may include one or more of the following: accuracy, relevance, timeliness, consistency, availability, and uniqueness.

The accuracy is, for example, correctness of data value or calculation, the relevance is, for example, index relation, function relation and the like of retrieval, the consistency is, for example, data consistency related to data synchronization or comparison, ring ratio caliber data calculation and the like, including but not limited to consistency in aspects of data coding, naming, meaning and the like, the timeliness is, for example, timeliness of periodic data update or timeliness of data transmission, the availability is, for example, real-time availability, including whether data can be downloaded, invoked and the like, and the uniqueness is, for example, data uniqueness in data call display, data calculation value, main data management and the like.

In one embodiment, the determination of the metric dimension may be based on the business rules for which the inspection is directed to collect, classify, abstract and summarize, and adapt to the application scenario. For example, for search services, the relevance of search results is more important, and if combined with a download scenario, the availability of search results also needs to be considered.

In a further example, defining the inspection task related data further includes small granularity partitioning of the dimensions and defining corresponding metric rules. For example, for accuracy, it can be further divided into smaller granularity: completely accurate, substantially accurate, generally accurate, inaccurate, the metrology rules may be: determining that the inspected object is completely accurate in response to the inspected object being 100% accurate; determining that the inspected object is substantially accurate in response to the inspected object being less than 100% and more than 80% accurate; in response to the check object being less than 80% and more than 60% accurate, determining the check object as generally accurate; in response to the check object being less than 60% accurate, the check object is determined to be inaccurate. "above" in this context includes the number, i.e. for example above 80% includes 80% itself.

In one embodiment, the determination of the above-described metric rules may be authorized for operation by an administrator. The administrator can define the measurement rules according to various factors such as different businesses, different classes of checked objects, different application scenes and the like.

In one embodiment, defining the inspection task related data includes defining an inspection method for the inspection object.

The check object, which is referred to herein as a carrier for describing content of data quality problems of a business entity based on metric rules, is generally a business unit of a checkable granularity, and may include one or more information items, for example, the check object may be a data source change, including three information items of field type, data content and data format.

Different checking methods can be defined for different information items. The checking method is the main body of the checking task. For example, for field type changes, the method of checking is to determine whether it reports, for data content, whether it conforms to the data definition, for data format, whether it conforms to the defined data length, and so on.

Each checking method defines a specific checking execution script and associated attribute information according to the checking object. In one embodiment, the data quality management platform 130 provides a configuration template for the audit execution script to facilitate the definition of a particular audit execution script. The management end 110 can present a visual interface of the configuration template of the checking execution script and import the template to automatically generate the checking method, and the configuration template of the checking execution script can also automatically check the correctness of the checking method, identify the checking execution script with configuration errors and provide error information inquiry.

The definition of the above-mentioned related data of the checking task (such as the category of the checking object, the dimension of the quality of the checking object, the small granularity of dividing the dimension, the corresponding measurement rule, the checking method, etc.), and the subsequent management can be input by the administrator through the management end shown in fig. 1. Of course, it may also be automatically generated by system initialization, or in any other reasonable manner, which is not limited by the present disclosure.

At step 220, a checkup task is scheduled for execution. This step is the core step performed at the data quality management platform 130. And generating corresponding checking results by executing checking tasks to obtain problem data, wherein the problem data can directly reflect the quality problems of the data, and particularly comprises the quality problems of the data concerned by the consumption end of the checked object.

Fig. 3 illustrates a further flowchart of a method of checking for a check object, i.e., a flowchart of a method 300 of scheduling execution of a check task, in accordance with an embodiment of the present disclosure. The method 300 may be applied to a data system including the management side 110, the check object side 120, and the information platform 130 in the implementation environment described above in connection with fig. 1, and may further include the consumption side 160 of the check object. A flowchart of a method of moderately performing a checkup task in a data system is described below in conjunction with fig. 3 as a whole.

In step 2201, a check task for a check object is acquired. In one embodiment, each checking task is uniformly scheduled and managed, for example, the checking task is automatically generated in a working period of a business application of an enterprise where the checking object is located, so as to be scheduled and executed. A specific scheduling manner will be described below.

In step 2202, a plurality of attribute data of the inspection task is acquired. The plurality of attribute data may include one or more of:

tn: the identification number of the checking task is used for identifying the checking task by a unique mark;

tns: the current state information of the checkup task, in one embodiment, indicates whether the checkup task is conditioned (e.g., whether it is to be premised on the execution of another checkup task) such that, in a further example, the execution of

steps

2203 and 2204 is triggered only when the checkup task is conditioned. In another example, the current state includes waiting (e.g., represented by the numeral "1"), ready (e.g., represented by the numeral "0"), and completed (e.g., represented by the numeral "2"), wherein waiting indicates that the checkpointed task does not have an execution condition, e.g., the execution of the checkpointed task is premised on the execution of another checkpointed task that has not yet been generated or executed, and ready indicates that the checkpointed task has an execution condition, completion indicates that the checkpointed task has completed execution;

Tnr: the preparation time of the checking task, which indicates the starting time of the checking task with the execution condition;

tnw: the latency of the check task, it is understood that the latency of the check task can be calculated as: subtracting the preparation time Tnr of the checking task from the current time;

QPf: an indicator of frequency, herein referred to as a first indicator, for the purpose of distinguishing, indicating how frequently a quality problem has occurred in a first specified time period (e.g., a workday) of a subject during a first time period;

UOf: another indicator of frequency, referred to herein as a second indicator value for the purpose of distinguishing, indicates how frequently the subject is invoked by the subject consumer during a second specified time period (e.g., two weekdays) within a second time period. The first specified time period and the second specified time period may be the same or different.

The first time period is not less than the first specified time period, typically is an integer multiple of the first specified time period, the second time period is not less than the second specified time period, typically is an integer multiple of the second specified time period, and the first time period and the second time period include the current time, which represents timeliness. The first time period may be the same as the second time period or may be different.

In one embodiment, the first indication value and the second indication value are non-dimensionalized values determined with respect to corresponding statistics of a period of time, where the period of time is preferably a period of time between when the examination object is on line and a current time.

Referring to fig. 4a, in a further example, the first indication value is determined as follows: first, in step 22021, a first number of times and a second number of times are determined, where the first number of times is the number of times the quality problem occurs in the inspection object in a first specified time period, which is a time period including the current time (for distinction from a time period hereinafter referred to as a first time period), for example, a day of the last week, or a day of the last three days, or even a day of the last day (i.e., the first time period is equal to the first specified time period). For the first number determination, it is possible to count the number of times the quality problem occurs in the first period, and then calculate the number of times the quality problem occurs in the first specified period in proportion, so this is an average number of times. For the first time number determination, a time period within the above-mentioned time period may also be selected, and the number of times of occurrence of the quality problem within the time period is directly counted, which is not intended to be any limitation in the present disclosure. And the second number is an average number of times that the quality problem occurs in the first specified time period in a time period between the time when the check object comes on line and the current time; then, in step 22022, the first and second times are compared, and the first indication value is assigned based on the result of the comparison.

In a further example, assigning the first indication value further includes: determining to assign a first indicator value in a first value interval in response to the first number being greater than the second number, determining to assign a first indicator value in a second value interval in response to the first number being equal to the second number, and determining to assign a first indicator value in a third value interval in response to the first number being less than the second number, wherein:

values in the first value interval > values in the second value interval > values in the third value interval.

For example, the non-dimensionalized value may take the number in the interval [0, 10], the value in the interval [8, 10] when the first number is greater than the corresponding statistical value of the period of time, i.e., the second number, the value in the interval [4,8 ] when the first number is equal to the second number, and the value in the interval [0, 4) when the first number is less than the second number.

Illustratively, the specific value in each interval may also be determined based on the importance of the check object, e.g., in interval [8, 10], the value is 10 for the check object of the important business. In a further example, the specific value may be an integer such that the interval of [0, 10] is substantially divided into three intervals of [0,3], [4,7] and [8, 10 ].

Similarly, for the second indicator value, a similar processing manner is adopted to assign the value, that is, the corresponding statistical value of a period of time is also adopted as a reference to determine a dimensionless value, and particularly, the average number of times called by the consumer end of the inspected object in a second designated time period (for example, a working day) from the time when the inspected object is online is adopted as a reference.

Specifically, referring to fig. 4b, the above second indicator value is determined as follows: first, in step 22023, a third number of times and a fourth number of times are determined, where the third number of times is the number of times the check object is invoked by the check object consumer during a second specified time period, and in order to represent timeliness, this second specified time period is a time period (for distinguishing from the above, referred to herein as a second time period) that includes the current time, such as one day of the last week, or one day of the last three days, or even one day of the last day (i.e., the second time period is equal to the second specified time period). The second period of time may be equal to or different from the first period of time described above. For the third number of times, the number of times the consumer end of the checked object calls in the second period of time can be counted, and then the number of times the consumer end of the checked object calls in the second designated period of time can be calculated proportionally, so that the average number of times is obtained. For the third number of times of determination, a time period in the second time period may also be selected, and the number of times of the consumer call of the checked object in the time period is directly counted, which is not intended to be limited in any way by the present disclosure. The fourth time is the average time that the check object is called by the check object consumption end in a second designated time period in the time period from the time when the check object is on line to the current time; then, in step 22024, the third number of times and the fourth number of times are compared, and the second indicator value is assigned based on the result of the comparison.

It will be appreciated that steps 22021-22022, and steps 22023-22024 may be performed simultaneously or sequentially, and the disclosure is not intended to be limited in this regard.

In a further example, assigning the second indicator value further includes: determining to assign the second indicator value in the fourth numerical interval in response to the third number being greater than the fourth number, determining to assign the second indicator value in the fifth numerical interval in response to the third number being equal to the fourth number, and determining to assign the second indicator value in the sixth numerical interval in response to the third number being less than the fourth number, wherein:

values in the fourth value interval > values in the fifth value interval > values in the sixth value interval.

For example, the non-dimensionalized value may take the number in the interval [0, 10], the value in the interval [8, 10] when the third number is greater than the corresponding statistical value of the period of time, that is, the fourth number, the value in the interval [4,8 ] when the third number is equal to the fourth number, and the value in the interval [0, 4) when the third number is less than the fourth number.

It will be appreciated that the division of the intervals of non-dimensionalized value for the second indicator value and the sub-intervals therein may be the same as or different from the division of the intervals of non-dimensionalized value for the first indicator value and the sub-intervals therein described above.

The first indication value and the second indication value of the checked object are affected by multiple complex factors such as data grabbing, filling, cleaning, library dropping, dynamic calculation, service change, code iteration or even data transmission, and the like, and generally do not accord with normal distribution. The present disclosure uses the corresponding statistics of a period of time as a reference to determine a dimensionless value, and in particular uses the average number of quality problems occurring within a specified time period (e.g., one working day) since the inspection object was online, and is of further reference value. The inventor of the present application considers that the above corresponding statistics value gradually converges with the detection and the solution of the quality problem, and thus has a reference value.

In step 2203, an execution priority weight of the inspection task is calculated based on at least some of the plurality of attribute data.

The execution priority weights of the checking tasks are sequenced to obtain a checking task sequence, so that the data quality management platform executes the checking tasks according to the sequence of the checking task sequence. In one embodiment, the greater the execution priority weight value, the more preferred the execution is represented, the more the corresponding checktasks are ordered in the sequence of checktasks. In one embodiment, the execution priority weight is based on the latency T of the checkpointed task _nw The first indication value QP _f And the second indicator value UO _f The three are calculated:

T _np ＝αT _nw +βQP _f +γUO _f (1)

wherein T is _np Representing the execution priority weight, alpha being the weight of the waiting time of the check task, beta being the weight of the first indication value, and gamma being the weight of the second indication value.

In general, α in equation (1) is set to be much larger than β and γ so that the waiting time of the check task plays a major role, so that the execution of the check task basically satisfies the first-in first-out principle. However, in some cases, when the frequency of quality problems of the check object indicated by the first indication value in a period of time (for example, in the last week, one month, or two months) is relatively high in a first specified period of time (for example, a working day), for example, the total first time count is higher than a preset threshold value, or the total first time count is continuously higher, or the total first time count is higher than a preset threshold value. Similarly, when the second indicator value indicates that the frequency of the check object being invoked by the check object consumer is relatively high during a second specified time period (e.g., a working day), such as the third time being greater than a predetermined threshold, or going continuously high, or going high at a speed greater than a predetermined threshold, the weight γ of the second indicator value is increased. The above increases in weights β and γ are sufficient to affect a change in the order of the corresponding inspection tasks in the inspection task sequence.

Alternatively, instead of increasing the weights β and γ, the weights α may be selected to be decreased, which also affects the change of the order of the corresponding inspection tasks in the inspection task sequence.

In step 2204, the inspection task is added to the inspection task sequence according to the execution priority weight calculated in step 2203. The checking task sequence sorts the checking tasks according to the execution priority weights of the checking tasks so as to execute the checking tasks according to the sequence of the checking task sequence.

Each time a new inspection task is obtained, the execution priority weight of the task is calculated, and then according to the execution priority weight of each inspection task in the inspection task sequence, the position to which the new inspection task should be inserted in the sequence is determined. It will be appreciated that if the execution priority of the new checktasks is greatest, it is placed at the first position of the sequence, and if the execution priority of the new checktasks is least, it is placed at the end of the sequence.

According to the embodiment of the disclosure, by setting and calculating the execution priority weights based on some data attributes, the data attributes can influence the execution priority of the checking tasks, so that the execution sequence of the checking tasks is more scientific and reasonable, and the execution sequence can be customized based on specific factors. Further, the scheduling execution of the checking task is more intelligent and flexible by dynamically adjusting the weights of the data attributes.

In one embodiment, the inspection task may include a plurality of inspection subtasks, for example, a plurality of inspection methods. Accordingly, the method 300 further comprises: and distributing a corresponding number of threads to the checking task according to the number of the plurality of checking subtasks of the checking task so as to execute the plurality of checking subtasks in parallel. In this way, the inspection efficiency can be improved on a task-by-task basis. The method maintains the execution sequence of each checking task unchanged, and improves the checking efficiency by introducing a parallel execution mode.

In one embodiment, the method 300 further includes assigning an accompanying thread for recording a log of execution results of the checktasks, wherein the accompanying thread is started in synchronization with the execution threads of the checktasks, and when the checktasks include a plurality of checksubtasks and a corresponding number of threads are assigned to execute the checksubtasks, the accompanying thread is started in synchronization with the corresponding number of threads. By introducing the accompanying tasks to be executed in parallel, the efficiency is further improved.

Returning to fig. 2, at step 230, the inspection results are collected. The result data obtained by executing the checking method can be generated into corresponding checking result files, such as DAT files, XML files and the like, and the checking result files are stored in preset positions, such as folders with the checking date of the current checking task plus the application where the checking object is located as the name.

In one embodiment, a quantiz job scheduling framework is employed to perform polling acquisition of the checkresults. Quartz is an open source job scheduling framework, which is written entirely in Java and designed for use in J2SE and J2EE applications. It provides great flexibility without sacrificing simplicity. It can be used to create simple or complex schedules for executing a job. The Quartz implements functions in three layers: (1) task (Job): the specific work to be performed, in this disclosure, is collection of the inspection result; (2) Trigger (Trigger): triggering task execution by setting specific time conditions, including simple triggers and complex triggers; (3) Scheduler: the actual executor of the task is responsible for gluing the task and the trigger, based on which the task is executed.

The time condition set by the trigger may be single, for example, set to every fixed time period, but the collection operation of each inspection task may have a time difference due to different number of files of each inspection result or interference of other possible factors. This may result in that the last inspection result collection operation has not yet ended when the next inspection result collection operation is started. For example, the trigger setting polls once every 1 second, however, when the check result B is polled, the acquisition of the check result a has not yet ended.

In one embodiment, the acquisition of the checking result adopts a single-job execution mode, that is, when the job is started, the last acquisition job is not finished yet, the current acquisition job is finished automatically, and the next acquisition job is started until the last acquisition job is finished. In this way, resource contention issues between jobs of different batches may be avoided. For example, when the checking result B is polled, the checking result a is not acquired yet, and the checking result B is not acquired, but the checking result a is waited for to be acquired.

Fig. 5 depicts a flow chart of the above example. In step 2301, a triggering condition in time is set to trigger polling for collecting the execution result of the check task. In step 2302, it is determined whether the trigger condition is satisfied. When the condition is not satisfied, waiting is continued, and when the condition is satisfied, it is determined in step 2303 whether there is currently acquisition of the execution result being executed. When it is determined that there is no acquisition of the execution result being executed, acquisition of the execution result of the check task is executed in step 2304. When it is determined that there is an acquisition of the execution result being executed, a new acquisition is not performed, but waiting.

In one embodiment, the collection operation collects the inspection result files into a database, and performs data pre-summarization operation on the inspection results in the process, and simultaneously stores the detail data and summarized data of the inspection results into different files, such as a detail table and a summary table of the inspection results. If errors or anomalies occur in the process, rollback operation is performed on the database to avoid incomplete data, and the files with errors or anomalies are moved to a special directory.

In one embodiment, the verification result file performs a delete operation after collection into the database. For example, after all the inspection result files are collected, judging whether the inspection result files which are not collected exist at the preset position (for example, in a folder with the name of the application where the inspection object is located and the inspection date of the current inspection task) or not, and if not, deleting the current folder; if there are more files that have not been acquired, for example because the ready file did not arrive, then all files in the current folder are retained, waiting for the arrival of the next acquisition time point.

The inspection results may then be analyzed at step 240 using a database in which the inspection results are collected.

In one embodiment, the database may provide data retrieval, such as retrieval based on a single condition or a combination of conditions, enabling screening of the inspection results. Alternatively or optionally, the summary data or the detail data may be configured or ordered in a customized manner, for example, according to a service unit where the checking task is located, a checking category, and other customized data presentation manners. Alternatively or optionally, the trend of the change in the quality problem of the data in the inspection result, the result of the management of the quality problem, or the like may be displayed based on predetermined conditions, such as the inspection period, the inspection method, or the like. Alternatively or additionally, a data quality analysis report may be provided based on the inspection results, the report supporting text or graphical presentation, supporting export, etc. Alternatively or optionally, the checking method for the key monitoring can be customized according to different requirements of the data consumption end user based on the checking result, so that centralized analysis and management are facilitated.

Referring to fig. 6, in an embodiment of the present invention, a checking method for checking an object may be performed at the data quality management platform 130. Although only the data quality management platform 130 is illustrated in fig. 6 as one device 600, in fact, there may be multiple devices, each device 600 including a processor 604 including a hardware element 610. The processor 604 includes, for example, one or more Digital Signal Processors (DSPs), general purpose microprocessors, application Specific Integrated Circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for inspection of inspected objects, or incorporated in combined hardware and/or software modules. Also, the techniques may be fully implemented in one or more circuits or logic elements. The methods in this disclosure may be implemented in various components, modules or units, but need not be implemented by different hardware units. Rather, as noted above, various components, modules, or units may be combined or provided by a collection of interoperable hardware units (including one or more processors as noted above), in combination with suitable software and/or firmware.

In one or more examples, the content described above in connection with fig. 1-5 may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on the computer-readable medium 606 as one or more instructions or code or transmitted over the computer-readable medium 606 and executed by a hardware-based processor. The computer-readable medium 606 may comprise a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium that facilitates a computer program (including one or more of the instructions or code described above), such as any medium that is transferred from one place to another, according to a communication protocol. In this manner, computer-readable medium 606 may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be read by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this disclosure. The computer program product may include a computer-readable medium 606 and one or more instructions or codes stored thereon.

By way of example, and not limitation, such computer-readable storage media can comprise memory such as RAM, ROM, EEPROM, CD _ROM or other optical disk, magnetic disk memory or other magnetic memory, flash memory, or any other memory 612 that can be used to store desired program code in the form of instructions or data structures and that can be read by a computer. Also, any connection is properly termed a computer-readable medium 606. For example, if instructions are transmitted from a website, settlement system, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be appreciated, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory tangible storage media. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media 606.

Device 600 may also include I/O interfaces for transferring data, as well as other functions 614. The apparatus 600 may also be included in different terminals, such as a computer 616, a mobile device 618, and other terminals 620, among others. Each of these configurations includes devices that may have generally different configurations and capabilities, and thus devices for inspection objects may be configured according to one or more of the different device classes. The techniques of the present invention may also be implemented in whole or in part on a "cloud" 622 using a distributed system, such as through a platform 624 as described below.

Cloud 622 includes and/or represents a platform 624 for resources 626. Platform 624 abstracts underlying functionality of hardware (e.g., a settlement system) and software resources of cloud 622. The resources 626 may include applications and/or data that may be used when executing computer processing on a data system remote from the computing device 600. The resources 626 may also include services provided over the internet and/or over subscriber networks such as cellular or Wi-Fi networks.

Platform 624 may abstract resources and functionality to connect computing devices with other computing devices. Platform 624 may also be used to abstract a hierarchy of resources to provide a corresponding level of hierarchy of requirements encountered for resources 626 implemented via platform 624. Thus, in an interconnected device embodiment, the implementation of the functionality described herein may be distributed throughout the device 600. For example, the functionality may be implemented in part on a computing device and by platform 624 that abstracts the functionality of cloud 622.

According to the embodiment of the disclosure, by setting and calculating the execution priority weights based on the factors, the factors can influence the execution priority of the checking tasks, so that the execution sequence of the checking tasks is more scientific and reasonable, and the execution sequence can be customized based on specific factors. Further, the weights of the factors are dynamically adjusted, so that the dispatching execution of the checking task is more intelligent and flexible.

Method steps the description of method steps does not represent their execution in any order, and the method steps described may be performed in any order possible, without specific illustration or without any precondition being required (i.e. the execution of one step is premised on the execution of another step).

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of checking for a checking object, comprising the steps of:

acquiring a checking task aiming at a checking object;

acquiring a plurality of attribute data of the checking task;

calculating the execution priority weight of the checking task based on at least part of attribute data in the plurality of attribute data; and

adding the checking tasks into a checking task sequence according to the execution priority weights, and sequencing the checking tasks according to the execution priority weights of the checking tasks in the checking task sequence so as to execute the checking tasks according to the sequence of the checking task sequence.

2. The method of claim 1, wherein,

the at least part of the attribute data includes a first indication value indicating how often the quality problem occurs for a first specified time period of the inspected object within a first time period, and a second indication value indicating how often the inspected object is invoked by the inspected object consumer for a second specified time period of the inspected object within a second time period, the first time period being not less than the first specified time period, the second time period being not less than the second specified time period, and both the first time period and the second time period including a current time.

3. The method of claim 2, wherein the at least some attribute data further comprises a latency of the inspection task.

4. The method of claim 3, wherein the execution priority weight is calculated as:

T _np ＝αT _nw +βQP _f +γUO _f ，

wherein T is _np Is the execution priority weight of the checking task, T _nw Is the wait time of the checking task, QP _f Is the first indicated value, UO _f Is the second indicator value, α is the weight of the latency of the inspection task, β is the weight of the first indicator value, and γ is the weight of the second indicator value.

5. The method of claim 4, wherein the weight of the first indication value is increased to affect ordering of the inspection tasks in the inspection task sequence when the number of times the quality problem occurs in the first specified time period of the inspection object is higher than a first threshold, or is continuously increased, or is increased at a speed higher than a second threshold for a period of time, wherein the period of time includes a plurality of the first specified time periods.

6. The method of claim 4, wherein when the number of times the inspected object is invoked by the inspected object consumer over the second specified time period is above a third threshold, or is continuously increasing, or is increasing in speed above a fourth threshold, the weight of the second indicator value is increased to affect the ordering of the inspected tasks in the inspected task sequence, wherein the period of time comprises a plurality of the second specified time periods.

7. The method of claim 2, wherein the first indicator value and the second indicator value are non-dimensionalized values determined relative to corresponding statistics of a period of time between when the inspected object is online and a current time.

8. The method of claim 2, wherein the first indication value is determined as follows:

determining a first number of times, the first number of times being the number of times the quality problem occurs to the inspected object in the first specified time period in the first time period;

determining a second number of times, the second number of times being an average number of times that the quality problem occurs in the first specified time period in a time period between when the inspected object is online and a current time; and

comparing the first number of times with the second number of times, and assigning a value to the first indication value based on a result of the comparison.

9. The method of claim 8, wherein assigning the first indicator value further comprises:

determining to assign the first indicator value in a first numerical interval in response to the first number being greater than the second number, determining to assign the first indicator value in a second numerical interval in response to the first number being equal to the second number, and determining to assign the first indicator value in a third numerical interval in response to the first number being less than the second number,

10. The method of claim 2, wherein the second indicator value is determined as follows:

determining a third number of times, wherein the third number of times is the number of times that the check object is called by the check object consumption end in the second designated time period in the second time period;

determining a fourth number of times, the fourth number of times being an average number of times the second specified time period of the check object is invoked by the check object consumer in a time period between when the check object comes online and when the check object is current; and

comparing the third times with the fourth times, and assigning the second indicator value based on the result of the comparison.

11. The method of claim 10, wherein said assigning the second indicator value further comprises:

determining to assign the second indicator value in a fourth numerical interval in response to the third number being greater than the fourth number, determining to assign the second indicator value in a fifth numerical interval in response to the third number being equal to the fourth number, and determining to assign the second indicator value in a sixth numerical interval in response to the third number being less than the fourth number, wherein:

12. The method of claim 9 or 11, wherein the assigning further comprises:

and in the determined numerical value interval, performing assignment further based on the importance of the checked object.

13. The method of claim 1, wherein the plurality of attribute data includes status information of the checkpointed task indicating whether the checkpointed task is conditioned for execution, the method further comprising:

the following steps are triggered only when the check task has an execution condition:

calculating the execution priority weight of the checking task; and

and adding the checking task into a checking task sequence according to the execution priority weight.

14. The method of claim 2, wherein the first specified time period and the second specified time period comprise: one working day.

15. The method of claim 1, wherein the checktasks comprise a plurality of checksubtasks, the method further comprising: and distributing the threads to the checking tasks according to the number of the checking subtasks so as to execute the checking subtasks in parallel.

16. The method as recited in claim 15, further comprising: and allocating an accompanying thread, wherein the accompanying thread is used for recording an execution result log of the check task, and the accompanying thread is started synchronously with the number of threads.

17. The method of claim 3, wherein the plurality of attribute data comprises:

the ready time of the check task, which indicates the start time of the check task with execution conditions,

wherein the latency of the checkup task is calculated as: the ready time of the checking task is subtracted from the current time.

18. The method as recited in claim 1, further comprising:

setting a triggering condition in time to trigger polling for collecting an execution result of the checking task;

determining whether the acquisition of the execution result is currently being executed or not in response to the trigger condition being satisfied;

and in response to determining that there is no acquisition of an execution result currently being executed, performing acquisition of an execution result of the check task.

19. A checking apparatus for checking an object, comprising:

one or more processors; and

One or more memories configured to have stored thereon computer-executable instructions that, when executed in the one or more processors, cause the method of any of claims 1-18 to be implemented.

20. A computer-readable storage medium having instructions stored therein that, when executed on one or more computers, cause the one or more computers to implement the method of any one of claims 1-18.