WO2024039017A1

WO2024039017A1 - Method and apparatus for managing quality of data

Info

Publication number: WO2024039017A1
Application number: PCT/KR2023/007653
Authority: WO
Inventors: Rajasekhara Reddy Duvvuru Muni; Karthikeyan Kumaraguru; Payyavula Rao SRINIVASA; Sejoon Oh
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2022-08-16
Filing date: 2023-06-02
Publication date: 2024-02-22

Abstract

A system and method for managing quality of data for providing AI based services to one or more user devices. The method comprises defining a first set of data quality rules based on offline data stored in a database, generating a ML model based on the first set of data quality rules, wherein the ML model is deployed to provide the AI based services in association with the corresponding applications. The method comprises monitoring over a period of time, using the first set of data quality rules, incoming real-time, online data received by the deployed ML model across the one or more user devices. The method further comprises continually determining, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data and dynamically generating a second set of data quality rules.

Description

METHOD AND APPARATUS FOR MANAGING QUALITY OF DATA

The present disclosure generally relates to machine learning (ML) models, and more particularly relates to a system and a method for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices.

As is widely known, it is a tricky and time-consuming job to maintain good data quality for following ML tasks in fast-iterating service domains. Service changes, changes in content, and changes in features associated with the ML task could often cause unexpected data quality degradation and make poor ML task output and misleading data analysis. So, it is critical to build a system to monitor data/model drift in offline and online way for the sake of sustainable ML development, deployment, and innovative experiment on services.

However, the present ML model performance monitoring, and lack of data quality observability in online deployments, impacts user experience and business decisions. Application services dependent on data analytics for identification of potential churners to manage customer relations are sensitive to data quality. When the ML models built on poor quality data are compared against ML models built on good quality data, even a minor performance gain impacts the business in a big way. For example, out of 1 bn active users, 0.02% accuracy gain may detect approximately 400K users as high confidence churners. This is a valuable resource for Customer Relationship Management (CRM) teams generating significant engagement and revenue gains.

Inferences that are made based on machine learning models built with bad quality data could cost millions of dollars, cost trust issues, and reduce user engagement in various applications deploying ML models.

Accordingly, there is a need to address the aforementioned challenges and specifically, there is a need for methods and systems to improve real-time data readability and observability for ML models. Further, there is a need for automatic data quality management, observing quality issues in online data, detecting the data divergences, and identifying issues impacting the model performance proactively.

This technical solution is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the disclosure. This technical solution is neither intended to identify key or essential inventive concepts of the invention and nor is it intended for determining the scope of the disclosure.

In an embodiment, the present disclosure refers to a method for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices. The method comprises defining a first set of data quality rules based on offline data stored in a database, the offline data being associated with corresponding applications associated with the one or more user devices. The method further comprises generating a machine learning (ML) model based on the first set of data quality rules, wherein the ML model is deployed to provide the AI based services in association with the corresponding applications. The method also comprises monitoring over a period of time, using the first set of data quality rules, incoming real-time, online data received by the deployed ML model across the one or more user devices. The method further comprises continually determining, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data. The method also comprises dynamically generating a second set of data quality rules, by updating the first set of data quality rules based on the determined one or more characteristics, to facilitate providing the AI based services in association with the corresponding applications.

In an embodiment, the present disclosure refers to a system for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices. The system comprises a memory and at least one at least one processor communicatively coupled to the memory. The at least one at least one processor being configured to define a first set of data quality rules based on offline data stored in a database, the offline data being associated with corresponding applications associated with the one or more user devices. The at least one at least one processor being further configured to generate a machine learning (ML) model based on the first set of data quality rules, wherein the ML model is deployed to provide the AI based services in association with the corresponding applications. The at least one at least one processor being further configured to monitor over a period of time, using the first set of data quality rules, incoming real-time, online data received by the deployed ML model across the one or more user devices. The at least one processor also being configured to continually determine, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data and dynamically generate a second set of data quality rules, by updating the first set of data quality rules based on the determined one or more characteristics, to facilitate providing the AI based services in association with the corresponding applications.

To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope. The present disclosure will be described and explained with additional specificity and detail with the accompanying drawings.

These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

Figure 1 illustrates image classification using ML models, in accordance with existing art;

Figure 2 illustrates payment fraud detection using ML models, in accordance with existing art;

Figure 3 illustrates a flow chart depicting a method for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices, according to an embodiment of the present disclosure;

Figure 4 illustrates a block diagram of a system for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices, according to an embodiment of the present disclosure;

Figure 5 illustrates a block diagram of an intelligent rule recommender engine and an associated workflow, according to an embodiment of the present disclosure;

Figure 6 illustrates a data observability engine, according to an embodiment of the present disclosure;

Figure 7 illustrates a divergence detection engine and an associated workflow, according to an embodiment of the present disclosure;

Figures 8(a)-8(b) illustrates a comparison between existing techniques and the proposed method for data observability in real-time log streaming, according to various embodiments of the present disclosure;

Figure 9 illustrates a use case associated with accurate image classification using enhanced data observability, according to an embodiment of the present disclosure;

Figure 10 illustrates a use case associated with real time fraud detection in payment applications using enhanced data observability, according to an embodiment of the present disclosure;

Figure 11 illustrates a use case associated with recommending right content for interest deviation using enhanced data observability, according to an embodiment of the present disclosure; and

Figure 12 illustrates a use case associated with application recommendation based on user app usage pattern, according to an embodiment of the present disclosure.

Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the user device, one or more components of the user device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the disclosure relates.

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the disclosure and are not intended to be restrictive thereof.

Reference throughout this specification to "an aspect", "another aspect" or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrase "in an embodiment", "in another embodiment" and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more user devices or sub-systems or elements or structures or components proceeded by "comprises... a" does not, without more constraints, preclude the existence of other user devices or other sub-systems or other elements or other structures or other components or additional user devices or additional sub-systems or additional elements or additional structures or additional components.

In accordance with various embodiments of the present disclosure, a smart dynamic data quality management platform is provided that integrates rule extraction, validation on data streams of big-data applications and monitors dynamic drift in data that will impact the Machine Learning system's performance that impacts dependent services.

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

Figure 1 illustrates image classification using ML models, in accordance with existing art. As depicted, one or more rules may be manually generated for data cleaning of offline data provided for building of an ML model. The data may be cleaned and an ML model may be built using the cleaned data and the rules. Further, the built ML models may be used on the online data to classify the images.

As depicted, a cat may be predicted as tiger using the ML model, as the data quality management implemented in offline is not used to monitor the online data and the ML model may have failed. By the time the model monitoring systems detect performance issues, the damage to business may already have been done due to incorrect inferences.

Figure 2 illustrates payment fraud detection using ML models, in accordance with existing art. An ML model may be built using manual rules generated for offline data. However, the user data from financial payment applications may change over time and diverge from offline data used for ML model training, leading to predicting fraud transactions as non-fraud, or vice versa, thereby leading to incorrect predictions. For example, A Fraud detection ML Model is built using manual data cleaning. As per the offline data, Scientist Age should be more than 36 and income should be in the range of 40,000-53500 and credit score can be in the range of 354 - 695 to be detected as a legitimate transaction. However, in the online data, we got a Scientist, whose age is 19, income is 140000 and credit score is 354 which is violating the deployed model learnings. So ML Model will detect him as fraud, which is not accurate. Actually, what happened is, divergence in data. Salaries of scientists has increased recently. Transactions Per month are very around 60 because, the person demographics changed due to his marriage and credit score decreased due to multiple score checks. Such data divergence should be actively monitored and ML services should be updated.

Figure 3 illustrates a flow diagram depicting method for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices, in accordance with an embodiment of the present disclosure. Figure 4 illustrates a block diagram of system for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices. For the sake of brevity, the description of the Figs. 3 and 4 are explained in conjunction with each other.

The system 400 may include, but is not limited to, at least one processor 402, memory 404, modules 406, and data unit 408. The modules 406 and the memory 404 may be coupled to the at least one processor 402.

The at least one processor 402 can be a single processing unit or several modules, all of which could include multiple computing modules. The at least one processor 402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal at least one processors, central processing modules, state machines, logic circuitries, and/or any user devices that manipulate signals based on operational instructions. Among other capabilities, the at least one processor 402 is configured to fetch and execute computer-readable instructions and data stored in the memory 404.

Functions related to artificial intelligence according to the present disclosure are operated through at least one processor 402 and a memory 404. The processor 402 may include one or a plurality of processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, an AP, or a digital signal processor (DSP), a graphics-only processor such as a GPU or a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. For example, when one or more processors are processors dedicated to artificial intelligence, the processors dedicated to artificial intelligence may be designed as a hardware structure specialized for processing a specific artificial intelligence model.

One or more processors control input data to be processed according to predefined operating rules or artificial intelligence models stored in a memory. Alternatively, when one or more processors are processors dedicated to artificial intelligence, the processors dedicated to artificial intelligence may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

A predefined action rule or an artificial intelligence model is characterized in that it is created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created. Such learning may be performed in the device itself in which artificial intelligence according to the present disclosure is performed, or through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

An artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weight values. A plurality of weights possessed by a plurality of neural network layers may be optimized by a learning result of an artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from an artificial intelligence model is reduced or minimized during a learning process. The artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), A deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, but is not limited to the above examples.

The memory 404 may include any non-transitory computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes.

The modules 406 amongst other things, include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement data types. The modules 406 may also be implemented as, signal at least one processor(s), state machine(s), logic circuitries, and/or any other user device or component that manipulate signals based on operational instructions.

Further, the modules 406 can be implemented in hardware, instructions executed by a processing unit, or by a combination thereof. The processing unit can comprise a computer, a processor, such as the at least one processor 402, a state machine, a logic array, or any other suitable user devices capable of processing instructions. The processing unit can be a general-purpose processor which executes instructions to cause the general-purpose processor to perform the required tasks or, the processing unit can be dedicated to performing the required functions. In an embodiment of the present disclosure, the modules 406 may be machine-readable instructions (software) which, when executed by a at least one processor/processing unit, perform any of the described functionalities.

In an embodiment, the modules 406 may include an intelligent rule recommender engine 410, a data observability engine 412, and a divergence detection engine 414.

The various modules 410-414 may be in communication with each other. In an embodiment, the various modules 410-414 may be a part of the at least one processor 402. In an embodiment, the at least one processor 402 may be configured to perform the functions of modules 410-414. The data unit 408 serves, amongst other things, as a repository for storing data processed, received, and generated by one or more of the modules 406.

It should be noted that the system 400 may be a part of a user device. In an embodiment, the system 400 may be connected to the user device. It should be noted that the term "user device" refers to any electronic user devices used by a user such as a mobile user device, a desktop, a laptop, personal digital assistant (PDA) or similar user devices.

Referring to Figure 3, at step 301, the method comprises defining a first set of data quality rules based on offline data stored in a database. In an embodiment, the offline data is associated with corresponding applications associated with the one or more user devices. In particular, assuming that one or more applications are associated with the one or more user devices. The one or more applications may constantly send data and store data in the database warehouse for processing and service development. This data is considered as offline data.

In an embodiment, the intelligent rule recommender engine 410 may define the first set of data quality rules based on the offline data stored in the database. In an alternate embodiment, the intelligent rule recommender engine 410 may directly receive the offline data from the one or more applications.

In other words, the offline data may refer to data received from applications associated with the one or more user devices, the data being related to one or more interactions of respective users of the one or more user devices with the corresponding application. In an embodiment, the offline data may further relate to performance of the AI based services associated with the corresponding application. The offline data may thus be considered as non-real-time data that is received from the application associated with the one or more user devices and stored for further processing.

Figure 5 illustrates a block diagram of an intelligent rule recommender engine and an associated workflow, according to an embodiment of the present disclosure. As shown in Figure 5, the intelligent rule recommender engine 410 is configured to generate the first set of data quality rules for real-time ML model deployment by analyzing and classifying the retrieved offline data.

As shown in FIG. 5, the intelligent rule recommender engine 410 may include a quality analyzer module 505 and a rule configurator module 507. As can be seen from Figure 5, the intelligent rule recommender engine 410 is connected to the database or data warehouse 501 which stores, over a period of time, the offline data, where the offline data is received from the corresponding one or more applications associated with the one or more user devices. Hence, the database 501 is an offline data store capable of receiving data and serving data.

The offline data is retrieved by a profiler 503 which is connected to the intelligent rule recommender engine 410. The profiler 503 periodically profiles the offline data and generates profile information associated with the stored offline data. The generated profile information is shared with the quality analyzer module 505.

The quality analyzer module 505 then determines offline data parameters based on the profile information, wherein the offline data parameters comprise data types within the offline data and data characteristics associated with each of the data types. In particular, a type identifier 505a scans the profile information and determines data type associated with the offline data. The type identifier 505a then shares the data type with a rule identifier 507a of the rule configurator 507 and a threshold miner 505b, as depicted in Figure 5.

In a non-limiting example, the data may be an image and the offline data parameters may comprise object length, height, colour, pattern, and the like. In an non-limiting example, the data may be employee data and the offline data parameters may comprise user personal information, user preferences, user position, user income, and the like. As described above, each of offline data parameter may comprise data types and data characteristics. For instance, the user personal information may be a parameter, and the user personal information may comprise user age as a data type and specific values, such as, 20 years, as the data characteristic. In an example, image object may be a parameter, and the image object may comprise stripes as a data type and brown as a data characteristic.

Further, the threshold miner 505b identifies data type specific data thresholds dynamically and validates the same, i.e., the threshold miner 505b determines the data characteristics associated with the offline data. In an embodiment, a report generator 505c, along with type specific thresholds may generate a quality report with quality score of the offline data. In an embodiment, the quality report may comprise information related to the data types and data characteristics of the various offline data parameters.

For instance, in case of offline data comprising image objects, the quality reports may include information related to the data types and associated characteristics, such as, white stripes, brown stripes, etc. Similarly, in case of offline data comprising user details, the quality report may include information related to the data type and associated characteristics, such as, maximum age, median age, fill rate, and the like. In an embodiment, the generated data quality reports may be sent to users for review as alerts.

The rule configurator 507 may define the first set of data quality rules based on the offline data parameters. In an embodiment, the rule configurator 507 may automatically define the first set of data quality rules based on the offline data parameters.

In particular, the rule identifier 507a with inputs received from the type identifier 505a automatically identifies appropriate rules and forward the same to a parameter estimator 507b. The parameter estimator 507b with inputs from the rule identifier 507a and threshold miner 505b estimates accurate parameters for rule. Then, a rule definer 507c receives the rules with relevant parameters from the parameter estimator 507b and defines the first set of data quality rules. In an embodiment, the defined first set of data quality rules and optional inputs from users may be stored in a database such as a DQ Rules Repo 509. In an embodiment, the DQ Rules Repo 509 may optionally accept custom rules from users for integration with data observability engine 412.

Referring back to Figure 3, at step 303, the method 300 comprises generating/obtaining a ML model based on the first set of data quality rules. The ML model is deployed to provide the AI based services in association with the corresponding applications. In an embodiment, the ML model may be generated/obtained by the processor 402 based on set of data quality rules determined by the intelligent rule recommender engine 410. In an embodiments, the offline data may be filtered based on the first set of data quality rules determined by the intelligent rule recommender engine 410, and further, the ML model may be generated/obtained based on the filtered offline data.

Then, at step 305, the method 300 comprises monitoring over a period of time, using the first set of data quality rules, incoming real-time, online data received by the deployed ML model across the one or more user devices.

In an embodiment, the online data is related to one or more of interactions of respective users of the one or more user devices with the corresponding application and performance of the AI based services associated with the corresponding applications. In an embodiment, the one or more interactions of the users may comprise interactions done by the users via an application provided on the associated user device. For instance, an image classification application may be provided on the user device and the user may interact with the application to purchase an item and/or download other applications on the user device.

In an embodiment, the online data refers to real-time data being received from the applications associated with the one or more user devices. In an embodiment, the online data may differ from the offline data, in that, the online data may have some additional data or some deleted data as compared to the offline data.

In an embodiment, the data observability engine 412 may monitor incoming real-time, online data received by the deployed ML model across the one or more user devices. Figure 6 illustrates a data observability engine 412, according to an embodiment of the present disclosure.

As shown in figure 6, a stream processor associated with the applications may feed the real-time online data, such as in form of a data stream 601, that serves the deployed ML model to the data observability engine 412. The data observability engine 412 may fetch the first set of data quality rules from the intelligent rule recommender engine 410, and may filter the online data to identify clean and unclean data based on the first set of data quality rules. In an embodiment, the filtered data may be in a database or segregated data store 603 as logs for further processing. In an embodiment, the period of time may be configurable and may be configured by the data observability engine 412. The data observability engine 412 may feed the deployed ML model with the clean data and accordingly, the ML model may provide the AI based services based on the clean data.

Then, at step 307, the method 300 comprises determining, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data. In an embodiment, the divergence detection engine 414 may continually determine, based on monitoring of the online data, the one or more characteristics of the online data that are divergent from the offline data.

In an embodiment, the divergence detection engine 414 may determine the one or more characteristics of the online data that are divergent from the offline data, which is explained in reference to Figure 7.

Figure 7 illustrates a divergence detection engine 414 and an associated workflow, according to an embodiment of the present disclosure. The divergence detection engine 414 may be configured to determine data quality divergence in the incoming real-time data based on the monitoring performed by the data observability engine 412. In particular, the divergence detection engine 414 may access the clean data and the unclean data filtered from the online data. In an embodiment, the divergence detection engine 414 may receive the clean and the unclean data from the data observability engine 412. Then, the divergence detection engine 414 may process the unclean data and the clean data to detect a type of data divergence in the online data.

In order to process the clean and the unclean data, in an embodiment, the divergence detection engine 414 may include a label generator 701 to labelling the clean data and the unclean data to generate labelled data. In an embodiment, before labelling the data, the clean data is partitioned into reference and real-time data. The label generator 701 may label the clean data as target and reference.

Further, the divergence detection engine 414 may include a non-linear transformer 703 to perform non-linear transformations on the labelled data to generate transform data. Further, the divergence detection engine 414 may also include a linear divergence detector 705 to detect presence of data divergence based on the transformed data. Furthermore, the divergence detection engine 414 may include a linear divergence classifier 707 to determine the type of data divergence based on the detected presence of data divergence. In an embodiment, the linear divergence classifier 707 may recommend the ML model management strategy based on the type of data divergence.

Then, the divergence detection engine 414 may determine the one or more characteristics of the online data based on the type of data divergence and the first set of data quality rules. In an embodiment, the divergence detection engine 414 may also include a non-linear divergence quantifier 709 that may quantify the divergence magnitude and estimate the significance of divergence to identify and recommend second set of data quality rules generation to counter the data.

Referring back to Figure 3, at step 309, the method 300 comprises generating a second set of data quality rules, based on the determined one or more characteristics, to facilitate providing the AI based services in association with the corresponding applications. In an embodiment, the method 300 comprises dynamically generating a second set of data quality rules, by updating the first set of data quality rules based on the determined one or more characteristics to facilitate providing the AI based services in association with the corresponding applications.

In an embodiment, the method 300 comprises comparing the one or more characteristics of the online data with corresponding threshold ranges.

If it is determined that the one or more characteristics of the online data are within corresponding threshold ranges, the first set of data quality rules are updated to dynamically generate the second set of data quality rules based on the one or more characteristics of the online data. The second set of data quality rules may then be used by the data observability engine 412 to monitor the online data being provided by the applications. Then, the method 300 comprises generating an updated ML model, using the second set of data quality rules to provide the AI based services in association with the corresponding applications.

In an embodiment, if it is determined that the one or more characteristics of the online data are not within the corresponding threshold ranges, the ML model is updated to generate the updated ML model based on the one or more characteristics of the online data. In an embodiment, the updated ML model may be generated by updating the previously deployed ML model.

In an embodiment, the updated model may be generated by selecting an alternate model from a plurality of pre-developed models, the alternate model being developed so as to process the online data that comprises characteristics divergent from the offline data. In an embodiment, the plurality of pre-developed models may be stored in an ML model repository.

In an embodiment, the updated ML model may be generated by developing a new model based on the second set of data quality rules. In an embodiment, the online data may comprise characteristics divergent from the offline data such that a pre-developed model may not be available to process the online data. For instance, the online data may comprise characteristics that may not be recorded previously. In such cases, updating the ML model may developing a new ML model based on the second set of data quality rules.

In an embodiment, the updated ML model is deployed to provide the AI based services in association with the corresponding applications. Then, the online data being received by the updated ML model is continuing monitored across the one or more user devices. The online data is filtered based on the second set of data quality rules and the updated ML model provides the AI based services based on the filtered online data. Or the online data is filtered based on the first set of data quality rules and the updated ML model provides the AI based services based on the filtered online data.

In an embodiment, in response to the first set of data quality rules being updated and/or in response to the ML model being updated, an alert may be provided to a user associated with the one or more user devices and/or to a remote admin user associated with an admin device.

Figures 8(a)-8(b) illustrates a comparison between existing techniques and the proposed method for data observability in real-time log streaming, according to various embodiments of the present disclosure.

As shown in figure 8(a), in the current techniques, the offline data is manually cleaned, and an ML model is built to deploy on a user device or server. The deployed model may serve online inferences and the model performance is monitored periodically and degradation will be rectified after some delay.

However, as shown in figure 8(b), in accordance with the various embodiments of the present disclosure, the offline data is used to learn quality issues and rules to fix them. An ML model is built to deploy on a user device or a server. Further, during online inference, incoming data is scanned by data monitoring module using the learned rules and quality data will reach deployed ML model for inference. The deviation detector module linked to data monitoring module will periodically monitor the deviations in incoming data and alerts new rule generation requirements.

Thus, in the disclosed techniques, as opposed to the traditional system, the data gets automatically monitored by an intelligent rule recommender engine comprising a profiler to automatically profile the data at periodic intervals to find the data characteristics, a quality analyzer to find the quality parameters of the data that might impact the downstream analysis, a rule configurator that will formulate the rules to fix the quality issues detected therein, and a repository to maintain and serve the rules to the required. These quality issues and rules would be sent to data scientist via alerts module for an optional intervention. The observability engine may access the rules for data quality management from rule recommender to apply and clean the incoming stream data from application event logger. From the cleaned data, the features will be extracted which are necessary for the ML model and served via the feature generator. Further to serving the model, the stream at least one processor also serves classifier to classify clean/unclean data and sends the clean data to the database. Such clean data dynamically updated can be used for down-stream analysis by the ML model.

Further, the clean data persisted in the database is fed to data divergence detection engine to periodically monitor the data divergence and model anomalies in the application data. Further, the data divergence detection engine alerts downstream analyst(s) for corrective action or automatically generate a second set of rules. Also, the data divergence detection engine may automatically trigger the ML Model updating or deploying alternate model process.

Figure 9 illustrates a use case associated with accurate image classification using enhanced data observability, according to an embodiment of the present disclosure.

As shown in figure 9, a best performing image identification ML model is built 'offline' based automated rule recommender system (Intelligent rule recommender engine) (5 rules along with parameters) and deployed in a service online. The data observability engine may use the automatically generated data quality rules, to monitor the incoming online data before sending to deployed ML Model. So, cat image is dropped as it doesn't pass the data quality rules. Data divergence detection engine may analyze the dropped/Poor quality data (Cat image) and assess the rules (if any new rules has to be made) or rule parameters (Values, in this example, Brown stripes, Minimum and maximum values of object length) has to be updated making second set of rules.

Figure 10 illustrates a use case associated with real time fraud detection in payment applications using enhanced data observability, according to an embodiment of the present disclosure.

As shown in figure 10, intelligent rule recommender engine is configured on the application. Data quality rules (Employment type - Engineer, Scientist, IT & Age data cleaning by filling zero values with average by Married), along with parameters are learned. A best performing ML Model is built and deployed in production. The data observability engine will use the data quality rules, to monitor the incoming online data before sending to deployed ML Model. Good quality data is sent to Prediction. Data Observability engine finds the data with missing values (Age) will be filled by rules learned from data and sent to prediction service. In absence of proposed system this record would have been classified as fraud. Bad quality data is sent to data divergence detection engine. Divergent data [(Income of 245000), new employment type (Freelancer) and minimum (178) and maximum (795) values of credit score] are detected and sent to intelligent rule recommender engine to generate second set of rules.

Figure 11 illustrates a use case associated with recommending right content for interest deviation using enhanced data observability, according to an embodiment of the present disclosure.

As shown in figure 11, in the application installed on the user device, there are multiple tabs namely Watch, read, Play and listen. Each tab has multiple genre, and each genre has multiple channels. Users frequently switch between them. Multiple ML Models are built and stored in Repository using clean data as per the data quality rules and are ready to be deployed dynamically. The data observability engine and data divergence detection engine work hand in hand in real time to observer the preferred Channels and divergence in the Preferences and fetch relevant model to drive user engagement.

As shown in figure 12, an application store does recommendations of apps based on user past installs, usage captured by service using user consent. Based on a time slice, there are 4 different types of Users with different Preferences. An ML model is built using this data and deployed in production. At some point GDPR (General　Data Protection Regulation) rules are implemented and most of the customers revoked their consent for tracking app usages leading to skewed or biased data coming to recommend apps. In this situation, the recommender will not be able to recommend relevant apps to all users. Proposed data observability engine and divergence detection engine will detect the divergence in incoming data and can alert/deploy alternate model. At the same time, new set of rules will be generated and used to monitor the incoming data by observer engine.

Accordingly, the present disclosure provides following exemplary advantages:

Rule-based data quality monitoring in real-time online data systems serving data dependent applications;

The system of the present disclosure is independent of meta-data and is capable of learning from actual data automatically both offline and real-time;

Further, the system of the present disclosure is designed to learn data quality issues from offline data and observe online data to be served to deployed ML model and alert the rule generator to learn new rules and/or ML model management (e.g., by deploying a suitable model without retraining the existing ML model).

Embodiments of the disclosure can also be embodied as a storage medium including instructions executable by a computer such as a program module executed by the computer. A computer readable medium can be any available medium which can be accessed by the computer and includes all volatile/non-volatile and removable/non-removable media.

Further, the computer readable medium may include all computer storage and communication media. The computer storage medium includes all volatile/non-volatile and removable/non-removable media embodied by a certain method or technology for storing information such as computer readable instruction code, a data structure, a program module or other data. Communication media may typically include computer readable instructions, data structures, or other data in a modulated data signal, such as program modules. In addition, computer-readable storage media may be provided in the form of non-transitory storage media.

The 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (e.g., electromagnetic waves). This term does not distinguish a case in which data is stored semi-permanently in a storage medium from a case in which data is temporarily stored. For example, the non-transitory recording medium may include a buffer in which data is temporarily stored.

According to an embodiment of the disclosure, a method according to various disclosed embodiments may be provided by being included in a computer program product. The computer program product, which is a commodity, may be traded between sellers and buyers. Computer program products are distributed in the form of device-readable storage media (e.g., compact disc read only memory (CD-ROM)), or may be distributed (e.g., downloaded or uploaded) through an application store or between two user devices (e.g., smartphones) directly and online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored at least temporarily in a device-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server, or may be temporarily generated.

While specific language has been used to describe the present subject matter, any limitation arising on account thereto, are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein. The drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment.

Claims

A method for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices, the method comprising:

defining a first set of data quality rules based on offline data stored in a database, the offline data is associated with corresponding applications for the one or more user devices;

obtaining a machine learning (ML) model based on the first set of data quality rules, wherein the ML model provides the AI based services associated with the corresponding applications;

monitoring over a period of time, in real-time, using the first set of data quality rules, online data received by the ML model across the one or more user devices;

determining, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data; and

generating a second set of data quality rules, based on the determined one or more characteristics, to facilitate providing the AI based services associated with the corresponding applications.
The method of claim 1, further comprising:

obtaining an updated ML model, using the second set of data quality rules, to provide the AI based services associated with the corresponding applications.
The method of any of claims 1 to 2, wherein the obtaining the ML model comprises:

filtering the offline data based on the first set of data quality rules; and

developing the ML model based on the filtered offline data, thereby generating the ML model.
The method of any of claims 1 to 3, wherein the defining the first set of data quality rules comprises:

storing, over the period of time, the offline data in the database, the offline data is received from the corresponding applications for the one or more user devices;

retrieving the offline data from the database;

generating profile information associated with the stored offline data;

determining offline data parameters based on the profile information, wherein the offline data parameters comprise data types within the offline data and data characteristics associated with each of the data types; and

defining the first set of data quality rules based on the offline data parameters.
The method of any of claims 1 to 4, further comprising:

accessing the online data received by the ML model across the one or more user devices;

filtering, based on the first set of data quality rules, the online data to identify clean data and unclean data; and

providing, using the ML model, the AI based services based on the clean data.
The method of claim 5, wherein determining the one or more characteristics of the online data comprises:

accessing the clean data and the unclean data filtered from the online data;

processing the unclean data and the clean data to detect a type of data divergence in the online data; and

determining the one or more characteristics of the online data based on the type of data divergence and the first set of data quality rules.
The method of claim 6, wherein processing the clean data and the unclean data to detect the type of data divergence comprises:

labelling the clean data and the unclean data to generate labelled data;

subjecting the labelled data to non-linear transformations to generate transformed data;

detecting presence of data divergence based on the transformed data; and

determining the type of data divergence based on the detected presence of data divergence.
The method of any of claims 1 to 7, the generating the second set of data quality rules comprising:

comparing the one or more characteristics of the online data with corresponding threshold ranges;

upon determining that the one or more characteristics of the online data are within corresponding threshold ranges, updating the first set of data quality rules to generate the second set of data quality rules based on the one or more characteristics of the online data; and

upon determining that the one or more characteristics of the online data are not within the corresponding threshold ranges, updating the ML model to generate the updated ML model based on the one or more characteristics of the online data, wherein the updated ML model provides the AI based services in association with the corresponding applications.
The method of claim 8, further comprising:

deploying the updated ML model to provide the AI based services in association with the corresponding applications;

continuing monitoring the online data received by the updated ML model across the one or more user devices;

filtering the online data, based on the first set of data quality rules or the second set of data quality rules; and

providing, using the updated ML model, the AI based services based on the filtered online data.
A apparatus for managing quality of data for providing artificial intelligence (AI) based services to one or more user devices, the system comprising:

a memory storing instructions; and

at least one processor configured to execute the instructions to:

define a first set of data quality rules based on offline data stored in a database, the offline data is associated with corresponding applications for the one or more user devices;

obtain a machine learning (ML) model based on the first set of data quality rules, wherein the ML model provides the AI based services in association with the corresponding applications;

monitor over a period of time, in real-time, using the first set of data quality rules, online data received by the deployed ML model across the one or more user devices;

determine, based on the monitoring of the online data, one or more characteristics of the online data that are divergent from the offline data; and

generate a second set of data quality rules, based on the determined one or more characteristics, to facilitate providing the AI based services associated with the corresponding applications.
The apparatus of claim 10, wherein the at least one processor is further configured to:

obtain an updated ML model, using the second set of data quality rules to provide the AI based services associated with the corresponding applications.
The apparatus of any of claims 10 to 11, wherein the at least one processor is configured to:

access the online data being received by the deployed ML model across the one or more user devices;

filter, based on the first set of data quality rules, the online data to identify clean data and unclean data; and

provide, by the deployed ML model, the AI based services based on the clean data.
The apparatus of any of claims 10 to 12, wherein to determine the one or more characteristics of the online data, the at least one processor is configured to:

access the clean data and the unclean data filtered from the online data;

process the unclean data and the clean data to detect a type of data divergence in the online data; and

determine the one or more characteristics of the online data based on the type of data divergence and the first set of data quality rules.
The apparatus of any of claims 11 to 13, wherein the at least one processor is configured to:

deploy the updated ML model to provide the AI based services in association with the corresponding applications;

continue monitoring the online data being received by the updated ML model across the one or more user devices;

filter the online data, based on the first set of data quality rules or the second set of data quality rules; and

provide, by the updated ML model, the AI based services based on the filtered online data.
A computer readable medium containing instructions that, when executed, cause at least one processor of an electronic device to perform operations corresponding to the method of any one of claims 1-9.