US20230289698A1 - System and Methods for Monitoring Related Metrics - Google Patents

System and Methods for Monitoring Related Metrics Download PDF

Info

Publication number
US20230289698A1
US20230289698A1 US18/118,274 US202318118274A US2023289698A1 US 20230289698 A1 US20230289698 A1 US 20230289698A1 US 202318118274 A US202318118274 A US 202318118274A US 2023289698 A1 US2023289698 A1 US 2023289698A1
Authority
US
United States
Prior art keywords
data
metrics
metric
topic
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US18/118,274
Other languages
English (en)
Inventor
Adam Bly
David Kang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
System Inc
Original Assignee
System Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by System Inc filed Critical System Inc
Priority to US18/118,274 priority Critical patent/US20230289698A1/en
Assigned to SYSTEM INC. reassignment SYSTEM INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BLY, Adam, KANG, DAVID
Publication of US20230289698A1 publication Critical patent/US20230289698A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9024—Graphs; Linked lists

Definitions

  • references to “System” in the context of an architecture or to the System architecture or platform herein refer to the architecture, platform, and processes for performing statistical search and other forms of data organization described in U.S. patent application Ser. No. 16/421,249, entitled “Systems and Methods for Organizing and Finding Data”, filed May 23, 2019 (now issued U.S. Pat. No. 11,354,587, dated Jun. 7, 2022), which claims priority from U.S. Provisional Patent Application Ser. No. 62/799,981, entitled “Systems and Methods for Organizing and Finding Data”, filed Feb. 1, 2019, the entire contents of which are incorporated by reference in their entirety into this application.
  • KPIs key performance indicators
  • Other metrics to gauge the organization's status and to assist in making strategic decisions.
  • KPIs and metrics are increasingly part of news reporting as well (the level and percent change in the Dow Jones Industrial Average, the S&P 500 Index, the stock price of a key company, or the level and change in new weekly unemployment insurance claims, as examples).
  • Current approaches for monitoring such metrics rely on dashboards, data catalogs, and KPI trackers to provide a user with information about specific KPIs.
  • the conventional approaches have limitations and disadvantages.
  • the conventional approaches provide information about KPIs in relative isolation from other factors.
  • conventional approaches do not perform the tracking and monitoring of key metrics in the context of the modeling and statistical association work that is done by modern data science and analytics teams. This limits the ability of users to understand the significance of changes in KPIs and how those changes may be related to or may influence other metrics. This prevents a user from obtaining a more complete and more accurate understanding of the relationships between the various metrics, the data used to generate the metrics, and the performance of the company (or other entity) that generated the underlying data.
  • Embodiments of the systems and methods described herein are directed to solving these and related problems individually and collectively.
  • Embodiments of the disclosure are directed to a system and methods for improving the ability of a business or other entity to monitor business related metrics (such as KPIs) and the evaluation of the quality of the underlying data used to generate those metrics.
  • the disclosed systems and methods may comprise elements, components, functions, operations, or processes that are configured and operate to provide one or more of:
  • the disclosure is directed to a system for improving the ability of a business or other entity to monitor business related metrics (such as KPIs) and the evaluation of the quality (and hence accuracy and reliability) of the underlying data.
  • the system may include a set of computer-executable instructions stored in (or on) one or more non-transitory computer-readable media, and an electronic processor or co-processors. When executed by the processor or co-processors, the instructions cause the processor or co-processors (or an apparatus or device of which they are part) to perform a set of operations that implement an embodiment of the disclosed method or methods.
  • the disclosure is directed to one or more non-transitory computer-readable media including a set of computer-executable instructions, wherein when the set of instructions are executed by an electronic processor or co-processors, the processor or co-processors (or an apparatus or device of which they are part) perform a set of operations that implement an embodiment of the disclosed method or methods.
  • the systems and methods described herein may provide services through a SaaS or multi-tenant platform.
  • the platform provides access to multiple entities, each with a separate account and associated data storage.
  • Each account may correspond to a user, set of users, an entity providing datasets for evaluation and use in generating business-related metrics, or an organization, for example.
  • Each account may access one or more services, a set of which are instantiated in their account, and which implement one or more of the methods or functions described herein.
  • FIG. 1 ( a ) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a platform architecture 100 in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented;
  • FIG. 1 ( b ) is a flow chart or flow diagram illustrating a process, method, function, or operation for constructing a Feature Graph 150 using an implementation of an embodiment of the systems and methods disclosed herein;
  • FIG. 1 ( c ) is a flow chart or flow diagram illustrating a process, method, function, or operation for an example use case in which a Feature Graph is traversed to identify potentially relevant datasets, and which may be implemented in an embodiment of the systems and methods disclosed herein;
  • FIG. 1 ( d ) is a diagram illustrating an example of part of a Feature Graph data structure that may be used to organize and access data and information, and which may be created using an implementation of an embodiment of the system and methods disclosed herein;
  • FIG. 2 ( a ) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a platform architecture in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented. Specifically, FIG. 2 ( a ) depicts how a change in features from a dataset stored in a cloud database service may be monitored using an implementation of the disclosed Metrics Monitoring capability;
  • FIG. 2 ( b ) is a flow chart or flow diagram illustrating a set of elements, components, functions, processes, or operations that may be executed as part of a platform architecture in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented. Specifically, FIG. 2 ( b ) depicts certain of the steps in FIG. 2 ( a ) with a greater focus on the different user interactions and software elements that contribute to how the Metrics Monitoring functionality is implemented and made available to users;
  • FIG. 2 ( c ) is an example of a user interface display illustrating the most recent value, the percent change to that value and identification of the subpopulation with the biggest change (which can be calculated when the metric is created as an aggregation of values in a table where there are multiple subpopulations/dimensions in the data);
  • FIG. 2 ( d ) is an example of a user interface display illustrating the Metrics Monitoring panel on the page for Weekly Active User, a metric.
  • Metrics Monitoring is turned on for other metrics, and the edges between the nodes in the graph contain metadata that describe the statistical relationships between the metrics;
  • FIG. 2 ( e ) is an example of a user interface display illustrating the platform Catalog view of Metrics Monitoring, where it is turned on for the eight metrics on this page;
  • FIG. 2 ( f ) is an example of a user interface display illustrating a notification or notifications for the Metrics Monitoring function
  • FIG. 2 ( g ) is an example of a user interface display illustrating a simplified rule setting dialog.
  • the condition that will apply to this metric will be when the absolute value of the percent change is strictly greater than 4.5;
  • FIG. 2 ( h ) is a diagram illustrating elements, components, or processes that may be present in or executed by one or more of a computing device, server, platform, or system configured to implement a method, process, function, or operation in accordance with some embodiments;
  • FIGS. 3 - 5 are diagrams illustrating an architecture for a multi-tenant or SaaS platform that may be used in implementing an embodiment of the systems and methods described herein.
  • the present disclosure may be embodied in whole or in part as a system, as one or more methods, or as one or more devices.
  • Embodiments of the disclosure may take the form of a hardware implemented embodiment, a software implemented embodiment, or an embodiment combining software and hardware aspects.
  • one or more of the operations, functions, processes, or methods described herein may be implemented by one or more suitable processing elements (such as a processor, microprocessor, CPU, GPU, TPU, or controller, as non-limiting examples) that is part of a client device, server, network element, remote platform (such as a SaaS platform), an “in the cloud” service, or other form of computing or data processing system, device, or platform.
  • suitable processing elements such as a processor, microprocessor, CPU, GPU, TPU, or controller, as non-limiting examples
  • remote platform such as a SaaS platform
  • an “in the cloud” service or other form of computing or data processing system, device, or platform.
  • the processing element or elements may be programmed with a set of executable instructions (e.g., software instructions), where the instructions may be stored on (or in) one or more suitable non-transitory computer-readable data storage media or elements.
  • the set of instructions may be conveyed to a user through a transfer of instructions or an application that executes a set of instructions (such as over a network, e.g., the Internet).
  • a set of instructions or an application may be utilized by an end-user through access to a SaaS platform or a service provided through such a platform.
  • one or more of the operations, functions, processes, or methods described herein may be implemented by a specialized form of hardware, such as a programmable gate array, application specific integrated circuit (ASIC), or the like.
  • ASIC application specific integrated circuit
  • an embodiment of the disclosure may be implemented in the form of an application, a sub-routine that is part of a larger application, a “plug-in”, an extension to the functionality of a data processing system or platform, or other suitable form. The following detailed description is, therefore, not to be taken in a limiting sense.
  • the systems and methods described herein may provide services through a SaaS or multi-tenant platform.
  • the platform provides access to multiple entities, each with a separate account and associated data storage.
  • Each account may correspond to a user, set of users, an entity, or an organization, for example.
  • Each account may access one or more services, a set of which are instantiated in their account, and which implement one or more of the methods or functions described herein.
  • Embodiments of the disclosure are directed to a system and methods for improving the ability of a business or other entity to monitor business related metrics (such as KPIs) and to evaluate the quality of the underlying data used to generate those metrics.
  • business related metrics such as KPIs
  • discovery of and access to data is made more efficient by representing data in a particular format or structure.
  • the format or structure may include labels for one or more columns, rows, or fields in a data record.
  • Conventional approaches to identifying and discovering data of interest are typically based on semantically matching words with labels in (or referring to, or about) a dataset. While this method is useful for discovering and accessing data about a topic (a target or an outcome, for example) which may be relevant, it does not address the problem of discovering and accessing data about variables that cause, affect, predict, or are otherwise statistically associated with a topic of interest.
  • Embodiments of the system and methods disclosed herein may include the construction or creation of a graph database.
  • a graph is a set of objects that are presented together if they have some type of close or relevant relationship.
  • An example is two pieces of data that represent nodes and that are connected by a path.
  • One node may be connected to many nodes, and many nodes may be connected to a specific node.
  • the path or line connecting a first and a second node or nodes is termed an “edge”.
  • An edge may be associated with one or more values; such values may represent a characteristic of the connected nodes, or a metric or measure of the relationship between a node or nodes (such as a statistical parameter), as non-limiting examples.
  • a graph format may make it easier to identify certain types of relationships, such as those that are more central to a set of variables or relationships, or those that are less significant.
  • Graphs typically occur in two primary types: “undirected”, in which the relationship the graph represents is symmetric, and “directed”, in which the relationship is not symmetric (in the case of directed graphs, an arrow instead of a line may be used to indicate an aspect of the relationship between the nodes).
  • Feature Graph is a graph or diagram that includes nodes and edges, where the edges serve to “connect” a node to one or more other nodes.
  • a node in a Feature Graph may represent a variable (i.e., a measurable quantity), an object, a characteristic, a feature, or a factor, as examples.
  • An edge in a Feature Graph may represent a measure of a statistical association between a node and one or more other nodes.
  • the association may be expressed in numerical and/or statistical terms and may vary from an observed (or possibly anecdotal) relationship to a measured correlation, to a causal relationship, as examples.
  • the information and data used to construct a Feature Graph may be obtained from one or more of a scientific paper, an experiment, a result of a machine learning model, human-made or machine-made observations, or anecdotal evidence of an association between two variables, as non-limiting examples.
  • a Feature Graph may be constructed by accessing a set of sources that include information regarding a statistical association between a topic of a study and one or more variables considered in the study.
  • the information contained in the sources is used to construct a data structure or representation that includes nodes and edges connecting nodes. Edges may be associated with information regarding the statistical relationship between two nodes.
  • One or more nodes may have a dataset associated with it, with the dataset accessible using a link or other form of address or access element.
  • Embodiments may include functionality that allows a user to describe and execute a search over the data structure to identify datasets that may be relevant to training a machine learning model, with the model being used in making a specific decision or classification.
  • embodiments may generate a data structure which includes nodes, edges, and links to datasets.
  • the nodes and edges represent concepts, topics of interest, or a topic of a previous study.
  • the edges represent information regarding a statistical relationship between nodes.
  • Links (or another form of address or access element) provide access to datasets that establish (or support, demonstrate, etc.) a statistical relationship between one or more variables that were part of a study, or between a variable and a concept or topic.
  • Data Quality refers to the appropriateness and applicability of collected or acquired data for use in data analyses and machine learning (ML) modeling.
  • the assessment of data quality may include collecting information or facts about the data, such as source(s), date(s) of collection, and information about the collection process, as well as verification of different statistical properties of the data. These statistical properties may be used to identify datasets that are “better” (that is, more accurate or reliable) candidates for use in training a model or in evaluating the performance of a business or other entity.
  • assessing statistical characteristics of a dataset typically involves writing custom computer code to either query databases or otherwise access data, and then applying rules or heuristics (using additional custom code) to determine whether accessed data (or subsets contained within that data) are within the bounds of the rules or heuristics. This places a burden on many entities and requires an allocation of resources which they may not have access to or be able to afford.
  • Machine learning includes the study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying instead on identifying patterns and applying inference processes.
  • Machine learning algorithms build a mathematical “model” based on sample data (known as “training data”) and information about what the data represents (termed a label or annotation), to make predictions, classifications, or decisions without being explicitly programmed to perform the task.
  • Machine learning algorithms are used in a wide variety of applications, including email filtering and computer vision, where it is difficult or not feasible to develop a conventional algorithm to effectively perform the task. Because of the importance of the ML model being used for a task, researchers and developers of machine learning based applications spend time and resources to build the most “accurate” predictive models for their use-case.
  • the evaluation of a model's performance and the importance of each feature in the model are typically represented by specific metrics that are used to characterize the model and its performance. These metrics may include, for example, model accuracy, the confusion matrix, Precision (P), Recall (R), Specificity, the F1 score, the Precision-Recall curve, the ROC (Receiver Operating Characteristics) curve, or the PR vs. ROC curve.
  • P Precision
  • R Recall
  • Specificity the F1 score
  • the Precision-Recall curve the Precision-Recall curve
  • the ROC Receiveiver Operating Characteristics
  • KPIs key performance indicators
  • KPIs key performance indicators
  • Many company leadership teams are focused on maintaining KPI growth or otherwise using KPIs as the primary “signals” or indicators for the health or performance of their companies.
  • the importance of KPIs to business decisions and the quality of the data used in generating those KPIs are related. This is because the utility of KPIs and the justification for using them as indicators for company or team performance depends on their applicability and the statistical (or other) measure of the accuracy and/or reliability of the underlying data used to calculate a KPI.
  • Companies may invest in analysts and engineers to build “dashboards” and other analytics tools to highlight levels and changes in their company's KPIs and inform decision makers regarding those changes.
  • the characteristics of a dataset can be important factors in selecting training data and interpreting the results from a trained model. This can be particularly important in a business setting where data generated by a business is being used as training data or an input to a trained model to generate a metric of interest to the company.
  • a trained model may be used to generate a KPI that represents an aspect of the operation of the business, such as revenue growth, profit margin, marketing costs, or sales conversion rate, as non-limiting examples.
  • the described user interface (UI) and user experience (UX) may be implemented as part of an underlying data analysis platform, such as the System platform referenced herein, and described in U.S. patent application Ser. No. 16/421,249 (now issued U.S. Pat. No. 11,354,587), entitled “Systems and Methods for Organizing and Finding Data”.
  • the disclosed platform discovers, stores, and in some cases may generate statistical relationships between data, concepts, variables, or other features. The relationships may be generated from machine learning models or programmatically run correlations.
  • the disclosed Metrics Monitoring functionality provides a way to leverage the System data organization and analysis platform to show levels and changes in KPIs, similar to how conventional approaches such as dashboards, data catalogs, and KPI trackers may do.
  • the metadata about the “status” of a metric (such as its level and changes over time) may be displayed along with the relationship of that metric to other metrics that are measured or otherwise being monitored.
  • the Metrics Monitoring functionality shows each metric's level and change in the context of those levels, along with changes in other metrics.
  • this context is not based purely on concurrency (which can lead to spurious associations between metrics and incorrect causal assumptions), but on statistical relationships driven by the platform's underlying cataloging of machine learning model and correlation-based associations.
  • Metrics Monitoring capability is designed to be a part of the disclosed platform, one of ordinary skill in the art (e.g., a software engineer with an understanding of graph databases and HTTP requests) should find the disclosure enabling and be able to implement a metrics monitoring capability in the programming language of their choosing. Since the purpose of Metrics Monitoring is to track changes in important KPIs/metrics, Metrics Monitoring assumes that there is a source of data that is updating in an event-driven or otherwise automated fashion (which is often the case for datasets that are stored in cloud database services).
  • Metrics Monitoring can be valuable to users in the financial services sector, where data is assumed to be updated on a nearly continuous basis, but it may also be used by individuals conducting scientific research and working with administrative data (often published by governmental entities), which might be updated at a quarterly, annual, or even decennial rate.
  • FIG. 1 ( a ) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a platform architecture 100 in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented.
  • a brief description of the example architecture is provided below:
  • a Feature Graph that contains a specified set of variables, topics, targets, or factors may be constructed.
  • the Feature Graph for a particular user may include all the data and information in the platform database 108 or a subset thereof.
  • the Feature Graph ( 110 in FIG. 1 ( a ) ) for a specific Customer 104 may be constructed based on selecting data and information from SystemDB 108 that satisfy conditions such as the applicability of a given domain (e.g., public health) to the domain of concern of a customer (e.g., media).
  • a given domain e.g., public health
  • a customer e.g., media
  • data in database 108 may be filtered to improve performance by removing data that would not be relevant to the problem, concept, or topic being investigated.
  • the data used to generate a Feature graph may be proprietary to an organization or user.
  • the data used to construct a Feature graph may be obtained from an experiment, a set of customers or users, or a specific database of protected data, as non-limiting examples.
  • FIG. 1 ( b ) is a flow chart or flow diagram illustrating a process, method, function, or operation for constructing a Feature Graph 150 using an implementation of an embodiment of the systems and methods disclosed herein.
  • FIG. 1 ( c ) is a flow chart or flow diagram illustrating a process, method, function, or operation for an example use case in which a Feature Graph is traversed to identify potentially relevant datasets and/or perform another function of interest (such as one resulting from execution of a specific application, such as those suggested by element 112 in FIG. 1 ( a ) ), and which may be implemented in an embodiment of the systems and methods disclosed herein.
  • a Feature Graph is constructed or created by identifying and accessing a set of sources that contain information and data regarding statistical associations between variables or factors used in a study (as suggested by step or stage 152 ). This type of information may be retrieved on a regular or continuing basis to provide information regarding variables, statistical associations and the data used to support those associations (as suggested by 154 ). As disclosed herein, this information and data is processed to identify variables used or described in those sources, and the statistical associations between one or more of those variables and one or more other variables.
  • sources of data and information are accessed.
  • the accessed data and information are processed to identify variables and statistical associations found in the source or sources 154 .
  • processing may include image processing (such as OCR), natural language processing (NLP), natural language understanding (NLU), or other forms of analysis that assist in understanding the contents of a journal paper, research notebook, experiment log, or other record of a study or investigation.
  • Further processing may include linking certain of the variables to an ontology (e.g., the International Classification of Diseases) or other set of data that provides semantic equivalents or semantically similar terms to those used for the variables (as suggested by step or stage 156 ). This assists in expanding the variable names used in a specific study to a larger set of substantially equivalent or similar entities or concepts that may have been used in other studies.
  • an ontology e.g., the International Classification of Diseases
  • the variables which, as noted may be known by different names or labels
  • statistical associations are stored in a database ( 158 ), for example SystemDB 108 of FIG. 1 ( a ) .
  • the results of processing the accessed information and data are then structured or represented in accordance with a specific data model (as suggested by step or stage 160 ); this model will be described in greater detail herein, but it generally includes the elements used to construct a Feature Graph (i.e., nodes representing a topic or variable, edges representing a statistical association, measures including a metric or evaluation of a statistical association).
  • the data model is then stored in the database ( 162 ); it may be accessed to construct or create a Feature Graph for a specific user or set of users.
  • the process or operations described with reference to FIG. 1 ( b ) enable the construction of a graph containing nodes and edges linking certain of the nodes (an example of which is illustrated in FIG. 1 ( d ) ).
  • the nodes represent topics, targets or variables of a study or observation
  • the edges represent a statistical association between a node and one or more other nodes.
  • Each statistical association may be associated with one or more of a numerical value, model type or algorithm, and statistical properties that describe the strength, confidence, or reliability of a statistical association between the nodes (i.e., the variables, factors, or topics) connected by the edge.
  • the numerical value, model type or algorithm, and the statistical properties associated with the edge may be indicative of a correlation, a predictive relationship, a cause-and-effect relationship, or an anecdotal observation, as non-limiting examples.
  • FIG. 1 ( c ) is a flow chart or flow diagram illustrating a process, method, function, or operation 190 that may be used to construct a Feature Graph for a user, in accordance with an embodiment of the disclosed system and methods. In one embodiment, this may include the following steps or stages (some of which are duplicative of those described with reference to FIG. 1 ( b ) ):
  • FIG. 1 ( d ) is a diagram illustrating an example of part of a Feature Graph data structure 198 that may be used to organize and access data and information, and which may be created using an implementation of an embodiment of the system and methods disclosed herein.
  • a description of the elements or components of the Feature Graph 198 and the associated Data Model implemented is provided below.
  • the primary objects in a Feature Graph will typically include one or more of the following, with an indication of information that may be helpful to define that object:
  • Feature Graph is to enable a user to search a Feature Graph for one or more datasets that contain variables that have been demonstrated to be statistically associated with a target topic, variable, or concept of a study.
  • variables that have been demonstrated to be statistically associated with a target topic, variable, or concept of a study.
  • a user may input a concept (represented by C1 in 198 of FIG. 1 ( d ) ) such as “crime”, “wealth”, or “hypertension”.
  • a concept represented by C1 in 198 of FIG. 1 ( d )
  • the system and methods disclosed herein may identify one or more of the following using a combination of semantic and/or statistical search techniques:
  • FIG. 2 ( a ) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a platform architecture in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented.
  • FIG. 2 ( b ) is a flow chart or flow diagram illustrating a set of elements, components, functions, processes, or operations that may be executed as part of a platform architecture in which an embodiment of the disclosed system and methods for metrics monitoring may be implemented.
  • FIG. 2 ( b ) depicts certain of the steps in FIG. 2 ( a ) with a greater focus on the different user interactions and software elements that contribute to how the Metrics Monitoring functionality is implemented and made available to users.
  • FIG. 2 ( a ) depicts how a change in features from a dataset stored in a cloud database service (or “Data Warehouse” 204 ) may be monitored using an implementation of the disclosed Metrics Monitoring capability.
  • the blocks (for example, Dataset Metadata 206 ) representing elements, functions, or operations in the left column (indicated by element 202 ) are examples of how features and metrics are represented on the System platform (along with the measured statistical relationship between features), while the blocks representing elements, functions, or operations on the right side (indicated by element 203 ) illustrate user interactions, user inputs, and software computations or other executed code that the platform may use to process and store metadata about a dataset and its features.
  • the steps, stages, functions, operations, or processing flow illustrated in FIG. 2 ( a ) may include processing steps by which the platform's Data Warehouse Retrieval Integration computes and sends (typically via HTTP requests) relevant metadata to the platform's Backend APIs.
  • the Backend services store the metadata to the platform's Graph Database (such as element 108 of FIG. 1 ( a ) ), which contains the data that supports the Feature Graph functionality.
  • the Feature Graph is what users see and interact with using the platform's frontend and generated user interfaces.
  • Metrics Monitoring provides users with visual indications (on the Feature Graph) depending on the values or changes in values in the metrics (as well as in the platform's underlying data) and may generate alerts and notifications in emails or within the platform application itself.
  • the Metrics Monitoring functionality or capability will show changes in metrics in context with each other—as suggested in FIG. 2 ( a ) , for example, users of the platform will be able to see changes in Metric One ( 208 ) alongside changes in Metric Two ( 210 ), with a description of the statistical relationship measured between those metrics (as suggested by data 209 and 211 , respectively).
  • the platform's context for showing the changes in both metrics displays not only current levels and changes in metrics, but also may use output from machine learning models and other statistical relationships between the underlying features connected to the metrics to generate and display data and information to a user.
  • FIG. 2 ( b ) depicts certain of the steps in FIG. 2 ( a ) with a greater focus on the user interactions and software elements that contribute to how the Metrics Monitoring functionality is implemented and made available to users.
  • Each step, stage, element, function, or operation of the figure corresponds to a software component (or a software service) of the disclosed platform that contributes to a user being able to use the Metrics Monitoring capability.
  • the components shown are (in top-to-bottom sequence in the figure):
  • the disclosed platform includes, as a part of its architecture, software to automatically retrieve and process data from remote databases and write the computed metadata to a platform data storage (including metadata on the statistical relationships between features in datasets).
  • This architecture is based on microservices that are designed to run on a scheduled and/or event-driven basis. However, this form of implementation may not be required if the updated data is “retrieved” from a source and written to a storage location where the Metrics Monitoring software and functionality can access it. As mentioned, it is desirable for purposes of implementing the metrics monitoring functionality that the data is retrieved in a fashion where the values of interest of the data are associated with specific time periods or other form of index.
  • an associative array in JavaScript can be used to associate values of data with specific timestamp objects: ⁇ “2010-01-01 00:00:00Z”: 10.4, “2010-01-02 00:00:00Z”: 11.2 ⁇ , where the “keys” of this associative array represent timestamps in the “UTC” time standard, and the numbers following a key represent values of data that are associated with those timestamps.
  • This is one non-limiting example of a data structure that can hold numerical values and associate them with specific timestamps.
  • Embodiments may include specific ways of interpolating and aggregating data over different time periods and specifying the data values that should be associated with a time period.
  • the Metrics Monitoring functionality disclosed herein will assist users regardless of the method used to “decide” the time period or index associated with each value; however, since users will typically depend on the data to understand how metrics of interest are changing over time, the methodology for doing so should be made clear to the user.
  • software that implements the Metrics Monitoring functionality may include the following data organization operations or processes:
  • the disclosed Metrics Monitoring functionality is intended to provide users with the full statistical context and relationships of their monitored KPIs or other metrics.
  • the platform frontend depicts the feature graph that is constructed using the platform's architecture and the metadata it collects and identifies.
  • the visual cues from the Metrics Monitoring functionality combine with the visual cues of a feature graph to assist users to develop a deeper and fuller understanding of how the data in the graph are related.
  • the user interface (UI) displays associated with the Metrics Monitoring capability are generated from data stored on the platform backend.
  • the platform frontend applies a defined monitoring rule (or rules) to the most recent value of a metric and to any relevant previous values, and the view provided to a user by the platform may change as a result.
  • frontend JavaScript code is used (before rendering the visual representation of the metrics node, either in the feature graph that is part of the platform or for a specific Metric page generated by the platform) to process the defined rule, which is typically stored on the Metric object itself.
  • a rule may be expressed as a collection of the following:
  • a rule can be selected or defined in one or more places within the platform architecture where metadata about the metric can be edited. In one embodiment, this includes the Metric page, Metric “cards” (where metrics are referenced as part of other objects, such as in Models or Datasets), and in a Matching Console, where users can match Metrics to features. In one embodiment, the rule-setting may consist of three steps:
  • the Metrics Monitoring functionality may be performed regardless of whether a rule has been set. If a rule is not set, then the representation of the metric does not trigger an alert (either via notification or visually on the platform), but the latest value, the immediately previous value, and the percent change between the two values may be displayed wherever the metric is displayed (e.g., in the platform graph, on metric pages, and/or in a catalog of metrics being tracked).
  • the metric values are generated by the platform frontend using a graph query that finds the appropriate values of features used to measure the selected metric.
  • a graph query that finds the appropriate values of features used to measure the selected metric.
  • that feature is used for the Metrics Monitoring values. If multiple features that have time-specific data are connected to the metric, then the first feature that was connected to the metric is, by default, the feature used for Metrics Monitoring values (although a user may change this default to another feature).
  • the feature that supplies the values for Metrics Monitoring may be displayed at the top of the Metrics page, along with a link to the feature so that a user can examine each of the features used to generate the Metrics Monitoring data.
  • the disclosed platform and data model capture information about datasets and models to help users manage, discover, and use the statistical relationships generated from correlations and associations made by machine learning models.
  • the platform data model specifies features, datasets, models, and other objects as nodes, and the platform is built using a graph architecture to store edges between those objects and platform-created objects which encode information about those relationships.
  • the platform tracks (and may compute) relationship strength based on the statistical properties of datasets and models.
  • the platform may be regularly updated with scientific standards for how to assess relationship strength, starting with standard measures of statistical significance (such as computed confidence intervals and various forms of statistical hypothesis testing), statistical “rules of thumb,” (such as traditionally accepted levels of effect sizes as defined by Cohen (1962)), and other sources of specific domain knowledge encoded into the platform's backend and machine learning pipelines.
  • the disclosed Metrics Monitoring capability and functionality provides a user with regularly updated metric values from different data sources and may inform the user of important or significant changes in metric levels or metric growth rates.
  • the feature graph may be used to inform users about changes in KPIs/metrics that can or should be expected.
  • Correlations and machine learning models added to the platform that include data from a current time period may be incorporated into the measurement of statistical relationships; this has the effect of enabling the platform to continually “learn” and improve the knowledge and data that users can access and utilize in making decisions.
  • the graph database includes feature nodes, which may be connected to nodes that summarize the statistical information for each of the features, and edges between features and “association” nodes, which aggregate and summarize the statistical relationship(s) between features.
  • the feature nodes may also have edges to metrics nodes, where users (and the platform) store metadata about a metric, and the tracking or supporting information for the metric.
  • the disclosed systems and methods provide users with the ability to monitor business related metrics (such as KPIs) and more efficiently evaluate the quality of the underlying data used to generate those metrics.
  • This capability is expected to enable users to make more informed decisions regarding the operation of a business.
  • this may include implementation of one or more of the following functions or capabilities:
  • the disclosed metrics monitoring capability and functionality improve the KPI (or other metric) monitoring and data quality analysis process in an integrated fashion.
  • the metrics monitoring capability provides data quality monitoring that measures statistical properties of datasets, such as (but not limited to) the rate of missing observations in data, or changes in summary statistics (the minimum, maximum, or mean, as examples), and allows users to visualize and understand changes in data in a contextual environment.
  • a user may receive an alert or notification indicating a change in data, where these changes are compared across datasets from different sources and are displayed alongside relevant metadata about the data sources and/or the monitored metrics.
  • the disclosed system and methods also display monitored metrics in a graphical format or representation as part of (or in conjunction with) a feature graph. This enables important statistical relationships between metrics to be recognized and enables a user to identify the “co-movement” of important metrics. This capability provides users with an efficient and effective way of assessing the current level and/or growth rate of a metric and to anticipate the future level(s) and growth rates of related metrics.
  • an embodiment of the disclosed system and methods for monitoring metrics and evaluating the statistical associations of underlying datasets may be used in conjunction with the referenced platform operated by the assignee.
  • This platform may be used to reveal to users underlying relationships that drive tasks, teams, companies, and communities.
  • the task of data teams is to create understanding through the collection and analysis of data.
  • the disclosed platform can be used to aggregate that information and display to users the environment and context of the resulting knowledge.
  • teams may measure KPIs or other metrics to gauge the relative health of specific parts of their teams, companies, or communities.
  • the disclosed metrics monitoring functionality provides those teams with a better and more complete understanding of a team's (or company's or community's) health, as reflected or indicated by a set of metrics.
  • This automated retrieval capability allows the platform to store time-indexed statistical metadata.
  • a time-indexed feature such as a variable or parameter
  • users can indicate through a user interface that this is a metric that they would like to monitor. If a metric is monitored, then the user may be shown the current “level” of the data used to measure or determine the value of the metric, in addition to the previous value, and (in some embodiments) the percentage change between the previous and current values.
  • the metrics monitoring functionality is not dependent on an automatic retrieval functionality. Instead, when features exist with time indices, a user may be offered the same tools and may “monitor” the metric. This may include metrics that are not actually stored in a database, such as the values of a machine learning model's performance metrics, or the value of different features of importance in a model. These values can also be set for monitoring by a user.
  • a user may specify “rules” for monitoring a metric based (for example) on either the levels (the values of the metric) and/or percent changes between the current and previous values of the metric.
  • the Metrics Monitoring capability can also (or instead) recommend rules, based on similarly monitored metrics, where similarity may be determined by one or more of the statistical properties of the metric, semantic analysis of the name of the metric, or a user's previously specified Metrics Monitoring rules (as non-limiting examples).
  • Such “recommendations” may include prompts to the user of the form “The recommended threshold for changes in mean is 2.2% (this occurs in 5% of observations).”
  • the form of a user defined, or platform proposed rule depends on the structure and values of the data, but commonly includes rules based on (as examples):
  • a user may specify multiple rules and can specify whether to be notified/alerted when a specific rule is “violated” or if all the rules are “violated”, where a “violation” of a rule is when the condition specified by the rule is present or satisfied. That is, if the user sets a rule for a metric to be monitored when the value is negative, whenever the metric's value is negative the rule is said to be “violated”—i.e., the condition set in the rule is satisfied.
  • the platform may display whether the value (if rules are based on the value) or the change in value (if rules are based on the most recent change in value) is in “violation” of the set rule(s).
  • a “violation” represents an “alert” or notification generation state, and in response the platform may change the display of the value (or change in value) in a manner specified by the user.
  • a user may be provided with choices as to how the display changes—for example, by setting a color for the alert state and/or choosing an icon to be shown alongside the value or change in value.
  • a default change to the display of the metric is to show the value (or change in value, depending on the rule applied) in red when the rule is in the alert state (when the rule is “violated”) and in green when the rule is not in an alert state.
  • the monitoring may display a default color, which may be black.
  • the Metrics Monitoring functionality can provide users with monitoring of objects with which they are not yet familiar.
  • a team might be focused on KPIs and set up the Metrics Monitoring functionality with specific rules. Since the platform is capturing metadata and relationships between metrics, it may be the case that a different metric (or set of metrics), or a performance metric from a machine learning model that has been added to the platform is a “good” predictor or leading indicator of a monitored metric. In this situation, the platform's Metrics Monitoring function may suggest that this metric be monitored and can provide recommendations for more comprehensive and improved monitoring based on machine-learned relationships in the metadata added to the platform.
  • the platform has software processes that automatically calculate statistical relationships between different features and measures the relative strength of those relationships according to a calibration process.
  • closely related metrics can be identified via query, and when a newly-added metric is closely related to a metric that is currently being monitored, this information can be stored in the graph itself.
  • the platform can then prompt users with the appropriate role-based access with a suggestion to open the monitoring model and apply monitoring rules to a newly added metric.
  • the calibration process will continue to identify new metrics in the same fashion and can also identify existing metrics that are highly related to the set of metrics already being monitored.
  • Determined correlations or machine learning model outputs calculated using the data connected to these metrics are viewable and navigable on the platform generated feature graph, so a “map” of the company's core metrics will be viewable, navigable, and shareable.
  • An enterprise user might access the platform regularly to examine the levels of the core metrics and/or to see how a data team's work is creating additional (or improving existing) statistical relationships between the company's core metrics.
  • the Metrics Monitoring capability allows a user to track the important metrics that they use to gauge a company's operational status, and the platform feature graph allows them to find connections and/or relationships between metrics. For example, a user might select a UI element connecting two metrics to discover a colleague's models that explored how one metric can be used to “predict” another, as knowing these relationships can provide a more accurate and reliable understanding of operational status.
  • the metadata from models and correlations can quantify the predictive relationship between the average waiting time for orders and the likelihood that a customer reorders from a company, and thereby improve the company's decision making in several areas (e.g., marketing, fulfillment processing, or inventory management).
  • a user of a public version of the platform might encounter the Metrics Monitoring functionality through browsing a part of the platform feature graph that they are interested in.
  • the public version of the platform may have a metric defined as “Global Nitrogen Dioxide Emissions”. This metric might be connected to a feature that is part of a dataset published by NASA that measures global atmospheric emissions levels, and a user might have used that feature as the basis for Metrics Monitoring of Global Nitrogen Dioxide Emissions.
  • the public platform UI will then show Global Nitrogen Dioxide Emissions as a metric, and users can visit the metric's page to obtain information on levels or growth changes reported from the metadata retrieved from NASA's published dataset.
  • connections to other metrics are made, created, or discovered by the platform (whether through specific machine learning modeling, or based on statistical correlations that are computed between the features in the dataset and other features tracked over time on the platform), the connections will be displayed in the graph. This will enable the user to see if other metrics are related to nitrogen dioxide emissions.
  • the user will be able to see the levels and recent changes for those related metrics and can use the links provided in the platform feature graph to access the statistical and/or scientific basis for the relationships displayed in the graph (and if desired, observe the extent to which those relationships grow stronger or weaker over time).
  • this information can be made available to other applications via HTTP API requests (such as by gRPC, REST, and/or GraphQL requests).
  • HTTP API requests such as by gRPC, REST, and/or GraphQL requests.
  • a call to a metric endpoint will return the platform's metadata about metric(s)
  • a call to a metrics/associations endpoint will return metadata about which metrics are related to a given metric (and details about the statistical relationship, such as the evidence that substantiates the relationship and the types of models or correlations that contribute to the relationship).
  • the metadata made available for metrics that are relevant to the Metrics Monitoring functionality may include one or more of:
  • the data that generates the view(s) or display(s) provided by the platform can be used by a data journalist who covers financial markets.
  • the data journalist might query for metrics that have had levels or recent changes that have exceeded predefined thresholds, and then use queries to find related metrics.
  • the information contained in responses to these queries will provide the statistical context for why a metric of interest is at a certain level (or had changes of a particular magnitude) and provide a statistical basis for why other historically related metrics might be expected to move in a certain direction.
  • the data journalist might see that the price of silver traded in a particular commodities market has experienced a significant drop—modeling or correlations calculated using the price of silver would then inform the journalist what other market forces have recently (or historically) been associated with changes in the price of silver, and what further changes in the market might ensue.
  • the Metrics Monitoring capability can be utilized on data collected from different types of sources, including data that is generated from the platform itself.
  • model performance metrics may be collected according to a regular time interval.
  • This type of data can also be attached to a metric for monitoring, and statistical relationships between tracked model performance metrics and other measured metrics on the platform can be established (through correlation analysis or explicit modeling). This enables users of the platform to use Metrics Monitoring to manage their models' performance and metrics (as these metrics are often KPIs or key metrics for data science teams) in the context of their other collected data.
  • a visual interface change or indication may be used to notify a user that this is data that can be tracked or monitored.
  • the visual interface may also enable a user to set specific rules so that they can monitor these changes with a greater degree of visual distinction and receive alerts and notifications about changes in the values for a metric.
  • Metrics Monitoring functionality can configure these rules, which are defined in terms of comparing the most recent level of a metric or the change between recent values using a predefined set of comparison operators, as well as options for how to visually indicate when a metric “violates” or satisfies a condition expressed by a rule (and how to notify the user that a “violation” has occurred).
  • rules are defined in terms of comparing the most recent level of a metric or the change between recent values using a predefined set of comparison operators, as well as options for how to visually indicate when a metric “violates” or satisfies a condition expressed by a rule (and how to notify the user that a “violation” has occurred).
  • the visual indicators on the feature graph are set to reflect the chosen colors or format (or marked with an icon for users with a color vision concern), which distinguishes monitored metrics from those that can be monitored but have no rule set for them (which remain the default color or format).
  • the platform may generate a visualization showing how an underlying feature graph has changed over time or changes that have occurred between different sets of sources. This may be useful in identifying whether a previously identified statistical relationship was substantiated by later work, or if what was believed to be a valid relationship should now be interpreted differently.
  • This capability supplements metrics monitoring by highlighting the relationship values that have changed over user-identified periods of time. Users can use metrics monitoring to quickly identify important metrics and how their values have changed over time and use this type of capability (as presented in the form of a visualization, for example) to identify whether the values of key metrics changed because the values of metrics that are (statistically) closely related have changed, or whether an underlying statistical relationship is stronger or weaker than once thought.
  • This capability can be made available automatically to platform users, replacing exploratory modeling that a data analyst or scientist might do in a response to changes in key metrics.
  • the default rules are pre-filled for users depending on what field on the metric (e.g., current value, previous value, percent change) is being used to set the monitoring rule.
  • the default rules can be configured for different teams that use the platform, as each enterprise or team account will typically have a separate workspace for data and models. This enables configuration settings, including Metrics Monitoring rules, to be stored separately for each separate enterprise or team account.
  • the monitoring rules are typically set with rule-of-thumb levels (e.g., the standard rule for metrics might be to alert in red when the percent change in a value is greater than or equal to 5% in absolute value).
  • the platform can recommend that future alerts be set according to settings that already exist for metrics that are semantically similar (i.e., having a name, description, or type that is the same or sufficiently similar). For example, a team might have set a Metrics Monitoring rule to display a “yellow” alert when the value of the “Product X Inventory” is less than 100—a suggested rule for “Product Y Inventory” or “Product X Production” for that user or team might be to set the rule the same as set for “Product X Inventory.”
  • Rules may also be suggested when metrics are statistically similar. For example, if “Product X Production” is known to be statistically related to “Product X Inventory” because of a machine learning model or other determined statistical association, the suggested rule for “Product X Production” can be the same as for the related metric, or it can be configured to suggest a rule that would occur with similar likelihood to that of the alert set for “Product X Inventory.”
  • the Metrics Monitoring function can be used to discover or “learn” and apply monitoring rules, and this capability provides an advantage over conventional solutions that require rules be set in isolation, without considering the context for different metrics in the same system.
  • the Metrics Monitoring functionality is not limited to a particular type of metadata.
  • the metrics monitoring has been described with reference to levels or percent changes of actual features in a dataset, the monitoring functionality can be applied to other metadata collected on the platform that is associated with a corresponding time element.
  • the disclosed platform is designed as a knowledge management tool for the entire data stack, and Metrics Monitoring on the platform is a monitoring, alerting, and context-driven tool for understanding movements in important metrics where the sources for these metrics are distributed.
  • the platform may conduct its own automated machine learning modeling on metadata available to the platform. Since the metadata for metrics on the platform can be indexed to the same time span, the platform can “know” or “learn” statistical relationship(s) between the daily model performances (which are stored in the feature graph) and other metrics on the platform that are retrieved from database services (or added by users) and that have a time index.
  • This capability may enable the discovery of new and significant metrics that a team is not currently monitoring and/or suggest more effective rules for metrics monitoring that highlight key inflection points for the success of a model (e.g., via tracked model performance metrics), or levels/changes in metrics that predict known critical values for other metrics. This can be done unobtrusively through recommendations presented in a rule-setting panel (e.g., by suggesting “better” rules and explaining to users what the platform is “learning” through its automated machine learning).
  • the platform can be used to take metric monitoring data (which contains time-indexed indicators for whether a metric is in an “alert” status) and execute a classification model where the previous values (“lagged” values) for other metrics are used to “predict” whether a given metric is in an alert status.
  • the results of this model can be used to identify “better” thresholds for metrics being monitored (which is the case when a particular level or change in a metric is a good predictor of a different metric being in “notification” or “alert” status), or if levels/changes in model performance metrics are predictors of other metrics' alert status (which suggests that users might want to set Metrics Monitoring for that model performance metric).
  • the number of statistical comparisons that the platform automatically executes may be limited, to avoid highlighting spurious correlations, and for reasons of computational efficiency. Since the platform's metadata includes knowledge about metrics being monitored and the ones with high usage on the platform (whether in models or in users' browsing behavior), the automated rule generation and recommendation functions can be focused on metrics and objects of relatively high interest and high statistical importance on the platform.
  • the graph may be traversed to identify variables of interest to a topic or goal of a study, model, or investigation, and if desired, to retrieve datasets that support or confirm the relevance of those variables or that measure variables of interest.
  • the process by which a Feature Graph is traversed may be controlled by one of two methods: (a) explicit user tuning of the search parameters or (b) algorithmic based tuning of the parameters for variable/data retrieval.
  • FIG. 2 ( a ) depicts how a change in features from a dataset stored in a cloud database service (or “Data Warehouse” 204 ) may be monitored using an implementation of the disclosed Metrics Monitoring capability.
  • the dataset metadata 206 is illustrated for two statistically related features, indicated as Feature One and Feature 2.
  • a first metric (Metric One 208 ) is defined, and its most recent value(s) are displayed ( 209 ).
  • a rule governing the display of an alert or notification is shown ( 212 ), and the resulting information regarding Metric One is shown in display section 214 .
  • Metric Two 210 a second metric (Metric Two 210 ) is defined, its most recent values displayed ( 211 ), a rule governing the display of an alert or notification is shown ( 213 ), and the resulting information regarding Metric Two is shown in display section 215 .
  • a data warehouse integration process 220 operates to “retrieve” datasets and features from data warehouse 204 and computes or accesses relevant metadata. This retrieval process sends http requests to the platform's backend API with dataset and feature metadata.
  • the metadata includes statistical relationships between features (as suggested by process 222 ).
  • the platform backend writes dataset, feature, and relationship metadata to the platform graph database (as suggested by process 224 ). Users can see datasets, features, and relationships at an available website. When features have time indexes associated with values (such as the examples of feature one and feature two, shown at 206 ), and users associate feature one and feature two to metric one ( 208 ) and metric two ( 210 ), users can then activate or select the metrics monitoring functionality (as suggested by process 226 ).
  • a user can activate or select the metrics monitoring functionality and then define monitoring rules, which specify (among other aspects) visual alerts and set email/application notifications (as suggested by process 228 ).
  • metrics available on the platform's frontend reflect statistical relationships between features. Users can see the monitored metrics with detailed metadata and the full statistical context (e.g., levels, percent changes, feature history, alerts, and relationships), as suggested by process 230 .
  • FIGS. 2 ( c ) through 2 ( g ) are examples of user interface displays that may be generated by a platform or system configured to discover or determine and represent statistically meaningful relations between specified metrics, datasets, and machine learning models, in accordance with embodiments of the disclosed platform and system.
  • FIG. 2 ( c ) is an example of a user interface display illustrating the most recent value ( 314 , 779 ), the percent change to that value ( ⁇ 4%) and identification of the subpopulation with the biggest change (which can be calculated when a metric is defined as an aggregation of values in a table where there are multiple subpopulations/dimensions in the data).
  • FIG. 2 ( d ) is an example of a user interface display illustrating the Metrics Monitoring panel on the page for Weekly Active User, a defined metric.
  • the data source for weekly average user (wau) is connected and has a time index, so monitoring is available.
  • wau weekly average user
  • a user can set/define a rule for monitoring, and then specify the color of the monitoring and the frequency of email alerts.
  • Metrics Monitoring is turned on for other metrics, and the edges between the nodes in the graph contain metadata that describe the statistical relationships between the metrics. Knowing which metrics are in alert status and understanding the relationships between metrics allows a user to understand statistical drivers of the KPIs/key metrics within the context of their dataset.
  • FIG. 2 ( e ) is an example of a user interface display illustrating the platform Catalog view of Metrics Monitoring, where it is turned on for the eight metrics on the displayed page. While other solutions for data monitoring may have a view that is similar in some respects (or other chart views, in the case of dashboard tools), an advantage of the Metrics Monitoring function's approach can be seen in the collection of evidence on a given metric at the bottom of each “card” or section. Each metric is used in different models (some are the predicted outcomes for models), and metadata about each metric is viewable by clicking any of the cards, as well as metadata about the relationships between any metrics that have been included in the same machine learning model or in other statistical relationships established by users or by automated machine learning.
  • FIG. 2 ( f ) is an example of a user interface display illustrating a notification or notifications for the Metrics Monitoring function.
  • the latest and most recent values are displayed, as well as the values for related metrics.
  • These relationships are created from metadata taken from machine learning models added to the platform, from relationships directly added by users, and from automated machine learning that is applied to feature metadata added by users, retrieved from database services, or generated from regular updates from tracked models deployed in production.
  • FIG. 2 ( g ) is an example of a user interface display illustrating a simplified rule setting dialog.
  • the condition that will apply to this metric will be when the absolute value of the percent change is strictly greater than 4.5.
  • FIG. 2 ( h ) is a diagram illustrating elements, components, or processes that may be present in or executed by one or more of a computing device, server, platform, or system 280 configured to implement a method, process, function, or operation in accordance with some embodiments.
  • the disclosed system and methods may be implemented in the form of an apparatus or apparatuses (such as a server that is part of a system or platform, or a client device) that includes a processing element and a set of executable instructions.
  • the executable instructions may be part of a software application (or applications) and arranged into a software architecture.
  • an embodiment of the disclosure may be implemented using a set of software instructions that are designed to be executed by a suitably programmed processing element (such as a GPU, TPU, CPU, microprocessor, processor, controller, or computing device, as non-limiting examples).
  • a suitably programmed processing element such as a GPU, TPU, CPU, microprocessor, processor, controller, or computing device, as non-limiting examples.
  • modules typically arranged into “modules” with each such module typically performing a specific task, process, function, or operation.
  • the entire set of modules may be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform.
  • OS operating system
  • the modules and/or sub-modules may include a suitable computer-executable code or set of instructions, such as computer-executable code corresponding to a programming language.
  • a suitable computer-executable code or set of instructions such as computer-executable code corresponding to a programming language.
  • programming language source code may be compiled into computer-executable code.
  • the programming language may be an interpreted programming language such as a scripting language.
  • system 280 may represent one or more of a server, client device, platform, or other form of computing or data processing device.
  • Modules 282 each contain a set of executable instructions, where when the set of instructions is executed by a suitable electronic processor (such as that indicated in the figure by “Physical Processor(s) 298 ”), system (or server, or device) 280 operates to perform a specific process, operation, function, or method.
  • a suitable electronic processor such as that indicated in the figure by “Physical Processor(s) 298 ”
  • Modules 282 may contain one or more sets of instructions for performing a method or function described with reference to the Figures, and the disclosure of the functions and operations provided in the specification. These modules may include those illustrated but may also include a greater number or fewer number than those illustrated. Further, the modules and the set of computer-executable instructions that are contained in the modules may be executed (in whole or in part) by the same processor or by more than a single processor. If executed by more than a single processor, the co-processors may be contained in different devices, for example a processor in a client device and a processor in a server.
  • Modules 282 are stored in a memory 281 , which typically includes an Operating System module 284 that contains instructions used (among other functions) to access and control the execution of the instructions contained in other modules.
  • the modules 282 in memory 281 are accessed for purposes of transferring data and executing instructions by use of a “bus” or communications line 290 , which also serves to permit processor(s) 298 to communicate with the modules for purposes of accessing and executing instructions.
  • Bus or communications line 290 also permits processor(s) 298 to interact with other elements of system 280 , such as input or output devices 292 , communications elements 294 for exchanging data and information with devices external to system 280 , and additional memory devices 296 .
  • Each module or sub-module may correspond to a specific function, method, process, or operation that is implemented by execution of the instructions (in whole or in part) in the module or sub-module.
  • Each module or sub-module may contain a set of computer-executable instructions that when executed by a programmed processor or co-processors cause the processor or co-processors (or a device, devices, server, or servers in which they are contained) to perform the specific function, method, process, or operation.
  • an apparatus in which a processor or co-processor is contained may be one or both of a client device or a remote server or platform. Therefore, a module may contain instructions that are executed (in whole or in part) by the client device, the server or platform, or both.
  • Such function, method, process, or operation may include those used to implement one or more aspects of the disclosed system and methods, such as for:
  • FIG. 3 is a diagram illustrating a SaaS system in which an embodiment may be implemented.
  • FIG. 4 is a diagram illustrating elements or components of an example operating environment in which an embodiment may be implemented.
  • FIG. 5 is a diagram illustrating additional details of the elements or components of the multi-tenant distributed computing service platform of FIG. 4 , in which an embodiment may be implemented.
  • the system or services disclosed or described herein may be implemented as micro-services, processes, workflows, or functions performed in response to the submission of a user's responses.
  • the micro-services, processes, workflows, or functions may be performed by a server, data processing element, platform, or system.
  • the data analysis and other services may be provided by a service platform located “in the cloud”.
  • the platform may be accessible through APIs and SDKs.
  • the functions, processes and capabilities may be provided as micro-services within the platform.
  • the interfaces to the micro-services may be defined by REST and GraphQL endpoints.
  • An administrative console may allow users or an administrator to securely access the underlying request and response data, manage accounts and access, and in some cases, modify the processing workflow or configuration.
  • FIGS. 3 - 5 illustrate a multi-tenant or SaaS architecture that may be used for the delivery of business-related or other applications and services to multiple accounts/users, such an architecture may also be used to deliver other types of data processing services and provide access to other applications.
  • a platform or system of the type illustrated in FIGS. 3 - 5 may be operated by a 3rd party provider to provide a specific set of business-related applications, in other embodiments, the platform may be operated by a provider and a different business may provide the applications or services for users through the platform.
  • FIG. 3 is a diagram illustrating a system 300 in which an embodiment may be implemented or through which an embodiment of the services disclosed or described may be accessed.
  • ASP application service provider
  • users of the services described herein may comprise individuals, businesses, stores, organizations, etc.
  • a user may access the services using any suitable client, including but not limited to desktop computers, laptop computers, tablet computers, scanners, smartphones, etc.
  • a user interfaces with the service platform across the Internet 308 or another suitable communications network or combination of networks. Examples of suitable client devices include desktop computers 303 , smartphones 304 , tablet computers, or laptop computers 305 .
  • Platform 310 which may be hosted by a third party, may include a set of services to assist a user to access the data processing and metrics monitoring services described herein 312 , and a web interface server 314 , coupled as shown in FIG. 3 . It is to be appreciated that either or both the services 312 and the web interface server 314 may be implemented on one or more different hardware systems and components, even though represented as singular units in FIG. 3 .
  • Services 312 may include one or more functions or operations for enabling a user to access a feature graph and perform the metrics monitoring functions disclosed herein.
  • the set of functions, operations or services made available through platform 310 may include:
  • an application module or sub-module may contain computer-executable instructions which when executed by a programmed processor cause a system or apparatus to perform a function related to the operation of the service platform.
  • Such functions may include but are not limited to those related to user registration, user account management, data security between accounts, the allocation of data processing and/or storage capabilities, providing access to data sources other than SystemDB (such as ontologies or reference materials).
  • the platform or system shown in FIG. 3 may be hosted on a distributed computing system made up of at least one, but likely multiple, “servers.”
  • a server is a physical computer dedicated to providing data storage and an execution environment for one or more software applications or services intended to serve the needs of the users of other computers that are in data communication with the server, for instance via a public network such as the Internet.
  • the server, and the services it provides, may be referred to as the “host” and the remote computers, and the software applications running on the remote computers being served may be referred to as “clients.”
  • clients Depending on the computing service(s) that a server offers it could be referred to as a database server, data storage server, file server, mail server, print server, or web server, as examples.
  • a web server is a most often a combination of hardware and the software that helps deliver content, commonly by hosting a website, to client web browsers that access the web server via the Internet.
  • FIG. 4 is a diagram illustrating elements or components of an example operating environment 400 in which an embodiment may be implemented.
  • a variety of clients 402 incorporating and/or incorporated into a variety of computing devices may communicate with a multi-tenant service platform 408 through one or more networks 414 .
  • a client may incorporate and/or be incorporated into a client application (i.e., software) implemented at least in part by one or more of the computing devices.
  • suitable computing devices include personal computers, server computers 404 , desktop computers 406 , laptop computers 407 , notebook computers, tablet computers or personal digital assistants (PDAs) 410 , smart phones 412 , cell phones, and consumer electronic devices incorporating one or more computing device components, such as one or more electronic processors, microprocessors, central processing units (CPU), or controllers.
  • suitable networks 414 include networks utilizing wired and/or wireless communication technologies and networks operating in accordance with any suitable networking and/or communication protocol (e.g., the Internet).
  • the distributed computing service/platform (which may also be referred to as a multi-tenant data processing platform) 408 may include multiple processing tiers, including a user interface tier 416 , an application server tier 420 , and a data storage tier 424 .
  • the user interface tier 416 may maintain multiple user interfaces 417 , including graphical user interfaces and/or web-based interfaces.
  • the user interfaces may include a default user interface for the service to provide access to applications and data for a user or “tenant” of the service (depicted as “Service UI” in the figure), as well as one or more user interfaces that have been specialized/customized in accordance with user specific requirements (e.g., represented by “Tenant A UI”, . . . , “Tenant Z UI” in the figure, and which may be accessed via one or more APIs).
  • the default user interface may include user interface components enabling a tenant to administer the tenant's access to and use of the functions and capabilities provided by the service platform. This may include accessing tenant data, launching an instantiation of a specific application, causing the execution of specific data processing operations, etc.
  • Each application server or processing tier 422 shown in the figure may be implemented with a set of computers and/or components including computer servers and processors, and may perform various functions, methods, processes, or operations as determined by the execution of a software application or set of instructions.
  • the data storage tier 424 may include one or more data stores, which may include a Service Data store 425 and one or more Tenant Data stores 426 . Data stores may be implemented with any suitable data storage technology, including structured query language (SQL) based relational database management systems (RDBMS).
  • SQL structured query language
  • RDBMS relational database management systems
  • Service Platform 408 may be multi-tenant and may be operated by an entity to provide multiple tenants with a set of business-related or other data processing applications, data storage, and functionality.
  • the applications and functionality may include providing web-based access to the functionality used by a business to provide services to end-users, thereby allowing a user with a browser and an Internet or intranet connection to view, enter, process, or modify certain types of information.
  • Such functions or applications are typically implemented by one or more modules of software code/instructions that are maintained on and executed by one or more servers 422 that are part of the platform's Application Server Tier 420 .
  • the platform system shown in FIG. 4 may be hosted on a distributed computing system made up of at least one, but typically multiple, “servers.”
  • a business may utilize systems provided by a third party.
  • a third party may implement a business system/platform as described above in the context of a multi-tenant platform, where individual instantiations of a business' data processing workflow are provided to users, with each business representing a tenant of the platform.
  • One advantage to such multi-tenant platforms is the ability for each tenant to customize their instantiation of the data processing workflow to that tenant's specific business needs or operational methods.
  • Each tenant may be a business or entity that uses the multi-tenant platform to provide business services and functionality to multiple users.
  • FIG. 5 is a diagram illustrating additional details of the elements or components of the multi-tenant distributed computing service platform of FIG. 4 , in which an embodiment may be implemented.
  • the software architecture shown in FIG. 5 represents an example of an architecture which may be used to implement an embodiment of the invention.
  • an embodiment of the invention may be implemented using a set of software instructions that are designed to be executed by a suitably programmed processing element (such as a CPU, GPU, microprocessor, processor, controller, or computing device).
  • a processing element such as a CPU, GPU, microprocessor, processor, controller, or computing device.
  • modules typically arranged into “modules” with each such module performing a specific task, process, function, or operation.
  • the entire set of modules may be controlled or coordinated in their operation by an operating system (OS) or other form of organizational platform.
  • OS operating system
  • FIG. 5 is a diagram illustrating additional details of the elements or components 500 of a multi-tenant distributed computing service platform, in which an embodiment may be implemented.
  • the example architecture includes a user interface layer or tier 502 having one or more user interfaces 503 .
  • user interfaces include graphical user interfaces and application programming interfaces (APIs).
  • Each user interface may include one or more interface elements 504 .
  • interface elements For example, users may interact with interface elements to access functionality and/or data provided by application and/or data storage layers of the example architecture.
  • Examples of graphical user interface elements include buttons, menus, checkboxes, drop-down lists, scrollbars, sliders, spinners, text boxes, icons, labels, progress bars, status bars, toolbars, windows, hyperlinks, and dialog boxes.
  • Application programming interfaces may be local or remote and may include interface elements such as a variety of controls, parameterized procedure calls, programmatic objects, and messaging protocols.
  • the application layer 510 may include one or more application modules 511 , each having one or more sub-modules 512 .
  • Each application module 511 or sub-module 512 may correspond to a function, method, process, or operation that is implemented by the module or sub-module (e.g., a function or process related to providing data processing and services to a user of the platform).
  • Such function, method, process, or operation may include those used to implement one or more aspects of the disclosed system and methods, such as for one or more of the processes, functions, or operations disclosed or described herein.
  • the application modules and/or sub-modules may include any suitable computer-executable code or set of instructions (e.g., as would be executed by a suitably programmed processor, microprocessor, GPU, TPU, or CPU), such as computer-executable code corresponding to a programming language.
  • a suitably programmed processor, microprocessor, GPU, TPU, or CPU such as computer-executable code corresponding to a programming language.
  • programming language source code may be compiled into computer-executable code.
  • the programming language may be an interpreted programming language such as a scripting language.
  • Each application server (e.g., as represented by element 422 of FIG. 4 ) may include each application module.
  • different application servers may include different sets of application modules. Such sets may be disjoint or overlapping.
  • the data storage layer 520 may include one or more data objects 522 each having one or more data object components 521 , such as attributes and/or behaviors.
  • the data objects may correspond to tables of a relational database, and the data object components may correspond to columns or fields of such tables.
  • the data objects may correspond to data records having fields and associated services.
  • the data objects may correspond to persistent instances of programmatic data objects, such as structures and classes.
  • Each data store in the data storage layer may include each data object.
  • different data stores may include different sets of data objects. Such sets may be disjoint or overlapping.
  • FIGS. 3 - 5 are not intended to be limiting examples.
  • Further environments in which an embodiment of the disclosure may be implemented in whole or in part include devices (including mobile devices), software applications, systems, apparatuses, networks, SaaS platforms, IaaS (infrastructure-as-a-service) platforms, or other configurable components that may be used by multiple users for data entry, data processing, application execution, or data review.
  • Machine learning is being used more and more to enable the analysis of data and assist in making decisions in multiple industries.
  • a machine learning algorithm is applied to a set of training data and labels to generate a “model” which represents what the application of the algorithm has “learned” from the training data.
  • Each element (or instances or example, in the form of one or more parameters, variables, characteristics or “features”) of the set of training data is associated with a label or annotation that defines how the element should be classified by the trained model.
  • a machine learning model in the form of a neural network is a set of layers of connected neurons that operate to make a decision (such as a classification) regarding a sample of input data. When trained (i.e., the weights connecting neurons have converged and become stable or within an acceptable amount of variation), the model will operate on a new element of input data to generate the correct label or classification as an output.
  • certain of the methods, models or functions described herein may be embodied in the form of a trained neural network, where the network is implemented by the execution of a set of computer-executable instructions or representation of a data structure.
  • the instructions may be stored in (or on) a non-transitory computer-readable medium and executed by a programmed processor or processing element.
  • the set of instructions may be conveyed to a user through a transfer of instructions or an application that executes a set of instructions (such as over a network, e.g., the Internet).
  • the set of instructions or an application may be utilized by an end-user through access to a SaaS platform or a service provided through such a platform.
  • a trained neural network, trained machine learning model, or any other form of decision or classification process may be used to implement one or more of the methods, functions, processes, or operations described herein.
  • a neural network or deep learning model may be characterized in the form of a data structure in which are stored data representing a set of layers containing nodes, and connections between nodes in different layers are created (or formed) that operate on an input to provide a decision or value as an output.
  • a neural network may be viewed as a system of interconnected artificial “neurons” or nodes that exchange messages between each other.
  • the connections have numeric weights that are “tuned” during a training process, so that a properly trained network will respond correctly when presented with an image or pattern to recognize (for example).
  • the network consists of multiple layers of feature-detecting “neurons”; each layer has neurons that respond to different combinations of inputs from the previous layers.
  • Training of a network is performed using a “labeled” dataset of inputs in a wide assortment of representative input patterns that are associated with their intended output response. Training uses general-purpose methods to iteratively determine the weights for intermediate and final feature neurons.
  • each neuron calculates the dot product of inputs and weights, adds the bias, and applies a non-linear trigger or activation function (for example, using a sigmoid response function).
  • any of the software components, processes or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as Python, Java, JavaScript, C, C++, or Perl using conventional or object-oriented techniques.
  • the software code may be stored as a series of instructions, or commands in (or on) a non-transitory computer-readable medium, such as a random-access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive, or an optical medium such as a CD-ROM.
  • a non-transitory computer-readable medium is almost any medium suitable for the storage of data or an instruction set aside from a transitory waveform. Any such computer readable medium may reside on or within a single computational apparatus and may be present on or within different computational apparatuses within a system or network.
  • the term processing element or processor may be a central processing unit (CPU), or conceptualized as a CPU (such as a virtual machine).
  • the CPU or a device in which the CPU is incorporated may be coupled, connected, and/or in communication with one or more peripheral devices, such as display.
  • the processing element or processor may be incorporated into a mobile computing device, such as a smartphone or tablet computer.
  • the non-transitory computer-readable storage medium referred to herein may include a number of physical drive units, such as a redundant array of independent disks (RAID), a flash memory, a USB flash drive, an external hard disk drive, thumb drive, pen drive, key drive, a High-Density Digital Versatile Disc (HD-DV D) optical disc drive, an internal hard disk drive, a Blu-Ray optical disc drive, or a Holographic Digital Data Storage (HDDS) optical disc drive, synchronous dynamic random access memory (SDRAM), or similar devices or other forms of memories based on similar technologies.
  • RAID redundant array of independent disks
  • HD-DV D High-Density Digital Versatile Disc
  • HD-DV D High-Density Digital Versatile Disc
  • HDDS Holographic Digital Data Storage
  • SDRAM synchronous dynamic random access memory
  • Such computer-readable storage media allow the processing element or processor to access computer-executable process steps, application programs and the like, stored on removable and non-removable memory media, to off-load data from a device or to upload data to a device.
  • a non-transitory computer-readable medium may include almost any structure, technology, or method apart from a transitory waveform or similar medium.
  • These computer-executable program instructions may be loaded onto a general-purpose computer, a special purpose computer, a processor, or other programmable data processing apparatus to produce a specific example of a machine, such that the instructions that are executed by the computer, processor, or other programmable data processing apparatus create means for implementing one or more of the functions, operations, processes, or methods described herein.
  • These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement one or more of the functions, operations, processes, or methods described herein.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Educational Administration (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Economics (AREA)
  • Development Economics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Marketing (AREA)
  • Game Theory and Decision Science (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
US18/118,274 2022-03-09 2023-03-07 System and Methods for Monitoring Related Metrics Abandoned US20230289698A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/118,274 US20230289698A1 (en) 2022-03-09 2023-03-07 System and Methods for Monitoring Related Metrics

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202263318170P 2022-03-09 2022-03-09
US18/118,274 US20230289698A1 (en) 2022-03-09 2023-03-07 System and Methods for Monitoring Related Metrics

Publications (1)

Publication Number Publication Date
US20230289698A1 true US20230289698A1 (en) 2023-09-14

Family

ID=87931976

Family Applications (1)

Application Number Title Priority Date Filing Date
US18/118,274 Abandoned US20230289698A1 (en) 2022-03-09 2023-03-07 System and Methods for Monitoring Related Metrics

Country Status (6)

Country Link
US (1) US20230289698A1 (https=)
EP (1) EP4437702A4 (https=)
JP (1) JP2025512726A (https=)
CN (1) CN118511490A (https=)
CA (1) CA3240924A1 (https=)
WO (1) WO2023172541A1 (https=)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230281369A1 (en) * 2022-03-07 2023-09-07 Jpmorgan Chase Bank, N.A. Systems and methods for identifying and remediating architecture design defects
CN117278986A (zh) * 2023-11-23 2023-12-22 浙江小遛信息科技有限公司 共享出行的数据处理方法和数据处理设备
CN117634952A (zh) * 2023-11-24 2024-03-01 华中科技大学 一种产业发展评估方法、系统及电子设备
US20240256612A1 (en) * 2023-01-31 2024-08-01 Dell Products L.P. Insight gap recommendations
US12124436B1 (en) 2022-09-09 2024-10-22 Sigma Computing, Inc. Augmenting decision making via interactive what-if analysis
US20240386308A1 (en) * 2023-05-15 2024-11-21 Dell Products L.P. Dynamic action classification using machine learning techniques
US20240419705A1 (en) * 2023-06-13 2024-12-19 Microsoft Technology Licensing, Llc Data intelligence model for operator data queries
US12182109B1 (en) 2022-09-09 2024-12-31 Sigma Computing, Inc. Augmenting decision making via interactive what-if analysis
US20250245620A1 (en) * 2024-01-31 2025-07-31 Walmart Apollo, Llc Systems and methods for inventory placement and demand allocation
US12380096B1 (en) * 2024-03-13 2025-08-05 Microsoft Technology Licensing, Llc Natural language API for security graph exploration
US20250315768A1 (en) * 2024-04-04 2025-10-09 Verint Americas Inc. Systems and methods for key performance index prediction and improvement through feature analysis
US12505385B1 (en) * 2024-08-22 2025-12-23 MineSmart Technologies, LLC Generating visual frameworks to examine results of algorithmic decisions
US20260030265A1 (en) * 2024-07-24 2026-01-29 Capital One Services, Llc Data processing system with linkage analysis
US20260079944A1 (en) * 2024-09-13 2026-03-19 Kioxia Corporation Database management system and method for executing query processing to database
US20260080348A1 (en) * 2024-09-17 2026-03-19 Royal Bank Of Canada System and method for content retrieval and evaluation

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250111064A1 (en) * 2023-09-29 2025-04-03 International Business Machines Corporation Selective filtering of data based on data rules
US12536215B2 (en) * 2024-05-24 2026-01-27 Adobe Inc. Automated generation of governing label recommendations

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10009363B2 (en) * 2016-06-09 2018-06-26 Adobe Systems Incorporated Selecting representative metrics datasets for efficient detection of anomalous data
US10616251B2 (en) * 2017-02-23 2020-04-07 Cisco Technology, Inc. Anomaly selection using distance metric-based diversity and relevance
US10038611B1 (en) 2018-02-08 2018-07-31 Extrahop Networks, Inc. Personalization of alerts based on network monitoring
US20200036803A1 (en) * 2018-07-24 2020-01-30 Star2Star Communications, LLC Social Metrics Connection Representor, System, and Method
US11429627B2 (en) 2018-09-28 2022-08-30 Splunk Inc. System monitoring driven by automatically determined operational parameters of dependency graph model with user interface
US11354587B2 (en) 2019-02-01 2022-06-07 System Inc. Systems and methods for organizing and finding data

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230281369A1 (en) * 2022-03-07 2023-09-07 Jpmorgan Chase Bank, N.A. Systems and methods for identifying and remediating architecture design defects
US12314646B2 (en) * 2022-03-07 2025-05-27 Jpmorgan Chase Bank, N.A. Systems and methods for identifying and remediating architecture design defects
US12124436B1 (en) 2022-09-09 2024-10-22 Sigma Computing, Inc. Augmenting decision making via interactive what-if analysis
US12182109B1 (en) 2022-09-09 2024-12-31 Sigma Computing, Inc. Augmenting decision making via interactive what-if analysis
US20240256612A1 (en) * 2023-01-31 2024-08-01 Dell Products L.P. Insight gap recommendations
US20240386308A1 (en) * 2023-05-15 2024-11-21 Dell Products L.P. Dynamic action classification using machine learning techniques
US12436982B2 (en) * 2023-06-13 2025-10-07 Microsoft Technology Licensing, Llc Data intelligence model for operator data queries
US20240419705A1 (en) * 2023-06-13 2024-12-19 Microsoft Technology Licensing, Llc Data intelligence model for operator data queries
CN117278986A (zh) * 2023-11-23 2023-12-22 浙江小遛信息科技有限公司 共享出行的数据处理方法和数据处理设备
CN117634952A (zh) * 2023-11-24 2024-03-01 华中科技大学 一种产业发展评估方法、系统及电子设备
US20250245620A1 (en) * 2024-01-31 2025-07-31 Walmart Apollo, Llc Systems and methods for inventory placement and demand allocation
US12380096B1 (en) * 2024-03-13 2025-08-05 Microsoft Technology Licensing, Llc Natural language API for security graph exploration
US20250315768A1 (en) * 2024-04-04 2025-10-09 Verint Americas Inc. Systems and methods for key performance index prediction and improvement through feature analysis
US20260030265A1 (en) * 2024-07-24 2026-01-29 Capital One Services, Llc Data processing system with linkage analysis
US12505385B1 (en) * 2024-08-22 2025-12-23 MineSmart Technologies, LLC Generating visual frameworks to examine results of algorithmic decisions
US20260079944A1 (en) * 2024-09-13 2026-03-19 Kioxia Corporation Database management system and method for executing query processing to database
US20260080348A1 (en) * 2024-09-17 2026-03-19 Royal Bank Of Canada System and method for content retrieval and evaluation

Also Published As

Publication number Publication date
CA3240924A1 (en) 2023-09-14
EP4437702A4 (en) 2025-10-08
JP2025512726A (ja) 2025-04-22
EP4437702A1 (en) 2024-10-02
CN118511490A (zh) 2024-08-16
WO2023172541A1 (en) 2023-09-14

Similar Documents

Publication Publication Date Title
US20230289698A1 (en) System and Methods for Monitoring Related Metrics
US20230046324A1 (en) Systems and Methods for Organizing and Finding Data
US12271834B2 (en) Systems and methods for organizing, finding, and using data
Corrales et al. A case-based reasoning system for recommendation of data cleaning algorithms in classification and regression tasks
Wen‐Shung Tai et al. Effective e‐learning recommendation system based on self‐organizing maps and association mining
US9304991B2 (en) Method and apparatus for using monitoring intent to match business processes or monitoring templates
US12536440B2 (en) Developer activity modeler engine for a platform signal modeler
US12340333B2 (en) Interactive tree representing attribute quality or consumption metrics for data ingestion and other applications
US20230306033A1 (en) Dashboard for monitoring current and historical consumption and quality metrics for attributes and records of a dataset
Tran In-depth analysis and evaluation of ETL solutions for big data processing
US20240362194A1 (en) System and method for enriching and normalizing data
US20110191143A1 (en) Method and Apparatus for Specifying Monitoring Intent of a Business Process or Monitoring Template
Khan Business Intelligence and Data Analysis in the Age of AI
Kesavan Big Data Analytics: Tools, Technologies, and Real-World Applications–A Review
Ben Sassi et al. Data science with semantic technologies: application to information systems development
US11151653B1 (en) Method and system for managing data
CN117312774A (zh) 一种大数据的智能聚合可视化与管控系统
Leogrande Unlocking Hidden Value: A Framework for Transforming Dark Data in Organizational Decision-Making
HK40111754A (zh) 监控相关指标的系统和方法
Awasthi et al. Strategic integration of Big Data analytics for enhancing small to medium enterprises in Zimbabwe: A conceptual framework
US11561982B2 (en) Intelligent and automatic exception handling
US20240361890A1 (en) Interactive patent visualization systems and methods
Duma Recognizing the value of data in business operations: Data analytics for business operation
Ayyavaraiah Data Mining For Business Intelligence
HK40061310A (en) Systems and methods for organizing and finding data

Legal Events

Date Code Title Description
STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: SYSTEM INC., NEW YORK

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BLY, ADAM;KANG, DAVID;REEL/FRAME:063715/0642

Effective date: 20230519

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STCC Information on status: application revival

Free format text: WITHDRAWN ABANDONMENT, AWAITING EXAMINER ACTION