Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/906—Clustering; Classification
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning

Definitions

• the present disclosure relates generally to improvement of the functionality of a computerized automatic recommendation engine.
• the present disclosure and the embodiments contained therein relate to a method for improving the resolution of queries tickets through the utilization of historical solution data obtained from multiple sources, by providing a methodology for automatically identifying uncertain classifications and recommendations such that they can be flagged for subsequent human review, thereby improving the training process.
• Machine learning-based recommendation engines or recommendation systems offer a dramatic improvement over the old ways of doing things. By being able to selectively iterate through massive amount of data while performing a variety of similarity calculations, these systems are able to quickly and accurately provide known answers or potential answers contained within an attached database to an end-user.
• An aspect of an example embodiment in the present disclosure is to provide a method for identifying inaccurate classifications made by a computerized recommendation engine.
• the method begins by providing a plurality of classes in electronic format, where the classes are contained within the recommendation engine, as well as providing a set of data points, wherein each data point is transformed into a multi-dimensional vector and wherein each data point contains a solution. From there, a query in electronic format is provided, where the query is to be processed by the computerized recommendation engine.
• the computerized recommendation engine is then utilized to provide a subset of data points, selected from the set of datapoints, where the subset of data points is created by performing a similarity calculation on each data point in the set of data points and selecting data points with the highest similarity above a predetermined threshold.
• An average similarity of the subset of data points compared to the query is then computed, along with a range of entropy in the subset of data points.
• a composite score of all data points within the subset of data points is subsequently computed by taking the division product of the range of entropy over the average similarity.
• the range of entropy is from 0 to 1, inclusive.
• a maximum threshold and a minimum threshold are determined.
• a confidence value based on the composite score of the data points within the subset of data points, where the confidence value indicates whether the classification is inaccurate is calculated.
• a given classification is identified as being inaccurate if the confidence value is between the minimum threshold and the maximum threshold.
• the maximum threshold is determined by retrieving a confidence value for each data point in the set of data points, then separating the data points into a first sequence and a second sequence, separating the data points into a first sequence and a second sequence, wherein the first sequence contains data points that have been marked by a human user as being a known accurate answer, wherein the second sequence contains data points that have been marked by the human user as a known inaccurate answer;
• a first mean and first standard deviation for the first sequence are computed, as well as a second mean and a second standard deviation for the second sequence. From there an optimal value of a hyperparameter is determined, and the maximum threshold is determined as the second mean subtracted from the product of the optimal value of the hyperparameter and the second standard deviation.
• the minimum threshold is determined by taking the first mean subtracted from the product of the optimal value of the hyperparameter and the first standard deviation.
• the optimal value of the hyperparameter is less than the second mean minus the first mean, divided by the sum of the first standard deviation and the second standard deviation.
• Implementations may include one or a combination of any two or more of the aforementioned features. [0013]
• FIG. 1 a flow chart showing an embodiment of the method in accordance with the present disclosure.
• FIG. 2 is a flow chart showing an embodiment of a method for determining a maximum threshold in accordance with the present disclosure.
• the present disclosure provides for a method for automatically identifying uncertain recommendations and classifications thereof made by a computerized recommendation engine. While the methodology contained herein may be used with a variety of artificial intelligence, machine learning, and recommendation engine systems.
• said systems include a resolver containing a first user interface configured for use by end-users and a recommendation engine. Further, these systems also feature a monitoring hub containing a second interface configured for service management personnel, as well as a service management interface layer configured to import and normalize information provided from the end-users into the system, a transport layer for moving information between various parts of the system, and a system health layer which is configured to monitor the overall health of the system and to detect outages.
• the first user interface allows an end-user to submit their service requests into the system.
• this will be done in the form of a service ticket.
• this service ticket may contain just a single field of text, it preferably contains at least three fields: a title field; a description field; and a resolution field.
• the service request ticket contains information as to the owner, subsystem, date of creation, date of closure, ticket type, historical information, and the like.
• These systems are also capable of combining the information in the service ticket with other associated information about the end-user. This is achieved through a data adaptation layer parsing this other associated information into the recommendation engine.
• the service ticket is also sent to the recommendation engine through one or more machine- learning oriented REST APIs.
• the recommendation engine performs a large majority of the method in accordance with the present disclosure, as described below. Once the recommendation engine has made a recommendation as to the most-similar solution ticket present, these systems provide that solution ticket to the end-user. In embodiments where a BOT is identified as the most-similar solution ticket, the BOT will be delivered to the end-user as an executable file.
• the monitoring hub of these systems enables them to be scaled to process multiple problem tickets from multiple sources simultaneously.
• the monitoring hub is capable of monitoring various stats of the system, such as CPU utilization, memory utilization, the amount of open ports, number of events being processed, number of alters, and disk utilization. These stats are then utilized by the system health layer to determine the health of the system, predict when the system is about to fail, and in some embodiments, diagnose the cause of the potential failure.
• These systems provide for the ability to automatically and algorithmically provide solutions to service management tickets.
• these systems begin by assembling one or more issues in the form of service tickets.
• these tickets are provided in JavaScript Object Notation format, but these tickets can also be in html, csv, tsv, excel, database tables, and similar formats.
• These tickets are assembled by the data adaptation layer of these system and are pushed into an aggregator for aggregation. The aggregated tickets are then sent to the transport layer such that the aggregated tickets may be normalized and indexed for use by the recommendation engine.
• the service tickets preferably have multiple dimensions, and those dimensions likely have varying importance for use in determining the most relevant solution to the ticket.
• This normalization and indexing is done on a per-dimension basis.
• This normalization can consist of processing the portion of text with a language parser, replacing each word in the portion of text with all-lowercase versions of each word, removing stop words from the portion of text, and reducing each word on the portion of text to its word stem.
• the normalization may also utilize word vectors, paragraph vectors, tfidf vectors, hot-encoded vectors. Note that because these systems utilize abstracted vectors, there are virtually no limits as to what normalization techniques may be employed.
• a vector is generated for the field which is subsequently inverted.
• This inversion may utilize tf-idf, tf, idf, length-of-document, or word vectors, however, word vectors, paragraph vectors, tfidf vectors, and hot-encoded vectors can also be utilized. Note that because the system utilizes abstracted vectors, there are virtually no limits as to what inversion techniques may be employed.
• the above normalization, indexing, and inversion is performed for both the actual ticket or tickets, as well as on a test set of a related issue. These results are then merged into a database which is used by the recommendation engine.
• the recommendation engine performs a similarity calculation between each field of the problem ticket(s) and fields of one or more solution tickets. Sometimes these fields are given equal weight, however these fields can also be given unequal weights, where the fields with higher weights are more relevant to the similarity calculation.
• the dynamic weighting of each field will differ based on the type of problem ticket. This type of dynamic weighting is achieved by assigning weights to specifics present in a certain type of ticket.
• this type of dynamic weighting can be achieved by correlating the weights to user click-throughs. That is, how users interact with solutions tickets provided to them would be logged, and through the use of a support vector machine, the system could begin to make determinations as to how to weight various fields based on the type of problem ticket submitted. Preferably, this is done with an eye towards minimizing the normalized discounted cumulative gain.
• the similarity calculation can consist of, for each field present in the problem ticket(s) first initiating the top ‘K' matches of the solution tickets. For example, the “title” field of the problem ticket, would be compared against the title of each solution ticket, and the top ‘fC matches would be provided, where ‘K' is a user-configurable variable.
• the top matches are preferably determined through calculating the pair’s cosine similarity, but other methodologies such as Okapi BM25, normalized taxicab geometry, and Minkowski distances can be used.
• a list of the matches is compiled. This list may contain the actual tickets themselves or merely some sort of reference ID of each ticket.
• each of the tickets in the list is joined into a big space having a number of subspaces based on the fields present in the listed tickets.
• These subspaces may be weighted equally or may be weighted dynamically.
• machine learning will be utilized to dynamically assign the weights of the fields for certain types of queries and problems.
• the weighting of these fields can be used to address tickets with incomplete information (i.e., tickets with blank fields). Note that the accuracy of this matching determination can be improved if the top ( K * a) matches are provided, where ‘a’ is an integer multiple of ‘K.’
• the method in accordance with the present disclosure begins by providing a plurality of classes in electronic format and a set of data points where each data point contained an answer, potential answer, or a solution and is transformed into a multi-dimension vector.
• the plurality of classes can be expressed by and the set of data points can be expressed by is a data point in d-dimensional space.
• a query, in electronic format is then provided to the recommendation engine where the query has at least one field.
• the recommendation engine will then provide a subset of data points selected from the set of data points by performing a similarity calculation on each data point in the set of data points against the query and selecting the data points with their similarity calculation being above a predetermined threshold.
• the similarity calculation is performed by the recommendation engine generating a plurality of inverted indices for the at least one field, and then extending the dimensions of the query are extended. A first similarity measure is then calculated for each of the plurality of inverted indices against each of the data points. The recommendation then joins each of the data points when their respective first similarity measure is above a threshold amount. The data points are then each extended to match the dimensions of the extended dimensions of the query. The recommendation engine then performs a second similarity measure of extended dimensions of the query with each field of the joined data points.
• the recommendation engine then provides the top x results of the data points based on a composite of the first similarity measure and the second similarity measure, sampled from the top (x*a) results of the plurality of potential answers, where a is an integer multiple of x.
• the recommendation engine will report the probabilities of the classes in the set of classes C as a probability vector such that, After the subset of data points has been generated, an average similarity of the subset is computed using the following equation:
• a range of entropy for the subset of data points is then defined.
• the range of entropy will be defined as .
• a composite score of all the data points within the subset is then calculated by dividing the range of entropy E range by the average similarity (s).
• a maximum threshold and minimum threshold must be determined.
• these thresholds are determined automatically from the historical data contained in the database of the recommendation engine.
• a hyperparameter, k must be determined.
• the data points from the set of data points must have a label retrieved, where the label indicates whether the data point provided an accurate solution or an inaccurate solution.
• the data points are then split into two groups, or sequences, one group containing all of the data points that have been marked by a human user as being a known accurate answer, the second group containing all of the data points that have been marked by a human user as being a known inaccurate answer.
• a first mean p good , and a first standard deviation ⁇ good are prepared from taking the mean and standard deviation of the first group, and a second mean ⁇ bad and a second standard deviation a bad .
• the recommendation engine tests whether the following inequality is true: If this is true then If the inequality is not true, then subset of data points must be expanded until the inequality becomes true. If the subset of data points cannot be expanded to make the inequality true, the method fails, and an error message is returned to the end user.
• a confidence value is assigned to the subset. If the composite score is in between the minimum threshold and the maximum threshold, you know the answer, potential answer, or classification provided by the recommendation engine cannot be trusted and must be forwarded to a human user for review and manual marking.
• These computer-executable program instructions may be loaded onto a general-purpose computer, a special-purpose computer, a processor, or other programmable data processing apparatus to produce a particular machine, such that the instructions that execute on the computer, processor, or other programmable data processing apparatus create means for implementing one or more functions specified in the flow diagram block or blocks.
• These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement one or more functions specified in the flow diagram block or blocks.
• embodiments of the invention may provide for a computer program product, comprising a computer-usable medium having a computer-readable program code or program instructions embodied therein, said computer-readable program code adapted to be executed to implement one or more functions specified in the flow diagram block or blocks.
• the computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational elements or steps to be performed on the computer or other programmable apparatus to produce a computer- implemented process such that the instructions that execute on the computer or other programmable apparatus provide elements or steps for implementing the functions specified in the flow diagram block or blocks.
• blocks of the block diagrams and flow diagrams support combinations of means for performing the specified functions, combinations of elements or steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, can be implemented by special-purpose, hardware-based computer systems that perform the specified functions, elements or steps, or combinations of special purpose hardware and computer instructions.
• the present disclosure that relates generally to improvement of the functionality of a computerized automatic recommendation engine.
• the present disclosure and the embodiments contained therein relate to a method for improving the resolution of queries tickets through the utilization of historical solution data obtained from multiple sources, by providing a methodology for automatically identifying uncertain classifications and recommendations such that they can be flagged for subsequent human review, thereby improving the training process.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Databases & Information Systems (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Evolutionary Computation (AREA)
• Medical Informatics (AREA)
• Computing Systems (AREA)
• Mathematical Physics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=78525380

Family Applications (1)

Country Status (2)

Family Cites Families (4)

• 2021
  • 2021-04-22 EP EP21803781.0A patent/EP4150474A4/de not_active Withdrawn
  • 2021-04-22 WO PCT/IB2021/053337 patent/WO2021229330A1/en not_active Ceased

Also Published As

Similar Documents

Legal Events