WO2013034448A1 - Method and system for optimizing and streamlining troubleshooting - Google Patents

Method and system for optimizing and streamlining troubleshooting Download PDF

Info

Publication number
WO2013034448A1
WO2013034448A1 PCT/EP2012/066423 EP2012066423W WO2013034448A1 WO 2013034448 A1 WO2013034448 A1 WO 2013034448A1 EP 2012066423 W EP2012066423 W EP 2012066423W WO 2013034448 A1 WO2013034448 A1 WO 2013034448A1
Authority
WO
WIPO (PCT)
Prior art keywords
tickets
management system
ticket management
predictive models
trouble
Prior art date
Application number
PCT/EP2012/066423
Other languages
French (fr)
Inventor
Carolina VAZQUEZ GARCIA
Pedro GARCIA PARRA
Francisco Javier MOLINERO VELASCO
Anabel MONTERO CARRALAFUENTE
Original Assignee
Telefonica, S.A.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonica, S.A. filed Critical Telefonica, S.A.
Publication of WO2013034448A1 publication Critical patent/WO2013034448A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/06Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q50/60

Definitions

  • the present invention relates to a method for optimizing and streamlining troubleshooting, wherein said trouble is indicated in tickets generated by a ticket management system, and more specifically to a method which comprises, by means of using a module parallel to said ticket management system, collecting said tickets, solving the type of trouble in real time by means of predictive models built at least from historical data stored in said ticket management system and performing at least one of the following actions: launching an action suggestion, notifying the ticket management system of the action to be performed and/or automatically launching a task depending on a prediction obtained by means of said predictive models.
  • a second aspect of the invention relates to a system provided for implementing the method of the first aspect.
  • Trouble ticketing systems are used to improve efficiency of the management of these processes, and they are basically responsible for managing tickets from the time they are created until they can be solved and closed. Tickets are generally created manually by the operator or automatically by means of automatic alarm detection systems. Once the ticket is created, the technician is responsible for analyzing and diagnosing the problem, making a decision concerning the action or actions to be taken to solve it or forwarding it to another specialized group if he/she is not qualified to do it.
  • the operator can perform actions such as forwarding the ticket to another area of operation, establishing ticket processing priorities, deciding whether or not to send a technician to the site of the malfunction to solve the problem in-situ, etc.
  • Machine Learning and Artificial Intelligence techniques have begun to be implemented in recent years in a number of fields. Examples include device malfunction diagnostics, medical diagnostics, market analysis, voice and image recognition, industrial process parameter optimization, advertising applications, fraud detection, IT security, electric power demand prediction [1] [4] .
  • Machine Learning techniques There are supervised learning algorithms within Machine Learning techniques which have desired inputs and outputs and the models are responsible for establishing correspondence between both.
  • a clear example of such techniques are classification problems, where the system tries to label each of the samples or examples it receives in one of the values included among the possible categories or classes. The knowledge acquired by these systems is based on previously labeled and validated examples.
  • AUC area under the ROC curve
  • the algorithms which are more widely used for classification problem solving are based on decision trees, algorithms based on Bayes' theorem, neural networks, SVM, rule- based algorithms (JRip) , as well as the Bagging method [12] and Boosting method [13] to attain more robust and stable models.
  • the present invention proposes a method for optimizing and streamlining troubleshooting, wherein said trouble is indicated in tickets generated by a ticket management system.
  • the method of the invention characteristically comprises, by means of using a module parallel to said ticket management system:
  • the present invention proposes a system for optimizing and streamlining troubleshooting provided for implementing the method of the first aspect.
  • FIG. 1 illustrates the context in which the present invention is encompassed.
  • Figure 2 shows the main sub-modules of the platform proposed in the present invention.
  • Figure 3 shows a detailed diagram of the blocks integrating each sub-module of the platform proposed in the present invention.
  • Figure 4 shows the sub-modules and blocks supported by the execution platform, those of the database and those of the web server according to a possible implementation of the architecture of the platform proposed in the present invention.
  • Figure 5 shows the sub-modules and blocks involved in generating the predictive models in the platform of the present invention .
  • Figure 6 shows the information flow that is exchanged between the sub-modules and blocks involved in generating the predictive models in the platform of the present invention.
  • Figure 7 shows the sub-modules and blocks involved in classifying the ticket received by the platform of the present invention .
  • Figure 8 shows the information flow exchanged between the sub-modules and blocks involved in classifying the ticket received by the platform of the present invention.
  • Figure 9 shows the sub-modules and blocks involved in retrieving feedback both from skilled technicians and users of the platform as well as that derived from the actions performed on the tickets to try to know the actual value of response to the problem to be predicted in the platform of the present invention .
  • Figure 10 shows the information flow exchanged between the sub-modules and blocks involved in retrieving feedback both from technicians as well as that derived from the actions performed on the tickets to try to know the actual value of response to the problem to be predicted in the platform of the present invention .
  • the present invention relates to a method and system that works in real time for optimizing and streamlining ticket resolution of any type in principle, but specific to the area of telecommunications operator maintenance operations. To that end, it is based on action suggestions that can be integrated in the TT system or the automation of specific tasks, being supported by supervised learning techniques.
  • the TT system can be easily integrated into the workflow of the TT system since it can collect data at the time the system receives it by means of several types of interface. It processes the information and calculates the predictions in a server outside the system generating the data. Finally, depending on the configuration, it can send a notification with the suggested action or automatically launch a specific task according to the value of the prediction.
  • the method starts from the historical data stored in TT systems, being applicable to any problem associated with tickets that can be solved by means of classification techniques.
  • Specific cases of application of the proposed mechanism include integrating this system with a complaint management system of a telecommunications operator to try to anticipate the repetition of a customer malfunction.
  • the invention tries to predict if a specific complaint is not going to be solved satisfactorily, causing within a determined time period the reoccurrence of the malfunction.
  • a similar case would be the detection of those installations of a service that are going to cause a malfunction in a relatively short time period after concluding the installation.
  • the system When applied to another TT system responsible for managing trouble of elements of the installed plant, at the time of creating the ticket, the system is capable of deciding whether a technician should be sent to solve the problem or if the problem can be solved remotely, preventing the unnecessary travel of technicians. It is also possible to determine whether the ticket really needs attention or if it could be closed automatically because it was actually not a problem.
  • the platform is capable of interacting with the TT system in real time and becoming enriched with the information of the ticket throughout its life cycle in the system.
  • the ticket is received by both the TT system and by the platform, as shown in Figure 1.
  • TT system is a platform outside the TT system made up of a set of modules collecting information of the TT systems, preparing it for applying the predictive models to them, launching the corresponding prediction about the problems to be dealt with and triggering a preconfigured action (launching a notification, performing a specific operation on the ticket,%)
  • the platform is made up of a series of modules that communicate with one another. These modules automate data collection from the source system, prepare said data and launch a prediction with a determined probability of the problem in question.
  • This prediction can be in the form of a notification, an action suggestion or the launch of an automatism of a specific task subject to prior configuration.
  • Another group of modules are responsible for checking whether or not the predictive models are still valid. If they are not, they automatically rebuild them and render them available so that the administrator managing the platform can perform the relevant tests and include them in the tool. It is possible for the tool to automatically incorporate them into the classifying modules responsible for launching predictions based on established allowed error rate thresholds.
  • the Knowledge Generator collects data of the TT system for generating the necessary training sets and test sets and builds the predictive models with them. It further uses the validations or corrections of cases retrieved from the Feedback Collector for correcting or reinforcing the previously generated models.
  • the Online Engine works in real time receiving the tickets while they are created in the TT system, it prepares them and applies the predictive models to them to obtain the result of the prediction. This result could be translated into an action suggestion through a web console, a notification about the TT system or a preconfigured action on a specific ticket in the TT system. The last two cases need the TT system to support such operations through some type of standardized interface.
  • the Feedback Collector is a module that retrieves information about the final decision made for each case either because the technicians enters it directly in the platform or because the platform itself is responsible for finding the action which the technician ultimately performed. This information will be used for correcting or reinforcing the models .
  • Figure 3 shows the diagram of the sub-modules within the main modules:
  • Data Collector Module that retrieves information from the TT System in order to start working with it in the platform.
  • the mechanism for extracting information from the source system can be by means of probing the database, web services, file interface, according to the system requirements. It is possible to configure this information retrieval to work in real time as it is used in the Online Engine or to retrieve data to build models in the Knowledge Generator.
  • Preprocessing Data Module that performs the operations necessary to prepare the algorithm input instances. It is used both in the Online Engine for generating the inputs necessary for the predictive models, and in the Knowledge Generator for preparing the training sets and test sets for building the models .
  • Predictive Engine Classifying module embedded in the predictive model, it receives data from the preprocessed tickets, applies the model to them and obtains the result of the classification for the proposed problem.
  • Effector Module responsible for performing the actions configured in the platform on the TT system, ranging from sending a notification with an action suggestion to a direct action on the system, automatically performing an operation on the ticket.
  • Models Builder Predictive model generator which, starting from a series of training tickets, that can be configured by the administrator, automatically builds and tests a pre-established list of both classification and attribute selection algorithms. The best option is automatically established based on comparing the ROC curves of the different models generated. Both the models and the results of their statistics are stored in the Models Repository. The administrator can see these results by means of an output display tool and perform configurable tests with different sets of data or pre-configure the automatic insertion of the model in the Predictive Engine.
  • Models Checker Module that automatically checks the goodness of fit of the models using the statistics thereof. When the model starts to produce results worse than those established in the threshold that can be configured by the administrator, an alarm is launched.
  • Models Repository Repository storing the predictive models that have been built by means of the Models Builder. It also saves information about the statistics of the models and the comparisons calculated by means of the Models Checker.
  • User Feedback Collector Module whereby the technician enters feedback into the platform, validating or correcting the value of the predictions, aiding in reinforcing or correcting the decisions to achieve an improvement in the models.
  • System Feedback Collector Module whereby feedback from the TT System itself is retrieved, revealing the decisions finally made on the tickets, comparing them with the prediction of the platform and using them for reinforcing or correcting the knowledge stored in the models.
  • This module selects the feedback to be sent to the Models Builder or to the Models Checker depending on what has previously been configured (if both feedback retrieval methods are active, which of the two has a higher weight in the event of an inconsistency can be decided by means of configuration) .
  • a possible implementation software of the system could be as follows, though it is not subject to any restriction.
  • Execution Platform Container where the different processes are run with a communication mechanism between them.
  • Database It allows the persistence of information necessary for retrieval in the event of possible errors, as well as of the generic data model storing the information about the configuration and administration of the different modules. It stores the results of the predictions, the statistics of the models and the remaining information necessary for display. It also maintains the models repository if the retrieval of any of them at any time is necessary.
  • Web Server The tools that can be accessed by the technician for entering inputs into the system and displaying outputs are deployed on the web server.
  • Sub-module interconnection and operation within the main modules are described below according to the operating mode of the system. It must be taken into account that normal operation comprises the simultaneous running of all the main modules, but it is possible to active or deactivate those considered appropriate by means of configuration.
  • the Knowledge Generator is responsible for generating the predictive models making up the learning module of the system.
  • Figure 5 shows the modules involved and the steps detailing this functionality.
  • Figure 6 shows a block diagram with information flow :
  • the Data Collector module is connected to the TT System for retrieving the data with which the training set and test set subsequently used to generate the predictive models will be generated .
  • the Feedback Collector If the Feedback Collector is activated and has feedback, it corrects the tickets retrieved from the TT System for improving the data for generating the models.
  • the instances are passed along to the Preprocessing Data module for preparing them and generating the training set and the test sets.
  • the Preprocessing Data passes the training set and the test set along to the Models Builder, which tests different combinations of attributes selected by means of InfoGain, GreedyStepwise or BestFirst methods, on different types of algorithms (Naive Bayes, Bayes Net, C.45, AdaBoost, Bagging, JRip, SVM) .
  • the parameters of these algorithms are heuristically optimized until finding the one that provides the best behavior by means of comparing their statistics (Area ROC, accuracy and error percentage) .
  • the Models Builder sends the generated models, their statistics and the comparisons to the Models Repository.
  • the Models Checker performs tests on the chosen model and checks that the statistics comply with a predefined threshold which considers the model to be good.
  • the new model is sent from the Models Repository to the Predictive Engine.
  • the order to replace the module manually or automatically is allowed to be sent.
  • the Predictive Engine is manually or automatically restarted, according to the configuration, resuming operation with the new model.
  • the Data Collector retrieves the tickets from the TT system that are pending processing by the system. Depending on the system requirements, the retrieval will be implemented by means of different mechanisms (databases, web services or files) .
  • the Data Collector keeps the tickets in a temporary repository to deal with possible errors of communication with the system.
  • the Data Collector informs the Preprocessing Data of the following ticket to be processed and prepares it to pass it along to the Predictive Engine and adjust it to the format which the predictive model expects to receive.
  • the Predictive Engine passes the instance through the model and launches the prediction (classification with YES/NO type values in response to the proposed problem) .
  • the Effector can return an outgoing message or automatically perform an action on the TT System.
  • the output generated by the Predictive Engine is passed along to the Models Checker.
  • the manual or automatic feedback is collected and passed along to the Models Checker so that it compares it with the outputs of the Predictive Engine and checks the validity of the model .
  • the Area ROC of the model is calculated by comparing the predictions launched by the Predictive Engine with the Feedback retrieved manually or automatically
  • the Feedback Collector retrieves the feedback both from technicians and that derived from the actions performed on the tickets to know the actual value of response to the problem to be predicted. This information serves to provide feedback to both the Models Builder and the Models Checker.
  • Figure 9 shows the modules involved and Figure 10 shows a block diagram with the information flow:
  • the System Feedback Collector retrieves from the TT system the actions that were actually performed on each ticket.
  • the System Feedback Collector sends the retrieved feedback to the Feedback Selector.
  • the User Feedback Collector retrieves the validations or corrections of the predictions entered by the technician and sends them to the Feedback Selector.
  • the Feedback Selector sends the final feedback to the Models Checker so that it can validate the precision of the models .
  • the order is sent for the Models Builder to regenerate the model using the data received from the Preprocessing Data enriched with the data received from the Feedback Collector as the training and test data.
  • An expert user with platform administration permits is capable of connecting to the system ticket log and retrieving a given range of data for training use.
  • the model is generated with such data by means of the Models Builder module, which automatically builds the optimal model from among those proposed (Naive Bayes, BayesNet, Adaboost, Bagging, SVM, Comite, C.45, JRip) . It performs input attribute selection for those that work the best (InfoGain, GreedyStepwise, BestFirst) and a heuristic search for the parameters that optimize it.
  • the administrator is capable of selecting different sets of test by connecting to the TT system and validating the outputs. Once the statistics of the model exceed the allowed thresholds, it is capable of replacing it in the Predictive Engine which, by means of restarting the module, would start to work with the new model.
  • the new tickets retrieved by the Data Collector enter gradually. They are processed by the Preprocessing Data module and the Predictive Engine decides to generate the prediction for the particular problem based on the knowledge integrated in the predictive model .
  • this prediction is sent to the output for a technician to analyze it. If the technician thinks it is appropriate, he/she can enter feedback, reinforcing the decision if the decision was correct and correcting it if the decision was incorrect in order to use it in the future the next time the model is rebuilt.
  • Messages can further be sent directly to the TT System by means of a notification system suitable for the interface designed between both systems. It is also possible to launch an automatism simulating a specific action of a technician on the ticket .
  • the platform is capable of anticipating whether or not each of the malfunctions will be repeated in the future. This prediction is shown to the operators and a set of actions to be performed on the ticket, such as for example marking the customer in question for special tracking or establishing control calls to check if he/ she has any problem with the ticket resolution, is recommended.
  • a similar case would be that applied to detecting possible early malfunctions after installing a service. These malfunctions are those which occur very shortly after the installation and may be related with an incorrect installation of the service.
  • the platform connects to the TT system which collects the installation requests.
  • the platform receives the request and the prediction is launched in real time. According to whether or not it is decided that an early malfunction is going to occur, the possible actions to be performed can be similar to the previous case, marking customers for special tracking, marking the ticket for post-installation analytics or another type of operation defined by the administrator.
  • the platform connects to an TT system responsible for managing trouble in the installed plant of a telecommunications company.
  • the system proposed enables predicting the cases in which it is necessary for a technician to travel to the indicated site of the malfunction or if, in contrast, it is a problem that can be solved remotely.
  • the expenses associated with both cases are very different.
  • the Knowledge Generator allows automating process of searching for the optimal model for each specific problem. This module can be enlarged to contemplate a larger number of algorithms allowed. Models which are working in real time can be validated and their statistics stored from here, checking whether the deviation of the results is within the allowed range. If it is not within this range, the models are automatically rebuilt. Temporary deviations of the results from the predictions are corrected or reduced with this mechanism.
  • the Feedback Collector facilitates a predictive model correction or reinforcement mechanism based on the experience of expert technicians who validate or correct the predictions and on the experience of technicians who, without explicitly validating or correcting the results, have recorded their actions in the TT system.
  • the types of algorithms that are going to be launched for automatic model generation as well as the automatic attribute selection and the heuristic search for parameters optimizing the result of the algorithms can be established by means of configuration. It is not necessary to go through all the types of algorithms if those offering the best behavior are already known.
  • the thresholds of the statistics of the models which decide whether or not they are still valid as new tickets enter the platform over time can be established. These values can be checked and it can be decided whether or not to replace the old model with the one that was just generated. This model can be tested with test sets that can be configured by the administrator. If everything is okay, the order to replace the models can be sent, stopping the modules, replacing the model and restarting them again.
  • the action to be performed after launching the prediction, sending messages that are visible from the TT system, messages to the output, performing actions directly on the TT system can be configured by means of an interface for interacting as a user of the system that can perform actions .
  • this Predictive Analytics platform provides the operator with benefits in terms of reducing OPEX, since efficiency is gained with it, optimizing resource management. It also translates into increased customer satisfaction since it reduces ticket resolution time in addition to helping find the best resolution to such tickets.

Abstract

In the method of the invention, the trouble is indicated in tickets generated by a ticket management system, and it comprises, by means of using a module parallel to said ticket management system: - collecting said tickets; - solving the type of trouble in real time by means of predictive models built at least from historical data stored in said ticket management system; and - performing at least one of the following actions: launching an action suggestion, notifying the ticket management system of the action to be performed and/or automatically launching a task depending on a prediction obtained by means of said predictive models. The system is provided for implementing the method of the present invention.

Description

METHOD AND SYSTEM FOR OPTIMIZING AND STREAMLINING
TROUBLESHOOTING
Field of the Art
In a first aspect, the present invention relates to a method for optimizing and streamlining troubleshooting, wherein said trouble is indicated in tickets generated by a ticket management system, and more specifically to a method which comprises, by means of using a module parallel to said ticket management system, collecting said tickets, solving the type of trouble in real time by means of predictive models built at least from historical data stored in said ticket management system and performing at least one of the following actions: launching an action suggestion, notifying the ticket management system of the action to be performed and/or automatically launching a task depending on a prediction obtained by means of said predictive models.
A second aspect of the invention relates to a system provided for implementing the method of the first aspect.
Prior State of the Art
For some time, large telecommunication companies have sought to streamline and automate maintenance processes, to the greatest extent possible, by implementing systems which detect problems in their network in order to solve them as quickly as possible or which manage their resources more efficiently. Both cost reduction and increased customer satisfaction are therefore achieved.
Trouble ticketing systems are used to improve efficiency of the management of these processes, and they are basically responsible for managing tickets from the time they are created until they can be solved and closed. Tickets are generally created manually by the operator or automatically by means of automatic alarm detection systems. Once the ticket is created, the technician is responsible for analyzing and diagnosing the problem, making a decision concerning the action or actions to be taken to solve it or forwarding it to another specialized group if he/she is not qualified to do it.
Throughout the life cycle of the ticket, the operator can perform actions such as forwarding the ticket to another area of operation, establishing ticket processing priorities, deciding whether or not to send a technician to the site of the malfunction to solve the problem in-situ, etc.
Until now these jobs were being steered towards solutions based on proprietary systems and expert systems which needed to be supplied with the knowledge acquired by experienced technicians with a broad business perspective to make these systems intelligent and to take advantage of technicians' knowledge .
From the viewpoint of trouble ticketing (TT) systems, attempts are being made to automate ticket management, starting with the automatic creation of the tickets [11] and continuing with the automation of specific operations [7].
Machine Learning and Artificial Intelligence techniques have begun to be implemented in recent years in a number of fields. Examples include device malfunction diagnostics, medical diagnostics, market analysis, voice and image recognition, industrial process parameter optimization, advertising applications, fraud detection, IT security, electric power demand prediction [1] [4] .
More specific cases which are also more closely related to the area of telecommunications are the optimization of the calls of a Call Center [2], churn detection [3], network traffic prediction [4] and detection of hardware failures within a network, correlating the different alarms included [5].
There are supervised learning algorithms within Machine Learning techniques which have desired inputs and outputs and the models are responsible for establishing correspondence between both. A clear example of such techniques are classification problems, where the system tries to label each of the samples or examples it receives in one of the values included among the possible categories or classes. The knowledge acquired by these systems is based on previously labeled and validated examples.
Work is being done on implementations and methods encouraging the application of predictive analytics to solving the largest number of problems [9], in the automatic update of predictive models used in a production setting [10] as well as in the reinforcement of learning modules [8] . These are examples of the problems arising when integrating all these techniques into production settings.
One of the methods currently being used to decide on the optimal model is the comparison of certain statistics of these models. The study of the area under the ROC curve, also referred to as AUC, is based on the ratio between the sensitivity and specificity of the model, which represent the fraction of true positives of one experiment with respect to the fraction of true negatives of the same experiment, respectively.
The algorithms which are more widely used for classification problem solving are based on decision trees, algorithms based on Bayes' theorem, neural networks, SVM, rule- based algorithms (JRip) , as well as the Bagging method [12] and Boosting method [13] to attain more robust and stable models.
When examining another type of proprietary solutions, difficulties can be encountered when they are integrated in the workflow of the TT system in real time and receive information, without needing to deal with adaptations on TT systems, which are fairly expensive in most cases.
The development and inclusion of a similar solution for each new similar problem that needs to be solved by means of predictive analytics techniques is likewise found to be expensive and slow. There is no mechanism that streamlines the building and selection of optimal models for each problem and that allows incorporating them into the workflow of the TT system, minimizing adaptations thereof.
Despite there being previously designed algorithms that could be used on the data of TT systems, the application of these algorithms on certain data is not so simple. This data can vary considerably due to business changes which make it necessary to have a mechanism that allows checking the validity of the predictions over time. Model degradation effects readily appear in the area of application due to data variability and it should be possible to interact with the platform, automating the process as much as possible.
Since the solution is aimed at using the logs indicating the operations performed on tickets, it must be taken into account that not all the knowledge that is extracted is valid, since sometimes the models can even be forced to learn erroneous procedures for ticket resolution. For this reason the direct application of classification algorithms on the data of the TT system can lead to generating predictive models which, instead of streamlining and improving ticket management, slows it down due to the accumulation of errors. A mechanism is necessary in order to be capable of correcting those cases which, a priori, these algorithms do not provide, giving the technician the possibility to validate or correct the predictions of the models .
Summary of the Invention
It is necessary to find an alternative to the state of the art that allows making up for the detected deficiencies, particularly relating to the lack of proposals that can be integrated in the workflow of a TT system in real time and that receive information without needing to deal with adaptations as well as the lack of a mechanism that streamlines the building and selection of optimal models for each problem that presents.
To that end, in a first aspect the present invention proposes a method for optimizing and streamlining troubleshooting, wherein said trouble is indicated in tickets generated by a ticket management system.
Unlike the already known proposals, the method of the invention characteristically comprises, by means of using a module parallel to said ticket management system:
- collecting said tickets;
- solving the type of trouble in real time by means of predictive models built at least from historical data stored in said ticket management system; and
performing at least one of the following actions: launching an action suggestion, notifying said ticket management system of the action to be performed and/or automatically launching a task depending on a prediction obtained by means of said predictive models.
Other aspects of the method of the first aspect of the invention are described according to the attached claims 2 to 13.
In a second aspect the present invention proposes a system for optimizing and streamlining troubleshooting provided for implementing the method of the first aspect.
Other features of the system of the second aspect of the invention are described according to the attached claims 14 and 15.
Brief Description of the Drawings
The foregoing and other advantages and features will be better understood from the following detailed description of several embodiments in reference to the attached illustrative and non-limiting drawings in which:
Figure 1 illustrates the context in which the present invention is encompassed.
Figure 2 shows the main sub-modules of the platform proposed in the present invention.
Figure 3 shows a detailed diagram of the blocks integrating each sub-module of the platform proposed in the present invention.
Figure 4 shows the sub-modules and blocks supported by the execution platform, those of the database and those of the web server according to a possible implementation of the architecture of the platform proposed in the present invention.
Figure 5 shows the sub-modules and blocks involved in generating the predictive models in the platform of the present invention .
Figure 6 shows the information flow that is exchanged between the sub-modules and blocks involved in generating the predictive models in the platform of the present invention.
Figure 7 shows the sub-modules and blocks involved in classifying the ticket received by the platform of the present invention .
Figure 8 shows the information flow exchanged between the sub-modules and blocks involved in classifying the ticket received by the platform of the present invention.
Figure 9 shows the sub-modules and blocks involved in retrieving feedback both from skilled technicians and users of the platform as well as that derived from the actions performed on the tickets to try to know the actual value of response to the problem to be predicted in the platform of the present invention .
Figure 10 shows the information flow exchanged between the sub-modules and blocks involved in retrieving feedback both from technicians as well as that derived from the actions performed on the tickets to try to know the actual value of response to the problem to be predicted in the platform of the present invention .
Detailed Description of several Embodiments
The present invention relates to a method and system that works in real time for optimizing and streamlining ticket resolution of any type in principle, but specific to the area of telecommunications operator maintenance operations. To that end, it is based on action suggestions that can be integrated in the TT system or the automation of specific tasks, being supported by supervised learning techniques.
It is based on predictive models built from the historical data stored in the trouble ticketing system. These models can be rebuilt automatically in those cases in which the tool detects that the model is no longer reliable. It contains a functionality of obtaining feedback from both the technicians and from the trouble ticketing system itself to correct or reinforce the previously generated knowledge, whichever may be appropriate .
It can be easily integrated into the workflow of the TT system since it can collect data at the time the system receives it by means of several types of interface. It processes the information and calculates the predictions in a server outside the system generating the data. Finally, depending on the configuration, it can send a notification with the suggested action or automatically launch a specific task according to the value of the prediction.
It is a method or mechanism for optimizing and streamlining ticket resolution, recommending the best use of resources in terms of cost reduction and increased customer satisfaction in real time. Supervised learning techniques that can be integrated in already running TT systems are used.
The method starts from the historical data stored in TT systems, being applicable to any problem associated with tickets that can be solved by means of classification techniques.
Specific cases of application of the proposed mechanism include integrating this system with a complaint management system of a telecommunications operator to try to anticipate the repetition of a customer malfunction. In other words, the invention tries to predict if a specific complaint is not going to be solved satisfactorily, causing within a determined time period the reoccurrence of the malfunction. A similar case would be the detection of those installations of a service that are going to cause a malfunction in a relatively short time period after concluding the installation.
When applied to another TT system responsible for managing trouble of elements of the installed plant, at the time of creating the ticket, the system is capable of deciding whether a technician should be sent to solve the problem or if the problem can be solved remotely, preventing the unnecessary travel of technicians. It is also possible to determine whether the ticket really needs attention or if it could be closed automatically because it was actually not a problem.
The platform is capable of interacting with the TT system in real time and becoming enriched with the information of the ticket throughout its life cycle in the system. At the time of creating the ticket, the ticket is received by both the TT system and by the platform, as shown in Figure 1.
As discussed, it is a platform outside the TT system made up of a set of modules collecting information of the TT systems, preparing it for applying the predictive models to them, launching the corresponding prediction about the problems to be dealt with and triggering a preconfigured action (launching a notification, performing a specific operation on the ticket,...)
The platform is made up of a series of modules that communicate with one another. These modules automate data collection from the source system, prepare said data and launch a prediction with a determined probability of the problem in question. This prediction can be in the form of a notification, an action suggestion or the launch of an automatism of a specific task subject to prior configuration.
Another group of modules are responsible for checking whether or not the predictive models are still valid. If they are not, they automatically rebuild them and render them available so that the administrator managing the platform can perform the relevant tests and include them in the tool. It is possible for the tool to automatically incorporate them into the classifying modules responsible for launching predictions based on established allowed error rate thresholds.
There are three main modules: Knowledge Generator, Online Engine and Feedback Collector, as shown in Figure 2.
The Knowledge Generator collects data of the TT system for generating the necessary training sets and test sets and builds the predictive models with them. It further uses the validations or corrections of cases retrieved from the Feedback Collector for correcting or reinforcing the previously generated models.
The Online Engine works in real time receiving the tickets while they are created in the TT system, it prepares them and applies the predictive models to them to obtain the result of the prediction. This result could be translated into an action suggestion through a web console, a notification about the TT system or a preconfigured action on a specific ticket in the TT system. The last two cases need the TT system to support such operations through some type of standardized interface.
The Feedback Collector is a module that retrieves information about the final decision made for each case either because the technicians enters it directly in the platform or because the platform itself is responsible for finding the action which the technician ultimately performed. This information will be used for correcting or reinforcing the models .
There is a mechanism for inputting information into the system to provide it with the capacities of stopping, starting and configuring all the modules, as well as more specific functionalities for monitoring the behavior of the predictive models. Examples of actions that can be performed using capacities of configuring the modules are:
Determining the types of algorithms from among those contemplated by the platform that will be used for rebuilding and comparing several models then selecting the models with optimal algorithms.
- Defining a range of tickets to be used in the training set and test set for rebuilding models both manually and automatically .
- Establishing the manual or automatic nature of replacing the old predictive models with the recently generated predictive models .
- Configuring the actions to be performed according to the result of the prediction (displaying the result in the web console, action suggestion, notification, performing an automatism, ... )
Figure 3 shows the diagram of the sub-modules within the main modules:
Data Collector: Module that retrieves information from the TT System in order to start working with it in the platform. The mechanism for extracting information from the source system can be by means of probing the database, web services, file interface, according to the system requirements. It is possible to configure this information retrieval to work in real time as it is used in the Online Engine or to retrieve data to build models in the Knowledge Generator.
Preprocessing Data: Module that performs the operations necessary to prepare the algorithm input instances. It is used both in the Online Engine for generating the inputs necessary for the predictive models, and in the Knowledge Generator for preparing the training sets and test sets for building the models .
Predictive Engine: Classifying module embedded in the predictive model, it receives data from the preprocessed tickets, applies the model to them and obtains the result of the classification for the proposed problem.
Effector: Module responsible for performing the actions configured in the platform on the TT system, ranging from sending a notification with an action suggestion to a direct action on the system, automatically performing an operation on the ticket.
Models Builder: Predictive model generator which, starting from a series of training tickets, that can be configured by the administrator, automatically builds and tests a pre-established list of both classification and attribute selection algorithms. The best option is automatically established based on comparing the ROC curves of the different models generated. Both the models and the results of their statistics are stored in the Models Repository. The administrator can see these results by means of an output display tool and perform configurable tests with different sets of data or pre-configure the automatic insertion of the model in the Predictive Engine.
Models Checker: Module that automatically checks the goodness of fit of the models using the statistics thereof. When the model starts to produce results worse than those established in the threshold that can be configured by the administrator, an alarm is launched.
Models Repository: Repository storing the predictive models that have been built by means of the Models Builder. It also saves information about the statistics of the models and the comparisons calculated by means of the Models Checker.
User Feedback Collector: Module whereby the technician enters feedback into the platform, validating or correcting the value of the predictions, aiding in reinforcing or correcting the decisions to achieve an improvement in the models.
System Feedback Collector: Module whereby feedback from the TT System itself is retrieved, revealing the decisions finally made on the tickets, comparing them with the prediction of the platform and using them for reinforcing or correcting the knowledge stored in the models.
Feedback Selector: This module selects the feedback to be sent to the Models Builder or to the Models Checker depending on what has previously been configured (if both feedback retrieval methods are active, which of the two has a higher weight in the event of an inconsistency can be decided by means of configuration) .
• Architecture
A possible implementation software of the system could be as follows, though it is not subject to any restriction.
Execution Platform: Container where the different processes are run with a communication mechanism between them.
Database: It allows the persistence of information necessary for retrieval in the event of possible errors, as well as of the generic data model storing the information about the configuration and administration of the different modules. It stores the results of the predictions, the statistics of the models and the remaining information necessary for display. It also maintains the models repository if the retrieval of any of them at any time is necessary.
Web Server: The tools that can be accessed by the technician for entering inputs into the system and displaying outputs are deployed on the web server.
The sub-modules that could be supported in the Execution Platform, those that could use the Database layer and those that could be deployed on a Web Server are shown for each of the main modules in Figure 4.
• Module operation and interconnection
Sub-module interconnection and operation within the main modules are described below according to the operating mode of the system. It must be taken into account that normal operation comprises the simultaneous running of all the main modules, but it is possible to active or deactivate those considered appropriate by means of configuration.
Predictive model building operation
The Knowledge Generator is responsible for generating the predictive models making up the learning module of the system. Figure 5 shows the modules involved and the steps detailing this functionality. Figure 6 shows a block diagram with information flow :
1. The Data Collector module is connected to the TT System for retrieving the data with which the training set and test set subsequently used to generate the predictive models will be generated .
2. If the Feedback Collector is activated and has feedback, it corrects the tickets retrieved from the TT System for improving the data for generating the models.
3. The instances are passed along to the Preprocessing Data module for preparing them and generating the training set and the test sets.
4. The Preprocessing Data passes the training set and the test set along to the Models Builder, which tests different combinations of attributes selected by means of InfoGain, GreedyStepwise or BestFirst methods, on different types of algorithms (Naive Bayes, Bayes Net, C.45, AdaBoost, Bagging, JRip, SVM) . The parameters of these algorithms are heuristically optimized until finding the one that provides the best behavior by means of comparing their statistics (Area ROC, accuracy and error percentage) .
5. The Models Builder sends the generated models, their statistics and the comparisons to the Models Repository.
6. The Models Checker performs tests on the chosen model and checks that the statistics comply with a predefined threshold which considers the model to be good.
7. The new model is sent from the Models Repository to the Predictive Engine. The order to replace the module manually or automatically is allowed to be sent.
8. The Predictive Engine is manually or automatically restarted, according to the configuration, resuming operation with the new model.
- Classification operation in real time
In the Online Engine, it is important to start from the fact that the Knowledge Generator has previously been used for building the model optimizing the problem and has been embedded in the Predictive Engine. Figure 7 shows the modules involved and Figure 8 shows a block diagram with the information flow:
1. The Data Collector retrieves the tickets from the TT system that are pending processing by the system. Depending on the system requirements, the retrieval will be implemented by means of different mechanisms (databases, web services or files) . The Data Collector keeps the tickets in a temporary repository to deal with possible errors of communication with the system.
2. The Data Collector informs the Preprocessing Data of the following ticket to be processed and prepares it to pass it along to the Predictive Engine and adjust it to the format which the predictive model expects to receive.
3. The Predictive Engine passes the instance through the model and launches the prediction (classification with YES/NO type values in response to the proposed problem) .
4. According to its configuration, the Effector can return an outgoing message or automatically perform an action on the TT System.
5. The output generated by the Predictive Engine is passed along to the Models Checker.
6. The manual or automatic feedback is collected and passed along to the Models Checker so that it compares it with the outputs of the Predictive Engine and checks the validity of the model .
7. The Area ROC of the model is calculated by comparing the predictions launched by the Predictive Engine with the Feedback retrieved manually or automatically
8. If the Area ROC of the model is under the allowed value, the rebuilding of the models is triggered.
- Feedback retrieval operation
The Feedback Collector retrieves the feedback both from technicians and that derived from the actions performed on the tickets to know the actual value of response to the problem to be predicted. This information serves to provide feedback to both the Models Builder and the Models Checker.
Figure 9 shows the modules involved and Figure 10 shows a block diagram with the information flow:
1. It is possible to configure the feedback retrieval mode, being able to activate only the User Feedback Collector, only the System Feedback Collector or both, weighting the result by means of the Feedback Selector.
2. The System Feedback Collector retrieves from the TT system the actions that were actually performed on each ticket.
3. The System Feedback Collector sends the retrieved feedback to the Feedback Selector.
4. The User Feedback Collector retrieves the validations or corrections of the predictions entered by the technician and sends them to the Feedback Selector.
5. The Feedback Selector sends the final feedback to the Models Checker so that it can validate the precision of the models .
6. According to the results extracted from the Models Checker, the order is sent for the Models Builder to regenerate the model using the data received from the Preprocessing Data enriched with the data received from the Feedback Collector as the training and test data.
• Cases of use
The operation can be described as follows for any of the cases of use explained below. An expert user with platform administration permits is capable of connecting to the system ticket log and retrieving a given range of data for training use. The model is generated with such data by means of the Models Builder module, which automatically builds the optimal model from among those proposed (Naive Bayes, BayesNet, Adaboost, Bagging, SVM, Comite, C.45, JRip) . It performs input attribute selection for those that work the best (InfoGain, GreedyStepwise, BestFirst) and a heuristic search for the parameters that optimize it.
Once the model is built, the administrator is capable of selecting different sets of test by connecting to the TT system and validating the outputs. Once the statistics of the model exceed the allowed thresholds, it is capable of replacing it in the Predictive Engine which, by means of restarting the module, would start to work with the new model.
Once the Predictive Engine is ready, the new tickets retrieved by the Data Collector enter gradually. They are processed by the Preprocessing Data module and the Predictive Engine decides to generate the prediction for the particular problem based on the knowledge integrated in the predictive model .
According to what has been configured, this prediction is sent to the output for a technician to analyze it. If the technician thinks it is appropriate, he/she can enter feedback, reinforcing the decision if the decision was correct and correcting it if the decision was incorrect in order to use it in the future the next time the model is rebuilt.
Messages can further be sent directly to the TT System by means of a notification system suitable for the interface designed between both systems. It is also possible to launch an automatism simulating a specific action of a technician on the ticket .
Several cases of use of the platform in the area of problems specific to TT systems are described below.
- Detecting the reoccurrence of one and the same malfunction in a customer complaint TT system This is a case of being applied to a complaint TT system of a telecommunications operator. The problem to be solved is to determine whether a given complaint is going to occur again in the system after a given time period because it was not correctly solved in a first action of the technician.
The platform is capable of anticipating whether or not each of the malfunctions will be repeated in the future. This prediction is shown to the operators and a set of actions to be performed on the ticket, such as for example marking the customer in question for special tracking or establishing control calls to check if he/ she has any problem with the ticket resolution, is recommended.
Detecting the occurrence of early malfunctions after installing a service
A similar case would be that applied to detecting possible early malfunctions after installing a service. These malfunctions are those which occur very shortly after the installation and may be related with an incorrect installation of the service.
The platform connects to the TT system which collects the installation requests. The platform receives the request and the prediction is launched in real time. According to whether or not it is decided that an early malfunction is going to occur, the possible actions to be performed can be similar to the previous case, marking customers for special tracking, marking the ticket for post-installation analytics or another type of operation defined by the administrator.
Detecting the need to send a technician to solve a malfunction or possibility of solving it remotely.
In this case, the platform connects to an TT system responsible for managing trouble in the installed plant of a telecommunications company.
The system proposed enables predicting the cases in which it is necessary for a technician to travel to the indicated site of the malfunction or if, in contrast, it is a problem that can be solved remotely. The expenses associated with both cases are very different.
- Detecting productivity of the malfunction resolution
In this case, by using this system it can be predicted whether or not the work that should theoretically be done to solve the malfunction associated with a ticket is going to be productive, i.e. whether it is necessary to associate it with a worker (local or remote service) or whether it is best to directly close the ticket because no error is going to occur in the system.
• Advantages of the invention
Adapting to new scenarios is easier by means of the system. It is a very flexible mechanism for such problems, solving communication with the TT system for retrieving tickets by means of prepared file reader interfaces, probing a database or using the necessary connector.
Only minimal adaptations would have to be made in the Data Collector and Preprocessing Data modules. The Effector would further have to be adapted in cases where direct action on the TT system is configured. The rest are generic modules that do not depend on the architecture of the TT system.
The Knowledge Generator allows automating process of searching for the optimal model for each specific problem. This module can be enlarged to contemplate a larger number of algorithms allowed. Models which are working in real time can be validated and their statistics stored from here, checking whether the deviation of the results is within the allowed range. If it is not within this range, the models are automatically rebuilt. Temporary deviations of the results from the predictions are corrected or reduced with this mechanism.
The Feedback Collector facilitates a predictive model correction or reinforcement mechanism based on the experience of expert technicians who validate or correct the predictions and on the experience of technicians who, without explicitly validating or correcting the results, have recorded their actions in the TT system.
The types of algorithms that are going to be launched for automatic model generation as well as the automatic attribute selection and the heuristic search for parameters optimizing the result of the algorithms can be established by means of configuration. It is not necessary to go through all the types of algorithms if those offering the best behavior are already known. The thresholds of the statistics of the models which decide whether or not they are still valid as new tickets enter the platform over time can be established. These values can be checked and it can be decided whether or not to replace the old model with the one that was just generated. This model can be tested with test sets that can be configured by the administrator. If everything is okay, the order to replace the models can be sent, stopping the modules, replacing the model and restarting them again. The action to be performed after launching the prediction, sending messages that are visible from the TT system, messages to the output, performing actions directly on the TT system, can be configured by means of an interface for interacting as a user of the system that can perform actions .
From the commercial viewpoint, the application of this Predictive Analytics platform provides the operator with benefits in terms of reducing OPEX, since efficiency is gained with it, optimizing resource management. It also translates into increased customer satisfaction since it reduces ticket resolution time in addition to helping find the best resolution to such tickets.
A person skilled in the art would be able to introduce changes and modifications in the described embodiments without departing from the scope of the invention as defined in the attached claims . ACRONYMS
AUC Area Under Curve
ROC Receiver Operating Characteristic
SVM Support Vector Machine
TT Trouble Ticket
LITERATURE
[1] Fabien Girardin, Josep Blat, Francesco Calabrese, Filippo Dal Fiore, and Carlo Ratti . Digital footprinting : Uncovering tourists with user-generated content. IEEE Pervasive Computing, 7 (4) : 36B43, 2008
[2] Tye Rattenbury, Nathaniel Good, Mor Naaman : Towards automatic extraction of event and place semantics from flickr tags. SIGIR 2007: 103-110

Claims

1. - A method for optimizing and streamlining troubleshooting, said trouble being indicated in tickets generated by a ticket management system, characterized in that it comprises, by means of using a module parallel to said ticket management system:
- collecting said tickets;
- solving the type of trouble in real time by means of predictive models built at least from historical data stored in said ticket management system; and
performing at least one of the following actions: launching an action suggestion, notifying said ticket management system of the action to be performed and/or automatically launching a task depending on a prediction obtained by means of said predictive models.
2. - The method according to claim 1, wherein said troubleshooting is encompassed within the maintenance operations of a telecommunications operator.
3. - The method according to claim 1 or 2, wherein said predictive systems comprise a classification of said tickets depending on the type of trouble they indicate.
4. - The method according to claim 3, which comprises generating said predictive models by testing a pre-established list of algorithms by means of configurable training tickets, choosing those predictive models having better characteristics depending on Receiver Operating Characteristic curves, or ROC curves .
5. - The method according to claim 4, which comprises manually or automatically rebuilding said predictive models depending on validations or corrections concerning the final decision made for each trouble indicated in said tickets, said validations or corrections from a technician or from said ticket management system.
6. - The method according to any of the preceding claims, which comprises extracting data from said ticket management system by means of: a database, web services or file interface.
7. - The method according to claim 6 when it depends on claim 5, which comprises using said data in real time to apply them to said predictive models or using said data for said rebuilding of said predictive models, said data comprising said tickets .
8. - The method according to any of the preceding claims, which comprises calculating the statistics of said predictive models and checking the goodness of fit of said predictive models depending on at least one configurable threshold.
9. - The method according to claim 8, wherein said generation of said predictive models comprises:
performing said extraction of data from said ticket management system;
- correcting tickets associated with said data by means of said corrections from technicians or from said ticket management system;
- generating said configurable training tickets by means of preparing said corrected data and/or tickets;
- combining said configurable tickets with said algorithms;
- heuristically optimizing parameters of said algorithms depending on statistics obtained from said combinations;
- choosing at least one predictive model depending on said statistics; and
- performing said goodness of fit check of said predictive model .
10. - The method according to claim 9, wherein said algorithms are Naive Bayes, Bayes Net, C.45, AdaBoost, Bagging, JRip or Support Vector Machine type algorithms.
11. - The method according to claim 8, 9 or 10, wherein said classification of said tickets comprises:
performing said extraction of data from said ticket management system;
- adjusting tickets associated with said data to the format required by said predictive models;
- launching a classification of a trouble associated with one of said tickets as the result of applying said predictive models on said ticket; and
sending an outgoing message or performing an action automatically depending on said classification.
12. - The method according to any of the preceding claims, which comprises predicting the repetition of a trouble associated with said tickets if a prior action for solving said trouble is unsatisfactory.
13. - A system for optimizing and streamlining troubleshooting, said trouble being indicated in tickets generated by a ticket management system, characterized in that it comprises a module parallel to said ticket management system containing the following sub-modules:
- generator sub-module, which collects said tickets from said ticket management system and creates predictive models;
- predictor sub-module, which resolves the type of trouble in real time by means of said predictive models built at least from historical data stored in said ticket management system and performs at least one of the following actions: launches an action suggestion, notifies said ticket management system of the action to be performed and/or automatically launches a task depending on a prediction obtained by means of said predictive models; and
feedback collector sub-module, which corrects or reinforces said predictive models depending on information entered by a technician or information extracted from said ticket management system, said information referring to the final decision made to solve said trouble.
14. - The system according to claim 13, wherein said sub- modules of said module parallel to said ticket management system have a communication mechanism for communicating with one another .
15. - The system wherein said sub-modules implement the method according to any of the preceding claims 1 to 12.
PCT/EP2012/066423 2011-09-07 2012-08-23 Method and system for optimizing and streamlining troubleshooting WO2013034448A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
ES201131470A ES2408112B1 (en) 2011-09-07 2011-09-07 Method and system for optimization and speeding up incident resolution
ESP201131470 2011-09-07

Publications (1)

Publication Number Publication Date
WO2013034448A1 true WO2013034448A1 (en) 2013-03-14

Family

ID=46801463

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2012/066423 WO2013034448A1 (en) 2011-09-07 2012-08-23 Method and system for optimizing and streamlining troubleshooting

Country Status (2)

Country Link
ES (1) ES2408112B1 (en)
WO (1) WO2013034448A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866790A (en) * 2019-11-26 2020-03-06 上海景域文化传播股份有限公司 Scenic spot ticket sales prediction system and method
WO2020237535A1 (en) * 2019-05-29 2020-12-03 Nokia Solutions And Networks Oy Systems, methods, and computer readable mediums for controlling federation of automated agents
US11741065B2 (en) 2020-02-04 2023-08-29 International Business Machines Corporation Hardware, firmware, and software anomaly handling based on machine learning

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10771314B2 (en) 2017-09-15 2020-09-08 Accenture Global Solutions Limited Learning based incident or defect resolution, and test generation

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3268529B2 (en) * 1990-03-14 2002-03-25 株式会社日立製作所 Knowledge database processing system and expert system
US5666481A (en) * 1993-02-26 1997-09-09 Cabletron Systems, Inc. Method and apparatus for resolving faults in communications networks
US20050169452A1 (en) * 2004-02-03 2005-08-04 Sigma Dynamics, Inc. Method and apparatus for self-evaluation and randomization for predictive models
GB0624024D0 (en) * 2006-12-01 2007-01-10 Ibm Event correlation based trouble ticket resolution system incorporating adaptive rules optimization

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
FABIEN GIRARDIN; JOSEP BLAT; FRANCESCO CALABRESE; FILIPPO DAL FIORE; CARLO RATTI: "Digital footprinting: Uncovering tourists with user-generated content", IEEE PERVASIVE COMPUTING, vol. 7, no. 4, 2008, pages 36 - 43, XP011236560, DOI: doi:10.1109/MPRV.2008.71
JULIA GÓMEZ, YAIZA TEMPRADO, MARGARITA GALLARDO, CAROLINA GARCÍA, FRANCISCO JAVIER MOLINERO: "APPLICATION OF NEURAL NETWORK TO PREDICT ADVERSE SITUATIONS IN TROUBLE TICKETING REPORTS", 2009, pages 204 - 208, XP002686619, ISBN: 978-972-8924-87-4, Retrieved from the Internet <URL:http://www.iadisportal.org/digital-library/application-of-neural-network-to-predict-adverse-situations-in-trouble-ticketing-reports> [retrieved on 20121114] *
TYE RATTENBURY; NATHANIEL GOOD; MOR NAAMAN: "Towards automatic extraction of event and place semantics from flickr tags", SIGIR, 2007, pages 103 - 110
YAIZA TEMPRADO ET AL: "Knowledge Discovery from Trouble Ticketing Reports in a Large Telecommunication Company", COMPUTATIONAL INTELLIGENCE FOR MODELLING CONTROL&AUTOMATION, 2008 INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 10 December 2008 (2008-12-10), pages 37 - 42, XP031495990, ISBN: 978-0-7695-3514-2 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020237535A1 (en) * 2019-05-29 2020-12-03 Nokia Solutions And Networks Oy Systems, methods, and computer readable mediums for controlling federation of automated agents
CN110866790A (en) * 2019-11-26 2020-03-06 上海景域文化传播股份有限公司 Scenic spot ticket sales prediction system and method
US11741065B2 (en) 2020-02-04 2023-08-29 International Business Machines Corporation Hardware, firmware, and software anomaly handling based on machine learning

Also Published As

Publication number Publication date
ES2408112A2 (en) 2013-06-18
ES2408112B1 (en) 2014-02-28
ES2408112R1 (en) 2013-08-06

Similar Documents

Publication Publication Date Title
US10901727B2 (en) Monitoring code sensitivity to cause software build breaks during software project development
EP3889777A1 (en) System and method for automating fault detection in multi-tenant environments
US10877863B2 (en) Automatic prediction system for server failure and method of automatically predicting server failure
US11645191B2 (en) Review process for evaluating changes to target code for a software-based product
US10310968B2 (en) Developing software project plans based on developer sensitivity ratings detected from monitoring developer error patterns
US20050144151A1 (en) System and method for decision analysis and resolution
KR20190021560A (en) Failure prediction system using big data and failure prediction method
EP3920042A1 (en) Hybrid cloud integration deployment and management
Friederich et al. Towards data-driven reliability modeling for cyber-physical production systems
US8794513B2 (en) Self-service device servicing utilizing a hardware database
CN113516244B (en) Intelligent operation and maintenance method and device, electronic equipment and storage medium
WO2013034448A1 (en) Method and system for optimizing and streamlining troubleshooting
CN115664939B (en) Comprehensive operation and maintenance method based on automation technology and storage medium
CN114880312B (en) Flexibly-set application system service data auditing method
Pernici et al. Automatic learning of repair strategies for web services
CN111062827B (en) Engineering supervision method based on artificial intelligence mode
US20230072123A1 (en) Method and system for automating analysis of log data files
KR102489119B1 (en) Smart FMEA system and process management system and method using thereof
CN112069197A (en) Abnormal work order method and device
US20210302954A1 (en) System and method for increasing mean time between service visits in an industrial system
US20150156090A1 (en) Systems and Methods for Monitoring Multiple Services
CN110689144B (en) Management method and device of intelligent recovery device
CN115098739A (en) Message comparison method and device of decision engine
CN111858291B (en) Root cause determination method, equipment and system for data abnormity in charging system migration test
WO2024066346A1 (en) Alarm processing method and apparatus, and storage medium and electronic apparatus

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12756142

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12756142

Country of ref document: EP

Kind code of ref document: A1