US20210342313A1 - Autobuild log anomaly detection methods and systems - Google Patents
Autobuild log anomaly detection methods and systems Download PDFInfo
- Publication number
- US20210342313A1 US20210342313A1 US15/929,476 US202015929476A US2021342313A1 US 20210342313 A1 US20210342313 A1 US 20210342313A1 US 202015929476 A US202015929476 A US 202015929476A US 2021342313 A1 US2021342313 A1 US 2021342313A1
- Authority
- US
- United States
- Prior art keywords
- log
- log data
- test
- line
- matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1873—Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/368—Test management for test version control, e.g. updating test cases to a new software version
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
- G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
- G06F7/5443—Sum of products
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/81—Threshold
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
Abstract
Computing systems, database systems, and related methods are provided for detecting anomalies within a log file. One method involves obtaining log data for test runs executed with respect to a compiled version of executable code for an application platform, filtering the log data based on one or more performance metrics to obtain reference log data, converting the reference log data to a corresponding numerical representation and generating a matrix of the numerical representation. For each line of test log data associated with an update to the executable code, the method converts the line into a numerical representation, determines a difference between the numerical representation and the matrix, and provides an indication of an anomaly when the difference is greater than a detection threshold.
Description
- Embodiments of the subject matter described herein relate generally to database systems, and more particularly, to methods and systems that support detecting anomalies within a test log generated from tests run against executable code for an on-demand application platform.
- Modern software development often employs testing in order to eliminate as many errors or “bugs” as possible before the software is released for widespread use. Automated software testing techniques are often utilized to verify that a program performs consistent with its specified or intended behavior. Unit testing is one method of automated testing that targets individual functions, procedures, or other portions of source code, for example, by isolating parts of a software program to show that the individual constituent parts are correct. To this end, test cases are often created that specify the inputs, execution conditions, testing procedure, testing results, and/or the like that in combination define the respective tests to be performed on a given software program. Test cases are typically written for all functions and methods so that whenever a change causes a fault, it can be quickly identified and fixed, thereby ensuring that code meets its design and behaves as intended early in the development cycle.
- In practice, as code increases in length and complexity and the number of test cases increase accordingly, it becomes increasingly difficult to identify problems and root causes when aspects of the code are changed. For example, during development, one or more members of a development team may make changes to the source code for a piece of software, which may result in one or more failed tests when a batch of tests are run against the modified source code. However, manually analyzing the voluminous amount of information contained in a log of the test results for debugging purposes can be time consuming and prone to user error or lapses. Accordingly, it is desirable to reduce the amount of time and effort required for developers to analyze test results and identify and resolve errors within source code.
- A more complete understanding of the subject matter may be derived by referring to the detailed description and claims when considered in conjunction with the following figures, wherein like reference numbers refer to similar elements throughout the figures.
-
FIG. 1 is a block diagram of an exemplary computing system; -
FIG. 2 is a flow diagram of an exemplary vector space modeling process suitable for implementation in the computing system ofFIG. 1 in accordance with one or more embodiments; -
FIG. 3 is a flow diagram of an exemplary anomaly detection process suitable for implementation in the computing system ofFIG. 1 in connection with the vector space modeling process ofFIG. 2 in accordance with one or more embodiments; -
FIG. 4 depicts an exemplary log anomaly analysis graphical user interface (GUI) display suitable for presentation in connection with the anomaly detection process ofFIG. 3 in one or more exemplary embodiments; and -
FIG. 5 is a block diagram of an exemplary multi-tenant system suitable for use with the computing system ofFIG. 1 in accordance with one or more embodiments. - Embodiments of the subject matter described herein generally relate to computing systems and methods for detecting testing anomalies with respect to changes or updates to computer-executable code. For purposes of explanation, the subject matter may be described herein primarily in the context of detecting anomalies with respect to a compiled version of executable code for an application platform capable of concurrently providing instances of a virtual application to one or more different end users, such as, for example, an automatically compiled branch of source code for the application platform. In this regard, when a software developer makes a change with respect to the source code for a particular version of the application platform, an autobuild system supporting a continuous integration/continuous deployment (Cl/CD) pipeline may automatically compile the updated source code into an executable version of the updated source code for the application platform. The autobuild system compiling the updated source code may trigger a testing system executing a plurality of tests against the complied executable code for the application platform to generate a log file including data associated with the performance of the code with respect to the respective tests.
- As described in greater detail below, each line of the test log data contained in the log file is analyzed with respect to a vector space model associated with the respective version of the application platform to identify anomalies when a difference between the respective line of test log data and the vector space model is greater than a detection threshold. In this regard, the vector space model associated with the respective version of the application platform represents the non-anomalous or reference test performance of the tests with respect to that version of the application platform source code. For example, prior runs of the tests with respect to previous compilations of the respective version of the application platform source code that achieved satisfactory test performance or test performance metrics greater than some threshold may represent the reference test log result space for that application platform source code version. The test log data associated with those respective prior test runs may be utilized to generate a matrix that provides a multi-dimensional numerical representation of the non-anomalous test log result space for the application platform source code version.
- After obtaining a matrix that represents a reference, non-anomalous multidimensional vector space for the test results for the application platform source code version, each line of the test log data associated with the test run against the updated compilation of that application platform source code version is compared to the non-anomalous vector space to identify anomalous deviations from the vector space model. Each line of the test log data associated with the test run against the updated compilation of application platform source code is converted to a numerical representation in a corresponding manner as was used to generate the matrix, and then a deviation metric for the respective line may be calculated based on the difference between the respective numerical representation of the respective line of test log data and the vector space matrix, for example, by computing a dot product between the log line and the vector space matrix. In this regard, when the magnitude of the dot product is greater than a detection threshold, the line of test log data may be marked or otherwise flagged as anomalous. In this manner, the lines of the test log data may be analyzed with respect to the reference vector space matrix to identify a subset of anomalous lines within the test log data. Indicia of the anomalous lines may be output or otherwise provided to a user or other entity to facilitate debugging with respect to the changes or updates to the application platform source code. By identifying the anomalies or deviations relative to the aggregate multidimensional non-anomalous test result vector space, anomalous log lines may be identified even in scenarios where there may be one or more individually similar lines within the non-anomalous test result vector space but a relatively large deviation between those log lines and the test result vector space, thereby identifying potential anomalies or issues that could otherwise go unrecognized by manual reviewers or other automated approaches (e.g., nearest neighbor algorithms).
-
FIG. 1 depicts an exemplary embodiment of acomputing system 100 for detecting and providing corresponding indicia of anomalies within log data associated with changes or updates to executable code, such as a compiled version of executable source code for anapplication platform 124 provided by aserver 102. In the illustrated embodiment, aserver 102 is communicatively coupled to adatabase 104 and one or more instances ofclient devices 106 via acommunications network 108. Thecommunications network 108 may be realized as any sort or combination of wired and/or wireless computer network, a cellular network, a mobile broadband network, a radio network, the Internet, or the like. Atesting system 110 is communicatively coupled to thecommunications network 108 to support automated testing of the compiled executable code and generation of corresponding log files that maintain the performance data and other results associated with a respective test run, as described in greater detail below. A log anomaly detection (LAD)system 112 is communicatively coupled to thecommunications network 108 and configured to analyze log files generated by thetesting system 110 to detect and provide corresponding indicia of anomalous lines of a respective log file. - It should be noted that
FIG. 1 is a simplified representation of acomputing system 100 for purposes of explanation and is not intended to be limiting. For example, in practice, multiple instances ofclient devices 106 communicating on thenetwork 108 may be present, with any number of instances of applications being provided by theserver 102 to various instances ofclient devices 106. Additionally, practical implementations may include multiple instances of theserver 102 and/or thedatabase 104, which, in turn may reside behind or rely on one or more load balancers to manage resource utilization, as will be appreciated in the art. Furthermore, although thedatabase 104 is depicted separate from theapplication server 102, in practice, thedatabase 104 may be integrated or otherwise combined with theapplication server 102 as part of an integrated database system. Alternatively, thedatabase 104 may be integrated or otherwise combined with thetesting system 110 and/or theLAD system 112. Moreover, in some embodiments, the features and/or functionality described herein in the context of one or more of thesystems server 102 and/or thedatabase 104, and in which case arespective system computing system 100. - The
server 102 generally represents a computing device, computing system or another combination of processing logic, circuitry, hardware, and/or other components configured to support the processes, tasks, operations, and/or functions described herein. In this regard, theserver 102 generally includes aprocessing system 120, which may be implemented using any suitable processing system and/or device, such as, for example, one or more processors, central processing units (CPUs), controllers, microprocessors, microcontrollers, processing cores and/or other hardware computing resources configured to support the operation of theprocessing system 120 described herein. Theprocessing system 120 may include or otherwise access a data storage element 122 (or memory) capable of storing programming instructions for execution by theprocessing system 120, that, when read and executed, causeprocessing system 120 to support the processes described herein. Depending on the embodiment, thememory 122 may be realized as a random-access memory (RAM), read only memory (ROM), flash memory, magnetic or optical mass storage, or any other suitable non-transitory short or long-term data storage or other computer-readable media, and/or any suitable combination thereof. In one or more embodiments, the programming instructions cause theprocessing system 120 to create, generate, or otherwise facilitate anapplication platform 124 that supports or otherwise facilitates the processes, tasks, operations, and/or functions described herein. In some embodiments, theapplication platform 124 may also generate or otherwise provide instances of a virtual application at run-time (or “on-demand”) based at least in part upon code and other data that is stored or otherwise maintained by a database (e.g., database 104), such that theserver 102 functions as anapplication server 102. - Similarly, the
testing system 110 generally includes a server or other computing device, computing system or another combination of processing logic, circuitry, hardware, and/or other components configured to support the processes, tasks, operations, and/or functions described herein. In exemplary embodiments, thetesting system 110 includes processing system that includes or accesses a data storage element storing executable instructions or code that cause the processing system to create, generate, or otherwise facilitate a testing application that supports executing a plurality oftest cases 114, alternatively referred to herein as a test run, with respect to compiled code executed by theprocessing system 120 at theapplication server 102 to provide theapplication platform 124. The testing application stores or otherwise maintains the results of the test run in alog file 132 maintained in a data storage element associated with thetesting system 110. In one or more embodiments, the testing application is realized as an application programming interface (API), software agent, or the like that is capable of interacting with another element in thecomputing system 100, such as theclient device 106, theapplication server 102, thetesting system 110 and/or thedatabase 104, as described in greater detail below. - In exemplary embodiments, the LAD
system 112 includes one ormore computing devices 130, which may be realized as one or more server computing systems or other suitable processing systems or combinations of processing logic, circuitry, hardware, and/or other components configured to support the processes, tasks, operations, and/or functions described herein. TheLAD system 112 also includes a data storage element or memory configured to store or otherwise maintainvector space models 132 generated by theLAD system 112 and utilized to detect anomalous log lines for a particular code version, as described in greater detail below. The data storage element or memory associated with theLAD system 112 may also maintain executable instructions or code that cause acomputing device 130 of theLAD system 112 to create, generate, or otherwise facilitate an analysis application that supports analyzing a log file associated with a test run with respect to the appropriatevector space model 132 and provide corresponding indicia of anomalous log lines to another element of thecomputing system 100, such as theclient device 106, theapplication server 102, thetesting system 110 and/or thedatabase 104, as described in greater detail below. In one or more embodiments, the analysis application is realized as an application programming interface (API), software agent, or the like that is capable of interacting with another element of thecomputing system 100. - The
client device 106 generally represents an electronic device coupled to thenetwork 108 that may be utilized by a user to write or develop source code, update or otherwise make changes to existing source code, execute test cases on source code, review and/or analyze test results and corresponding log files to identify anomalies within log files, as described in greater detail below. In practice, theclient device 106 can be realized as any sort of personal computer, mobile telephone, tablet or other network-enabled electronic device. In exemplary embodiments, theclient device 106 includes a display device, such as a monitor, screen, or another conventional electronic display, capable of graphically presenting data and/or information along with a user input device, such as a touchscreen, a touch panel, a mouse, a joystick, a directional pad, a motion sensor, or the like, capable of receiving input from the user of theclient device 106. The illustratedclient device 106 executes or otherwise supports aclient application 107 that supports communication with theapplication platform 124 on theserver 102 using a networking protocol. In some embodiments, theclient application 107 is realized as a web browser or similar local client application executed by theclient device 106 that contacts theapplication server 102 and/orapplication platform 124 using a networking protocol, such as the hypertext transport protocol (HTTP) or the like. One ormore client applications 107 at theclient device 106 may variously be utilized to communicate with thetesting system 110, theLAD system 112, and/or thedatabase 104. - In exemplary embodiments, the
database 104 stores or otherwise maintains one or more versions of source code that are executable by theprocessing system 120 at theserver 102 to generate, facilitate, or otherwise provide theapplication platform 124 and generate one or more instances of avirtual application 126. For example, thedatabase 104 may store compiled versions of source code corresponding to different versions of theapplication platform 124. In this regard, an autobuild system or other continuous integration/continuous deployment (Cl/CD) pipeline may support and maintain compiled versions of source code for different versions of theapplication platform 124, such as, for example, a lower level version of theapplication platform 124 utilized by developers to make changes, modifications, or additions to the functionality of theapplication platform 124, one or more intermediate versions of theapplication platform 124 that may be utilized by developers and potentially some subset of end users or customers for debugging or performance analysis (e.g., beta testing), and a higher level version of the application platform 124 (e.g., a production version) that may be accessible to a broader set of end users, customers, or the like. In one embodiment, the autobuild system is configured to automatically compile the source code for the respective versions of theapplication platform 124 on a continual and periodic basis (e.g., hourly, daily, weekly, or the like) and store the most recently compiled version of the source code for the respective application platform version in thedatabase 104. For purposes of explanation, a compiled version of application platform version source code generated by an autobuild system is referred to herein asautobuild code 140. In some embodiments, thedatabase 104 may also store or otherwise maintain other data for integration with or invocation by avirtual application 126 in objects organized in object tables. -
FIG. 2 depicts an exemplary embodiment of a vectorspace modeling process 200 suitable for implementation in thecomputing system 100 to dynamically and automatically generate vector space models based on reference test result log files that may be utilized to identify anomalies within subsequent test result log files, as described in greater detail below in the context ofFIG. 3 . For illustrative purposes, the following description may refer to elements mentioned above in connection withFIG. 1 . In this regard, while portions of the vectorspace modeling process 200 may be performed by different elements of thecomputing system 100, for purposes of explanation, the subject matter is described herein in the context of the vectorspace modeling process 200 being primarily performed by theLAD system 112 and/or computing device(s) 130. It should be appreciated that the vectorspace modeling process 200 may include any number of additional or alternative tasks, the tasks need not be performed in the illustrated order and/or the tasks may be performed concurrently, and/or the vectorspace modeling process 200 may be incorporated into a more comprehensive procedure or process having additional functionality not described in detail herein. Moreover, one or more of the tasks shown and described in the context ofFIG. 2 could be omitted from a practical embodiment of the vectorspace modeling process 200 as long as the intended overall functionality remains intact. - The illustrated vector
space modeling process 200 begins by identifying or otherwise determining a code version for which to develop a vector space model and retrieving or otherwise obtaining log files for test runs associated with that identified code version (tasks 202, 204). For example, theLAD system 112 may perform the vectorspace modeling process 200 on a periodic basis (e.g., daily, weekly, monthly, and/or the like) to develop a differentvector space model 132 for each version ofautobuild code 140. In this regard, the autobuild system may automatically compile the different versions of source code for theapplication platform 124 into different instances ofautobuild code 140 corresponding to the respective application platform version, and then initiate testing of the respective versions ofautobuild code 140 by thetesting system 110. For example, upon compiling source code for a version of theapplication platform 124, the autobuild system may utilize an API or similar feature to trigger thetesting system 110 running a set oftest cases 114 against theautobuild code 140. To test theautobuild code 140, theautobuild code 140 is loaded or otherwise executed by theprocessing system 120 at anapplication server 102 to generate the corresponding version of theapplication platform 124, which is then accessible to thetesting system 110 via thenetwork 108. In this manner, upon the source code being compiled, the newly built version(s) will go be tested against a set of predefined tests which are most applicable to the respective version(s). In practice, theapplication server 102 may be part of the autobuild system, or the autobuild system may include a duplicative or redundant instance of theserver 102 that tests may be run against. - In exemplary embodiments, the
testing system 110 automatically runs one or more automated test cases, such as graphical user interface (GUI) tests, functional tests, unit tests, and/or the like, against the version of the application platform 124 (or with respect to an instance of thevirtual application 126 generated by the respective version of the application platform 124) and creates a corresponding log data file 116 associated with the test run. For eachtest 114 that makes up the test run, the associated test result log data file 116 includes a line, row, or similar delineated entry associated with that respective test that includes various information characterizing or describing the execution of thetest 114. For example, the log line may include, without limitation, descriptive information or data identifying the test 114 (e.g., one or more unique identifiers assigned to the test case), performance metrics or data associated with execution of the test 114 (e.g., the duration of time required to complete the test case, whether the result of the test corresponds to passing or failing the test case, etc.), contextual metadata characterizing the operating context for the test (e.g., the date and time of execution of the test, information identifying theparticular server 102 the test was performed on or with respect to, and/or the like), and potentially other test result or debugging data, such as, HTTP callouts, errors, exceptions or other events generated during the test, automated processes or workflows invoked or involved in the test, database changes during the test, and/or the like. The resulting test result log data file 116 then includes a plurality of lines of data (alternatively referred to herein as log lines) corresponding to eachindividual test 114 that made up the respective test run, with the log data file 116 being stored or otherwise maintained in association with the particular autobuild code version (e.g., by associating each test result log data file 116 with an identifier or other indicia of the corresponding version of autobuild code 140). TheLAD system 112 may thereby retrieve or otherwise obtain, from thetesting system 110 via thenetwork 108, a set of log data files 116 associated with a particular version ofautobuild code 140 to be modeled. For example, theLAD system 112 may obtain a subset of recent log data files 116 (e.g., the ten most recent log data files) associated with theautobuild code 140 of interest for further analysis. - Still referring to
FIG. 2 , the vectorspace modeling process 200 continues by filtering or otherwise excluding one or more log files from the obtained log files for the code version of interest to obtain a subset of reference log files for model development (task 206). In this regard, in exemplary embodiments, theLAD system 112 analyzes the performance data or metrics associated with the different obtainedlog files 116 to identify the subset of the log files that reflect the best performance across the test run or are otherwise likely to exhibit the fewest anomalies, that is, the log files having associated performance metrics that satisfy corresponding selection thresholds. In other words, the filtered subset of log files represents normal or good test results for the autobuild code version that may be utilized as a reference for identifying anomalous test results. For example, from within the ten mostrecent log files 116 obtained for a given autobuild code version, theLAD system 112 may preferentially select a subset of the fivelog files 116 that had the lowest associated execution time for completing the entire test run oftest cases 114, the highest aggregate rate of passingtest cases 114 across the entire test run oftest cases 114, the fewest number or rate of errors or exceptions across the entire test run oftest cases 114, and/or the like. In some embodiments, theLAD system 112 may perform a multivariate analysis to identify or optimize the retained subset oflog files 116 across different performance metrics or parameters. - After identifying a reference subset of log files, the vector
space modeling process 200 filters or otherwise removes nondescript text or other nondescript information from the log lines within the reference log files (task 208). In this regard, theLAD system 112 goes through each log line of each reference log file to remove various words or other alphanumeric characters or markup within each log line that does not pertain to the performance or execution of the respective test case. For example, theLAD system 112 may remove commonly used words (e.g., grammatical articles, prepositions, pronouns, and/or the like) dates, timestamps, or other metadata characterizing aspects of the test run or test environment that are ancillary, tangential, or otherwise not relevant to or probative of the performance of the underlying code being tested, such as universally unique identifiers (UUIDs) or the like. Thus, the result of theLAD system 112 filtering nondescript information from a reference log file results in a filtered reference log file with modified log lines that retain and include only descriptive text or information that remain after filtering. - After removing nondescript information from the reference log files, the vector
space modeling process 200 continues by generating or otherwise constructing a matrix from the modified log lines of the filtered reference log files (task 210). In this regard, theLAD system 112 may construct a multidimensional matrix out of individual vectors, where each vector corresponds to a respective log line of a respective reference log file, and the entries of the vector correspond to the remaining pieces of data or information retained within the respective modified log line of the respective reference log file. Thus, each entry of the resulting matrix corresponds to a discrete segment of alphanumeric text or characters that remains within a modified log line of a respective modified log line. It should be noted that the subject matter described herein is not limited to any particular arrangement or number of dimensions for the vector space matrix. - Still referring to
FIG. 2 , after constructing the matrix of remaining test log data within the filtered reference log files, the vectorspace modeling process 200 hashes or otherwise converts each entry within the matrix to a corresponding numerical value, resulting in a numerical representation of the vector space defined by the filtered reference log files (task 212). In this regard, each combination of alphanumeric characters may be assigned a unique numeric value, such that different words have different assigned numeric values but each instance of a given word within the matrix that appears within different log lines or log files is assigned the same value. For example, theLAD system 112 may use a hashing vectorizer, term frequency-inverse document frequency (TD-IDF), or another suitable technique or algorithm to assign numerical values to different segments of alphanumeric characters based on the estimated relevance or prevalence of the respective character segment. - The resulting numerical matrix representation of the filtered reference log file vector space is stored or otherwise maintained at the
LAD system 112 as thevector space model 132 associated with the particular version ofautobuild code 140. For example, if theLAD system 112 identifies the lower level mainline code branch of theapplication platform 124 as the version ofautobuild code 140 of interest, the LAD system 112 (e.g., one of the computing devices 130) may retrieve, from thetesting system 110, the log files 116 corresponding to the ten most recent test runs against the mainline autobuild code 140 (e.g.,tasks 202, 204). TheLAD system 112 and/orcomputing device 130 analyzes the test result log data maintained in the obtainedlog files 116 to identify and select the subset oflog files 116 that achieve a desired performance (e.g., a pass/fail rate above a threshold value, a cumulative execution time below a threshold, a number of errors or exceptions below a threshold, etc.) (e.g., task 206). TheLAD system 112 and/orcomputing device 130 then filters nondescript information from the log lines of reference subset of mainline autobuild log files and generates a corresponding numerical representation of the resulting modified log lines of the reference mainline autobuild log files (e.g.,tasks LAD system 112 as the mainline autobuild codevector space model 132. - In practice, the vector
space modeling process 200 may be periodically performed with respect to each potential code version of interest to dynamically update thevector space models 132 as the code versions and/or test cases change over time. For example, thetesting system 110 may automatically initiate theLAD system 112 performing the vectorspace modeling process 200 with respect to a particular code version each time a particular version ofautobuild code 140 is updated and retested by thetesting system 110. That said, in other embodiments, the vectorspace modeling process 200 may be triggered or otherwise initiated automatically by theLAD system 112 based on one or more performance metrics associated with a respectivevector space model 132. For example, if the mainline autobuild codevector space model 132 results in detection of a number, percentage or rate of anomalies that is greater than a model updating threshold when applied to a subsequent log file as described in greater detail below in the context ofFIG. 3 , theLAD system 112 may automatically initiate the vectorspace modeling process 200 to dynamically update the mainline autobuild codevector space model 132 to improve performance. -
FIG. 3 depicts an exemplary embodiment of ananomaly detection process 300 suitable for implementation in thecomputing system 100 to detect anomalous log lines using a vector space model resulting from the vectorspace modeling process 200 ofFIG. 2 . For illustrative purposes, the following description may refer to elements mentioned above in connection withFIG. 1 . In this regard, while portions of theanomaly detection process 300 may be performed by different elements of thecomputing system 100, for purposes of explanation, the subject matter is described herein in the context of theanomaly detection process 300 being primarily performed by theLAD system 112 and/or computing device(s) 130. It should be appreciated that theanomaly detection process 300 may include any number of additional or alternative tasks, the tasks need not be performed in the illustrated order and/or the tasks may be performed concurrently, and/or theanomaly detection process 300 may be incorporated into a more comprehensive procedure or process having additional functionality not described in detail herein. Moreover, one or more of the tasks shown and described in the context ofFIG. 3 could be omitted from a practical embodiment of theanomaly detection process 300 as long as the intended overall functionality remains intact. - Referring to
FIG. 3 , with continued reference toFIGS. 1-2 , in exemplary embodiments, theanomaly detection process 300 is performed after additions, deletions, or other updates or changes are made to the source code for a particular version of theapplication platform 124. For example, a developer looking to add a new feature or functionality may modify the source code for the lower level mainline code branch of theapplication platform 124 to incorporate the functionality. In this regard, theclient application 107 at theclient device 106 may be configured to provide a source code editor that allows a developer or other user of theclient device 106 to make changes to the source code via thenetwork 108. After updating the source code, the developer may initiate compiling of the modified source code (or the compiling may be initiated automatically by the application platform 124), for example, by the autobuild system, resulting in modified autobuild code for the mainline code branch version of theapplication platform 124. Thereafter, the developer may initiate the testing system 110 (e.g., via an API call by the client application 107) performing a test run of test cases against the modified autobuild code (or the testing may be initiated automatically by the autobuild system via an API call) to generate a testresult log file 116 that reflects the updates to the source code for the mainline code branch version of theapplication platform 124. Subsequently, the developer may initiate theLAD system 112 performing the anomaly detection process 300 (or the testing system may automatically initiate theLAD system 112 performing theanomaly detection process 300 in response to a poorly performing or failed test run via an API call) by retrieving or otherwise obtaining the test result log file of interest for analysis (task 302). For example, the developer may interact with theclient application 107 to cause theclient application 107 to make an API call to theLAD system 112 that includes identification of the testresult log file 116 to be analyzed (e.g., by providing a unique identifier associated with the modified autobuild code for the mainline code branch version that allows theLAD system 112 to retrieve the appropriate testresult log file 116 from the testing system 110). In other embodiments, thetesting system 110 may automatically initiate or trigger theanomaly detection process 300 when a test run fails to execute to completion. - Similar to the vector
space modeling process 200 ofFIG. 2 , theanomaly detection process 300 filters or otherwise removes nondescript information from the retrieved test result log file and converts the remaining information in the test result log file to numerical representation (tasks 304, 306). For example, as described above, theLAD system 112 goes through each log line of each reference log file to remove various words or other alphanumeric characters or markup within each log line that does not pertain to the performance or execution of the respective test case of the test run that corresponds to that log line (e.g., task 208). TheLAD system 112 then hashes or otherwise converts each remaining distinct segment of alphanumeric characters within a respective log line into a corresponding numerical value using the same hash function, tokenization technique, or algorithm used to develop the vector space model (e.g., task 212). - After filtering and converting each log line into a numerical vector or matrix representation, the
anomaly detection process 300 calculates or otherwise determines a difference metric for each respective log line with respect to the vector space model for the code version of interest, and detects or otherwise identifies an anomaly when the difference metric exceeds a detection threshold value (tasks 308, 310). In exemplary embodiments, theLAD system 112 converts each log line into a vector comprised of the respective numerical values of the log line in sequential order. TheLAD system 112 then calculates or otherwise determines the difference between a respective log line and the non-anomalous vector space by calculating a dot product of the log line vector and the vector space matrix. In this regard, continuing the above example, for each log line oftest log file 116 corresponding to the modified autobuild code for the mainline code branch version, theLAD system 112 calculates the dot product between that respective log line and the vectorspace matrix model 132 associated with the autobuild code for the mainline code branch version of theapplication platform 124 that was generated as described above in the context ofFIG. 2 . The resulting dot product value represents the deviation between the respective log line of the modified mainline autobuild code log file and the reference vector space for the mainline autobuild code. For example, the anomaly detection threshold may be set to a value of 0.3 to identify deviating log lines having a resultant dot product value greater than 0.3. That said, the anomaly detection threshold may vary depending on the needs of a particular application and to vary the number or frequency of potentially anomalous log lines identified, and the subject matter described herein is not limited to any particular anomaly detection threshold. - When the difference between a respective log line and the vector space matrix model is greater than the anomaly detection threshold, the
anomaly detection process 300 stores or otherwise maintains the respective log line identified as anomalous, the difference metric, and potentially other data associated with the anomalous log line (task 312). For example, theLAD system 112 may store or otherwise maintain an identifier associated with the log line (e.g., the location of the log line within the log file 116), the original unmodified content of the log line, the dot product value associated with the respective log line, and potentially other data or information that may be relevant to debugging or analyzing the anomalous log line. For example, in some embodiments, one or more preceding or following log lines in thelog file 116 may also be stored or maintained in association with the anomalous log line to facilitate temporal or sequential analysis of the log line with respect to adjacent log lines in thelog file 116. Theanomaly detection process 300 repeats the loop defined bytasks - After processing of the log file is complete, the
anomaly detection process 300 continues by creating or otherwise generating a log anomaly file that includes a listing of the anomalous log lines and their associated difference values and generating or otherwise providing graphical indicia of the anomalous log lines within the log file (tasks 316, 318). In this regard, theLAD system 112 may create an electronic file or other container that includes, for each anomalous log line, one or more of the identifier associated with the log line and/or the location of the log line within thelog file 116, the dot product value associated with the respective log line, the original unmodified contents of the log line, and/or potentially other data or information that may be relevant to debugging or analyzing the anomalous log line (e.g., an identification of thetest case 114 the log line pertains to). - The
LAD system 112 may transmit or otherwise provide the log anomaly file to one or more of theapplication server 102, thetesting system 110, and/or theclient device 106 for depicting a graphical representation of the anomalous log lines at theclient device 106. For example, avirtual application 126 at theapplication server 102 may generate a log anomaly analysis graphical user interface (GUI) display that includes a sequential listing of the anomalous log lines and their associated dot product values. It should be appreciated that the subject matter described herein is not intended to be limited to any particular GUI display or manner of presenting the anomalous log lines. In this regard, any number of dashboards, reports or visualizations may be generated based on the anomalous log lines and their corresponding test results, test case definitions and/or the like. The developer or other user at theclient device 106 may then analyze the original unmodified content of the log line to debug the changes or modifications he or she made to the source code for the mainline code branch version of theapplication platform 124. In this regard, by virtue of theanomaly detection process 300, the user may quickly identify anomalous log lines and thecorresponding test cases 114 to identify and remedy potential problems in changes made to source code without having to manually analyze the entire content of alog file 116. Theanomaly detection process 300 may also facilitate identifying problems with underlying hardware issues or control plane software systems that are executing and managing tests and base systems. - For example,
FIG. 4 depicts an exemplary log anomalyanalysis GUI display 400 that may be depicted by or within a client application 107 (e.g., on a web page associated with a virtual application). In this regard, it should be noted that although the subject may be described herein in the context of a log anomaly file, in some embodiments, the results of theanomaly detection process 300 may be streamed by theLAD system 112 to theclient application 107, theapplication server 102, and/or some other system over thenetwork 108 for presentation or storage by or at that respective destination device or system, in which case, theLAD system 112 need not package or encapsulate the results in a dedicated file at theLAD system 112. The log anomalyanalysis GUI display 400 includes graphical representations of thedot product values 402 associated withrespective log lines 404 of a test result log file with respect to a vector space model derived from a reference log file including six lines of “This is just status quo.” A user viewing the log anomalyanalysis GUI display 400 may readily identify the fifth log line as the anomalous log line within the test result log file based on its associated dot product value of 0.75 being greater than the remaining dot product values. In this regard, while the log anomalyanalysis GUI display 400 depicts the entire 9 log lines of the test result log file, in practice, the log anomalyanalysis GUI display 400 may be generated based on a log anomaly file that only includes anomalous log lines having an associated dot product value greater than an anomaly detection threshold (e.g., 0.3), such that only the fifth log line is depicted on the log anomalyanalysis GUI display 400. In exemplary embodiments, the log anomalyanalysis GUI display 400 includes the dot product value associated with an anomalous log line so that the user may readily ascertain the relative difference between the anomalous log line and the reference vector space as well as the contents of the anomalous log line, so that the user may view the contents of the log line and initiate debugging or further analysis of the log line. - Still referring to
FIG. 3 , in one or more embodiments, lines of the log file are analyzed concurrently, for example, by distributing or otherwise allocating individual log lines across multipledifferent computing devices 130 that perform the loop defined bytasks computing devices 130 may be configured to store or otherwise maintain information or other indicia associated with detected anomalous log lines in a shared data storage element or memory, which may then be assembled into a log anomaly file. In this regard, theanomaly detection process 300 can be run concurrently, on N number of compute units in any order, and additionally, N number of test result log files can be compared to N number of vector space models concurrently. The parallel computing system can thus be scaled up or down in size as desired. -
FIG. 5 depicts an exemplary embodiment of an on-demandmulti-tenant database system 500 suitable for use with thecomputing system 100 ofFIG. 1 . The illustratedmulti-tenant system 500 ofFIG. 5 includes aserver 502, such asserver 102, that dynamically creates and supports virtual applications 528 (e.g., virtual application 126) based upondata 532 from a common database 530 (e.g., database 104) that is shared between multiple tenants, alternatively referred to herein as a multi-tenant database. Data and services generated by thevirtual applications 528 are provided via a network 545 (e.g., network 108) to any number of client devices 540 (e.g., client device 106), as desired. Eachvirtual application 528 is suitably generated at run-time (or on-demand) using a common application platform 510 (e.g., application platform 124) that securely provides access to thedata 532 in thedatabase 530 for each of the various tenants subscribing to themulti-tenant system 500. In accordance with one non-limiting example, themulti-tenant system 500 is implemented in the form of an on-demand multi-tenant customer relationship management (CRM) system that can support any number of authenticated users of multiple tenants. - As used herein, a “tenant” or an “organization” should be understood as referring to a group of one or more users that shares access to common subset of the data within the
multi-tenant database 530. In this regard, each tenant includes one or more users associated with, assigned to, or otherwise belonging to that respective tenant. To put it another way, each respective user within themulti-tenant system 500 is associated with, assigned to, or otherwise belongs to a particular tenant of the plurality of tenants supported by themulti-tenant system 500. Tenants may represent customers, customer departments, business or legal organizations, and/or any other entities that maintain data for particular sets of users within the multi-tenant system 500 (i.e., in the multi-tenant database 530). For example, theapplication server 502 may be associated with one or more tenants supported by themulti-tenant system 500. Although multiple tenants may share access to theserver 502 and thedatabase 530, the particular data and services provided from theserver 502 to each tenant can be securely isolated from those provided to other tenants (e.g., by restricting other tenants from accessing a particular tenant's data using that tenant's unique organization identifier as a filtering criterion). The multi-tenant architecture therefore allows different sets of users to share functionality and hardware resources without necessarily sharing any of thedata 532 belonging to or otherwise associated with other tenants. - The
multi-tenant database 530 is any sort of repository or other data storage system capable of storing and managing thedata 532 associated with any number of tenants. Thedatabase 530 may be implemented using any type of conventional database server hardware. In various embodiments, thedatabase 530shares processing hardware 504 with theserver 502. In other embodiments, thedatabase 530 is implemented using separate physical and/or virtual database server hardware that communicates with theserver 502 to perform the various functions described herein. In an exemplary embodiment, thedatabase 530 includes a database management system or other equivalent software capable of determining an optimal query plan for retrieving and providing a particular subset of thedata 532 to an instance ofvirtual application 528 in response to a query initiated or otherwise provided by avirtual application 528. Themulti-tenant database 530 may alternatively be referred to herein as an on-demand database, in that themulti-tenant database 530 provides (or is available to provide) data at run-time to on-demandvirtual applications 528 generated by theapplication platform 510. - In practice, the
data 532 may be organized and formatted in any manner to support theapplication platform 510. In various embodiments, thedata 532 is suitably organized into a relatively small number of large data tables to maintain a semi-amorphous “heap”-type format. Thedata 532 can then be organized as needed for a particularvirtual application 528. In various embodiments, conventional data relationships are established using any number of pivot tables 534 that establish indexing, uniqueness, relationships between entities, and/or other aspects of conventional database organization as desired. Further data manipulation and report formatting is generally performed at run-time using a variety of metadata constructs. Metadata within a universal data directory (UDD) 536, for example, can be used to describe any number of forms, reports, workflows, user access privileges, business logic and other constructs that are common to multiple tenants. Tenant-specific formatting, functions and other constructs may be maintained as tenant-specific metadata 538 for each tenant, as desired. Rather than forcing thedata 532 into an inflexible global structure that is common to all tenants and applications, thedatabase 530 is organized to be relatively amorphous, with the pivot tables 534 and themetadata 538 providing additional structure on an as-needed basis. To that end, theapplication platform 510 suitably uses the pivot tables 534 and/or themetadata 538 to generate “virtual” components of thevirtual applications 528 to logically obtain, process, and present the relativelyamorphous data 532 from thedatabase 530. - Still referring to
FIG. 5 , theserver 502 is implemented using one or more actual and/or virtual computing systems that collectively provide thedynamic application platform 510 for generating thevirtual applications 528. For example, theserver 502 may be implemented using a cluster of actual and/or virtual servers operating in conjunction with each other, typically in association with conventional network communications, cluster management, load balancing and other features as appropriate. Theserver 502 operates with any sort ofconventional processing hardware 504, such as aprocessor 505,memory 506, input/output features 507 and the like. The input/output features 507 generally represent the interface(s) to networks (e.g., to thenetwork 545, or any other local area, wide area or other network), mass storage, display devices, data entry devices and/or the like. Theprocessor 505 may be implemented using any suitable processing system, such as one or more processors, controllers, microprocessors, microcontrollers, processing cores and/or other computing resources spread across any number of distributed or integrated systems, including any number of “cloud-based” or other virtual systems. Thememory 506 represents any non-transitory short or long-term storage or other computer-readable media capable of storing programming instructions for execution on theprocessor 505, including any sort of random access memory (RAM), read only memory (ROM), flash memory, magnetic or optical mass storage, and/or the like. The computer-executable programming instructions, when read and executed by theserver 502 and/orprocessor 505, cause theserver 502 and/orprocessor 505 to create, generate, or otherwise facilitate theapplication platform 510 and/orvirtual applications 528 and perform one or more additional tasks, operations, functions, and/or processes described herein. It should be noted that thememory 506 represents one suitable implementation of such computer-readable media, and alternatively or additionally, theserver 502 could receive and cooperate with external computer-readable media that is realized as a portable or mobile component or application platform, e.g., a portable hard drive, a USB flash drive, an optical disc, or the like. - The
application platform 510 is any sort of software application or other data processing engine that generates thevirtual applications 528 that provide data and/or services to theclient devices 540. In a typical embodiment, theapplication platform 510 gains access to processing resources, communications interfaces and other features of theprocessing hardware 504 using any sort of conventional orproprietary operating system 508. Thevirtual applications 528 are typically generated at run-time in response to input received from theclient devices 540. For the illustrated embodiment, theapplication platform 510 includes a bulkdata processing engine 512, aquery generator 514, asearch engine 516 that provides text indexing and other search functionality, and aruntime application generator 520. Each of these features may be implemented as a separate process or other module, and many equivalent embodiments could include different and/or additional features, components or other modules as desired. - The
runtime application generator 520 dynamically builds and executes thevirtual applications 528 in response to specific requests received from theclient devices 540. Thevirtual applications 528 are typically constructed in accordance with the tenant-specific metadata 538, which describes the particular tables, reports, interfaces and/or other features of theparticular application 528. In various embodiments, eachvirtual application 528 generates dynamic web content that can be served to a browser orother client program 542 associated with itsclient device 540, as appropriate. - The
runtime application generator 520 suitably interacts with thequery generator 514 to efficiently obtainmulti-tenant data 532 from thedatabase 530 as needed in response to input queries initiated or otherwise provided by users of theclient devices 540. In a typical embodiment, thequery generator 514 considers the identity of the user requesting a particular function (along with the user's associated tenant), and then builds and executes queries to thedatabase 530 using system-wide metadata 536, tenantspecific metadata 538, pivot tables 534, and/or any other available resources. Thequery generator 514 in this example therefore maintains security of thecommon database 530 by ensuring that queries are consistent with access privileges granted to the user and/or tenant that initiated the request. In this manner, thequery generator 514 suitably obtains requested subsets ofdata 532 accessible to a user and/or tenant from thedatabase 530 as needed to populate the tables, reports or other features of the particularvirtual application 528 for that user and/or tenant. - Still referring to
FIG. 5 , thedata processing engine 512 performs bulk processing operations on thedata 532 such as uploads or downloads, updates, online transaction processing, and/or the like. In many embodiments, less urgent bulk processing of thedata 532 can be scheduled to occur as processing resources become available, thereby giving priority to more urgent data processing by thequery generator 514, thesearch engine 516, thevirtual applications 528, etc. - In exemplary embodiments, the
application platform 510 is utilized to create and/or generate data-drivenvirtual applications 528 for the tenants that they support. Suchvirtual applications 528 may make use of interface features such as custom (or tenant-specific)screens 524, standard (or universal) screens 522 or the like. Any number of custom and/orstandard objects 526 may also be available for integration into tenant-developedvirtual applications 528. As used herein, “custom” should be understood as meaning that a respective object or application is tenant-specific (e.g., only available to users associated with a particular tenant in the multi-tenant system) or user-specific (e.g., only available to a particular subset of users within the multi-tenant system), whereas “standard” or “universal” applications or objects are available across multiple tenants in the multi-tenant system. For example, a virtual CRM application may utilizestandard objects 526 such as “account” objects, “opportunity” objects, “contact” objects, or the like. Thedata 532 associated with eachvirtual application 528 is provided to thedatabase 530, as appropriate, and stored until it is requested or is otherwise needed, along with themetadata 538 that describes the particular features (e.g., reports, tables, functions, objects, fields, formulas, code, etc.) of that particularvirtual application 528. For example, avirtual application 528 may include a number ofobjects 526 accessible to a tenant, wherein for eachobject 526 accessible to the tenant, information pertaining to its object type along with values for various fields associated with that respective object type are maintained asmetadata 538 in thedatabase 530. In this regard, the object type defines the structure (e.g., the formatting, functions and other constructs) of eachrespective object 526 and the various fields associated therewith. - Still referring to
FIG. 5 , the data and services provided by theserver 502 can be retrieved using any sort of personal computer, mobile telephone, tablet or other network-enabledclient device 540 on thenetwork 545. In an exemplary embodiment, theclient device 540 includes a display device, such as a monitor, screen, or another conventional electronic display capable of graphically presenting data and/or information retrieved from themulti-tenant database 530. Typically, the user operates a conventional browser application or other client program 542 (e.g., client application 107) executed by theclient device 540 to contact theserver 502 via thenetwork 545 using a networking protocol, such as the hypertext transport protocol (HTTP) or the like. The user typically authenticates his or her identity to theserver 502 to obtain a session identifier (“SessionID”) that identifies the user in subsequent communications with theserver 502. When the identified user requests access to avirtual application 528, theruntime application generator 520 suitably creates the application at run time based upon themetadata 538, as appropriate. As noted above, thevirtual application 528 may contain Java, ActiveX, or other content that can be presented using conventional client software running on theclient device 540; other embodiments may simply provide dynamic web or other content that can be presented and viewed by the user, as desired. - The foregoing description is merely illustrative in nature and is not intended to limit the embodiments of the subject matter or the application and uses of such embodiments. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the technical field, background, or the detailed description. As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations, and the exemplary embodiments described herein are not intended to limit the scope or applicability of the subject matter in any way.
- For the sake of brevity, conventional techniques related to querying and other database functions, software development, testing, debugging, multi-tenancy, cloud computing, on-demand applications, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. In addition, those skilled in the art will appreciate that embodiments may be practiced in conjunction with any number of system and/or network architectures, data transmission protocols, and device configurations, and that the system described herein is merely one suitable example. Furthermore, certain terminology may be used herein for the purpose of reference only, and thus is not intended to be limiting. For example, the terms “first,” “second” and other such numerical terms do not imply a sequence or order unless clearly indicated by the context.
- Embodiments of the subject matter may be described herein in terms of functional and/or logical block components, and with reference to symbolic representations of operations, processing tasks, and functions that may be performed by various computing components or devices. Such operations, tasks, and functions are sometimes referred to as being computer-executed, computerized, software-implemented, or computer-implemented. In practice, one or more processing systems or devices can carry out the described operations, tasks, and functions by manipulating electrical signals representing data bits at accessible memory locations, as well as other processing of signals. The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties corresponding to the data bits. It should be appreciated that the various block components shown in the figures may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment of a system or a component may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. When implemented in software or firmware, various elements of the systems described herein are essentially the code segments or instructions that perform the various tasks. The program or code segments can be stored in a processor-readable medium or transmitted by a computer data signal embodied in a carrier wave over a transmission medium or communication path. The “processor-readable medium” or “machine-readable medium” may include any non-transitory medium that can store or transfer information. Examples of the processor-readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, or the like. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic paths, or RF links. The code segments may be downloaded via computer networks such as the Internet, an intranet, a LAN, or the like. In this regard, the subject matter described herein can be implemented in the context of any computer-implemented system and/or in connection with two or more separate and distinct computer-implemented systems that cooperate and communicate with one another. In one or more exemplary embodiments, the subject matter described herein is implemented in conjunction with a virtual customer relationship management (CRM) application in a multi-tenant environment.
- While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or embodiments described herein are not intended to limit the scope, applicability, or configuration of the claimed subject matter in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the described embodiment or embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope defined by the claims, which includes known equivalents and foreseeable equivalents at the time of filing this patent application. Accordingly, details of the exemplary embodiments or other limitations described above should not be read into the claims absent a clear intention to the contrary.
Claims (21)
1. A method of detecting anomalies with respect to an update to executable code for an application platform, the method comprising:
obtaining log data associated with a plurality of tests executed with respect to a compiled version of the executable code for the application platform;
filtering the log data based on one or more performance metrics associated with the plurality of tests to obtain reference log data;
converting the reference log data to a corresponding numerical representation;
generating a matrix of the numerical representation of the reference log data;
obtaining test log data associated with one or more tests executed against an updated compilation of the executable code including the update; and
for each line of the test log data:
converting the respective line of the test log data into a respective numerical representation;
determining a difference between the respective numerical representation and the matrix; and
providing an indication of an anomaly associated with the respective line of the test log data when the difference is greater than a detection threshold.
2. The method of claim 1 , wherein obtaining the test log data comprises obtaining a test results log file associated with the updated compilation of the executable code from a testing system.
3. The method of claim 2 , wherein:
converting the respective line of the test log data into the respective numerical representation comprises converting a log line of the test results log file into a numerical vector representation; and
determining the difference comprises calculating a value of a dot product between the numerical vector representation and the matrix.
4. The method of claim 3 , wherein providing the indication comprises providing a graphical representation of the log line and the value of the dot product when the value of the dot product is greater than the detection threshold.
5. The method of claim 4 , wherein:
obtaining the log data comprises obtaining a plurality of test result log files associated with the plurality of tests previously executed with respect to compiled versions of the executable code for the application platform prior to the update;
filtering the log data comprises identifying a reference subset of the plurality of test result log files having respective values for the one or more performance metrics satisfying one or more selection thresholds; and
the reference log data comprises the reference subset of test result log files.
6. The method of claim 5 , further comprising filtering nondescript information from the reference subset of test result log files to obtain filtered reference test result log files, wherein:
converting the reference log data to the corresponding numerical representation comprises converting remaining content of each respective log line of each respective one of the filtered reference test result log files to a corresponding sequence of numerical values associated with the respective log line of the respective filtered reference test result log file.
7. The method of claim 6 , wherein:
generating the matrix comprises constructing a multidimensional matrix configured to maintain the corresponding sequence of numerical values in association with the respective log line of the respective filtered reference test result log file; and
calculating the value comprises calculating the value of the dot product between the numerical vector representation and the multidimensional matrix.
8. The method of claim 1 , wherein:
obtaining the log data comprises obtaining a plurality of test result log files associated with the plurality of tests previously executed with respect to compiled versions of the executable code for the application platform prior to the update;
filtering the log data comprises identifying a reference subset of the plurality of test result log files having respective values for the one or more performance metrics satisfying one or more selection thresholds; and
the reference log data comprises the reference subset of test result log files.
9. The method of claim 8 , further comprising filtering nondescript information from the reference subset of test result log files to obtain filtered reference test result log files, wherein:
converting the reference log data to the corresponding numerical representation comprises converting remaining content of each respective log line of each respective one of the filtered reference test result log files to a corresponding sequence of numerical values associated with the respective log line of the respective filtered reference test result log file.
10. The method of claim 9 , wherein generating the matrix comprises constructing a multidimensional matrix configured to maintain the corresponding sequence of numerical values in association with the respective log line of the respective filtered reference test result log file.
11. The method of claim 1 , wherein:
determining the difference comprises determining a value of a dot product of the respective numerical representation and the matrix; and
providing the indication comprises providing the indication when the value of the dot product is greater than the detection threshold.
12. The method of claim 1 , wherein the application platform is capable of providing instances of a virtual application generated using data maintained in a database.
13. The method of claim 1 , further comprising generating a file comprising an anomalous subset of lines of the test log data, wherein the difference associated with each respective line of the anomalous subset is greater than the detection threshold.
14. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing system, cause the processing system to:
obtain log data associated with a plurality of tests executed with respect to a compiled version of executable code for an application platform;
filter the log data based on one or more performance metrics associated with the plurality of tests to obtain reference log data;
convert the reference log data to a corresponding numerical representation;
generate a matrix of the numerical representation of the reference log data;
obtain test log data associated with one or more tests executed against an updated compilation of the executable code including an update to the executable code; and
for each line of the test log data:
convert the respective line of the test log data into a respective numerical representation;
determine a difference between the respective numerical representation and the matrix; and
provide an indication of an anomaly associated with the respective line of the test log data when the difference is greater than a detection threshold.
15-20. (canceled)
21. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the processing system to obtain the test log data by obtaining a test results log file associated with the updated compilation of the executable code from a testing system.
22. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processing system to:
convert the respective line of the test log data into the respective numerical representation by converting a log line of the test results log file into a numerical vector representation; and
determine the difference by calculating a value of a dot product between the numerical vector representation and the matrix.
23. The non-transitory computer-readable medium of claim 14 , wherein:
the log data comprises a plurality of test result log files associated with the plurality of tests previously executed with respect to compiled versions of the executable code for the application platform prior to the update; and
the reference log data comprises a reference subset of the plurality of test result log files having respective values for the one or more performance metrics satisfying one or more selection thresholds.
24. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the processing system to:
determine the difference by determining a value of a dot product of the respective numerical representation and the matrix; and
provide the indication by providing the indication when the value of the dot product is greater than the detection threshold.
25. The non-transitory computer-readable medium of claim 14 , wherein the instructions cause the processing system to generate a file comprising an anomalous subset of lines of the test log data, wherein the difference associated with each respective line of the anomalous subset is greater than the detection threshold.
26. A system comprising:
a data storage element; and
one or more computing devices coupled to the data storage element and a network to:
obtain log data associated with a plurality of tests executed with respect to a compiled version of executable code for an application platform via the network;
filter the log data based on one or more performance metrics associated with the plurality of tests to obtain reference log data;
convert the reference log data to a corresponding numerical representation;
generate a matrix of the numerical representation of the reference log data;
obtain test log data associated with one or more tests executed against an updated compilation of the executable code including an update to the executable code; and
for each line of the test log data:
convert the respective line of the test log data into a respective numerical representation;
determine a difference between the respective numerical representation and the matrix; and
provide an indication of an anomaly associated with the respective line of the test log data when the difference is greater than a detection threshold.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/929,476 US11163731B1 (en) | 2020-05-04 | 2020-05-04 | Autobuild log anomaly detection methods and systems |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/929,476 US11163731B1 (en) | 2020-05-04 | 2020-05-04 | Autobuild log anomaly detection methods and systems |
Publications (2)
Publication Number | Publication Date |
---|---|
US11163731B1 US11163731B1 (en) | 2021-11-02 |
US20210342313A1 true US20210342313A1 (en) | 2021-11-04 |
Family
ID=78292928
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/929,476 Active US11163731B1 (en) | 2020-05-04 | 2020-05-04 | Autobuild log anomaly detection methods and systems |
Country Status (1)
Country | Link |
---|---|
US (1) | US11163731B1 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11403326B2 (en) | 2020-12-03 | 2022-08-02 | International Business Machines Corporation | Message-based event grouping for a computing operation |
US11474892B2 (en) | 2020-12-03 | 2022-10-18 | International Business Machines Corporation | Graph-based log sequence anomaly detection and problem diagnosis |
US11513930B2 (en) | 2020-12-03 | 2022-11-29 | International Business Machines Corporation | Log-based status modeling and problem diagnosis for distributed applications |
US11599404B2 (en) * | 2020-12-03 | 2023-03-07 | International Business Machines Corporation | Correlation-based multi-source problem diagnosis |
US11797538B2 (en) | 2020-12-03 | 2023-10-24 | International Business Machines Corporation | Message correlation extraction for mainframe operation |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220214948A1 (en) * | 2021-01-06 | 2022-07-07 | Kyndryl, Inc. | Unsupervised log data anomaly detection |
US11743150B2 (en) * | 2021-05-21 | 2023-08-29 | Istreamplanet Co., Llc | Automated root cause analysis of underperforming video streams by using language transformers on support ticket systems |
US11921603B2 (en) * | 2021-08-05 | 2024-03-05 | Microsoft Technology Licensing, Llc | Automated interoperational tracking in computing systems |
US11513927B1 (en) * | 2022-01-25 | 2022-11-29 | Dell Products L.P. | Method and system for performing testing operations for information handling systems |
US20230297492A1 (en) * | 2022-03-19 | 2023-09-21 | Wipro Limited | Method and system for determining optimal event log set for evaluating behaviour of software-based systems |
US11914976B2 (en) | 2022-04-29 | 2024-02-27 | Bank Of America Corporation | Three-dimensional software code review engine |
CN115065611B (en) * | 2022-06-08 | 2023-09-26 | 上海万向区块链股份公司 | Monitoring and verifying method and system for data security multiparty collaborative calculation |
US11966721B2 (en) | 2022-07-12 | 2024-04-23 | Bank Of America Corporation | Extended reality platform and machine learning engine for automated code error resolution |
CN115658553B (en) * | 2022-12-26 | 2023-03-17 | 西安葡萄城软件有限公司 | Debugging method, system and device based on low-code platform |
Family Cites Families (132)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5649104A (en) | 1993-03-19 | 1997-07-15 | Ncr Corporation | System for allowing user of any computer to draw image over that generated by the host computer and replicating the drawn image to other computers |
US5608872A (en) | 1993-03-19 | 1997-03-04 | Ncr Corporation | System for allowing all remote computers to perform annotation on an image and replicating the annotated image on the respective displays of other comuters |
US7991347B1 (en) | 1994-04-07 | 2011-08-02 | Data Innovation Llc | System and method for accessing set of digital data at a remote site |
US5577188A (en) | 1994-05-31 | 1996-11-19 | Future Labs, Inc. | Method to provide for virtual screen overlay |
US7181758B1 (en) | 1994-07-25 | 2007-02-20 | Data Innovation, L.L.C. | Information distribution and processing system |
GB2300991B (en) | 1995-05-15 | 1997-11-05 | Andrew Macgregor Ritchie | Serving signals to browsing clients |
US5715450A (en) | 1995-09-27 | 1998-02-03 | Siebel Systems, Inc. | Method of selecting and presenting data from a database using a query language to a user of a computer system |
US5821937A (en) | 1996-02-23 | 1998-10-13 | Netsuite Development, L.P. | Computer method for updating a network design |
US5831610A (en) | 1996-02-23 | 1998-11-03 | Netsuite Development L.P. | Designing networks |
US5873096A (en) | 1997-10-08 | 1999-02-16 | Siebel Systems, Inc. | Method of maintaining a network of partially replicated database system |
US6604117B2 (en) | 1996-03-19 | 2003-08-05 | Siebel Systems, Inc. | Method of maintaining a network of partially replicated database system |
AU6183698A (en) | 1997-02-26 | 1998-09-18 | Siebel Systems, Inc. | Method of determining visibility to a remote database client of a plurality of database transactions having variable visibility strengths |
AU6668398A (en) | 1997-02-26 | 1998-09-18 | Siebel Systems, Inc. | Method of determining the visibility to a remote databaseclient of a plurality of database transactions using simplified visibility rules |
WO1998040804A2 (en) | 1997-02-26 | 1998-09-17 | Siebel Systems, Inc. | Distributed relational database |
AU6440398A (en) | 1997-02-26 | 1998-09-18 | Siebel Systems, Inc. | Method of using a cache to determine the visibility to a remote database client of a plurality of database transactions |
WO1998038762A2 (en) | 1997-02-26 | 1998-09-03 | Siebel Systems, Inc. | Determining visibility to a remote database client |
WO1998040805A2 (en) | 1997-02-27 | 1998-09-17 | Siebel Systems, Inc. | Method of synchronizing independently distributed software and database schema |
JP2001514776A (en) | 1997-02-27 | 2001-09-11 | シーベル システムズ,インコーポレイティド | A method of continuous level transport of software distribution incorporating local modifications. |
EP1015997A4 (en) | 1997-02-28 | 2006-03-29 | Siebel Systems Inc | Partially replicated distributed database with multiple levels of remote clients |
US6169534B1 (en) | 1997-06-26 | 2001-01-02 | Upshot.Com | Graphical user interface for customer information management |
US6560461B1 (en) | 1997-08-04 | 2003-05-06 | Mundi Fomukong | Authorized location reporting paging system |
US5918159A (en) | 1997-08-04 | 1999-06-29 | Fomukong; Mundi | Location reporting satellite paging system with optional blocking of location reporting |
US20020059095A1 (en) | 1998-02-26 | 2002-05-16 | Cook Rachael Linette | System and method for generating, capturing, and managing customer lead information over a computer network |
US6732111B2 (en) | 1998-03-03 | 2004-05-04 | Siebel Systems, Inc. | Method, apparatus, system, and program product for attaching files and other objects to a partially replicated database |
US6161149A (en) | 1998-03-13 | 2000-12-12 | Groupserve, Inc. | Centrifugal communication and collaboration method |
US6772229B1 (en) | 2000-11-13 | 2004-08-03 | Groupserve, Inc. | Centrifugal communication and collaboration method |
US5963953A (en) | 1998-03-30 | 1999-10-05 | Siebel Systems, Inc. | Method, and system for product configuration |
EP1105826A1 (en) | 1998-08-27 | 2001-06-13 | Upshot Corporation | A method and apparatus for network-based sales force management |
US6393605B1 (en) | 1998-11-18 | 2002-05-21 | Siebel Systems, Inc. | Apparatus and system for efficient delivery and deployment of an application |
US6728960B1 (en) | 1998-11-18 | 2004-04-27 | Siebel Systems, Inc. | Techniques for managing multiple threads in a browser environment |
US6601087B1 (en) | 1998-11-18 | 2003-07-29 | Webex Communications, Inc. | Instant document sharing |
WO2000033235A1 (en) | 1998-11-30 | 2000-06-08 | Siebel Systems, Inc. | State models for monitoring processes |
JP2002531896A (en) | 1998-11-30 | 2002-09-24 | シーベル システムズ,インコーポレイティド | Call center using smart script |
WO2000033238A2 (en) | 1998-11-30 | 2000-06-08 | Siebel Systems, Inc. | Assignment manager |
AU2035600A (en) | 1998-11-30 | 2000-06-19 | Siebel Systems, Inc. | Development tool, method, and system for client server appications |
US7356482B2 (en) | 1998-12-18 | 2008-04-08 | Alternative Systems, Inc. | Integrated change management unit |
US20020072951A1 (en) | 1999-03-03 | 2002-06-13 | Michael Lee | Marketing support database management method, system and program product |
US6574635B2 (en) | 1999-03-03 | 2003-06-03 | Siebel Systems, Inc. | Application instantiation based upon attributes and values stored in a meta data repository, including tiering of application layers objects and components |
US7698160B2 (en) | 1999-05-07 | 2010-04-13 | Virtualagility, Inc | System for performing collaborative tasks |
US8095413B1 (en) | 1999-05-07 | 2012-01-10 | VirtualAgility, Inc. | Processing management information |
US6621834B1 (en) | 1999-11-05 | 2003-09-16 | Raindance Communications, Inc. | System and method for voice transmission over network protocols |
US6535909B1 (en) | 1999-11-18 | 2003-03-18 | Contigo Software, Inc. | System and method for record and playback of collaborative Web browsing session |
US6324568B1 (en) | 1999-11-30 | 2001-11-27 | Siebel Systems, Inc. | Method and system for distributing objects over a network |
US6654032B1 (en) | 1999-12-23 | 2003-11-25 | Webex Communications, Inc. | Instant sharing of documents on a remote server |
US7266502B2 (en) | 2000-03-31 | 2007-09-04 | Siebel Systems, Inc. | Feature centric release manager method and system |
US6577726B1 (en) | 2000-03-31 | 2003-06-10 | Siebel Systems, Inc. | Computer telephony integration hotelling method and system |
US6336137B1 (en) | 2000-03-31 | 2002-01-01 | Siebel Systems, Inc. | Web client-server system and method for incompatible page markup and presentation languages |
US6732100B1 (en) | 2000-03-31 | 2004-05-04 | Siebel Systems, Inc. | Database access method and system for user role defined access |
US6665655B1 (en) | 2000-04-14 | 2003-12-16 | Rightnow Technologies, Inc. | Implicit rating of retrieved information in an information search system |
US7730072B2 (en) | 2000-04-14 | 2010-06-01 | Rightnow Technologies, Inc. | Automated adaptive classification system for knowledge networks |
US6842748B1 (en) | 2000-04-14 | 2005-01-11 | Rightnow Technologies, Inc. | Usage based strength between related information in an information retrieval system |
US6434550B1 (en) | 2000-04-14 | 2002-08-13 | Rightnow Technologies, Inc. | Temporal updates of relevancy rating of retrieved information in an information search system |
US6763501B1 (en) | 2000-06-09 | 2004-07-13 | Webex Communications, Inc. | Remote document serving |
US7069231B1 (en) | 2000-07-20 | 2006-06-27 | Oracle International Corporation | Methods and systems for defining, applying and executing customer care relationship plans |
KR100365357B1 (en) | 2000-10-11 | 2002-12-18 | 엘지전자 주식회사 | Method for data communication of mobile terminal |
US7581230B2 (en) | 2001-02-06 | 2009-08-25 | Siebel Systems, Inc. | Adaptive communication application programming interface |
USD454139S1 (en) | 2001-02-20 | 2002-03-05 | Rightnow Technologies | Display screen for a computer |
US7363388B2 (en) | 2001-03-28 | 2008-04-22 | Siebel Systems, Inc. | Method and system for direct server synchronization with a computing device |
US6829655B1 (en) | 2001-03-28 | 2004-12-07 | Siebel Systems, Inc. | Method and system for server synchronization with a computing device via a companion device |
US7174514B2 (en) | 2001-03-28 | 2007-02-06 | Siebel Systems, Inc. | Engine to present a user interface based on a logical structure, such as one for a customer relationship management system, across a web site |
US20030206192A1 (en) | 2001-03-31 | 2003-11-06 | Mingte Chen | Asynchronous message push to web browser |
US20030018705A1 (en) | 2001-03-31 | 2003-01-23 | Mingte Chen | Media-independent communication server |
US6732095B1 (en) | 2001-04-13 | 2004-05-04 | Siebel Systems, Inc. | Method and apparatus for mapping between XML and relational representations |
US7761288B2 (en) | 2001-04-30 | 2010-07-20 | Siebel Systems, Inc. | Polylingual simultaneous shipping of software |
US6728702B1 (en) | 2001-06-18 | 2004-04-27 | Siebel Systems, Inc. | System and method to implement an integrated search center supporting a full-text search and query on a database |
US6763351B1 (en) | 2001-06-18 | 2004-07-13 | Siebel Systems, Inc. | Method, apparatus, and system for attaching search results |
US6782383B2 (en) | 2001-06-18 | 2004-08-24 | Siebel Systems, Inc. | System and method to implement a persistent and dismissible search center frame |
US6711565B1 (en) | 2001-06-18 | 2004-03-23 | Siebel Systems, Inc. | Method, apparatus, and system for previewing search results |
US20030004971A1 (en) | 2001-06-29 | 2003-01-02 | Gong Wen G. | Automatic generation of data models and accompanying user interfaces |
US6993712B2 (en) | 2001-09-28 | 2006-01-31 | Siebel Systems, Inc. | System and method for facilitating user interaction in a browser environment |
US6826582B1 (en) | 2001-09-28 | 2004-11-30 | Emc Corporation | Method and system for using file systems for content management |
US6978445B2 (en) | 2001-09-28 | 2005-12-20 | Siebel Systems, Inc. | Method and system for supporting user navigation in a browser environment |
US7761535B2 (en) | 2001-09-28 | 2010-07-20 | Siebel Systems, Inc. | Method and system for server synchronization with a computing device |
US6724399B1 (en) | 2001-09-28 | 2004-04-20 | Siebel Systems, Inc. | Methods and apparatus for enabling keyboard accelerators in applications implemented via a browser |
US6901595B2 (en) | 2001-09-29 | 2005-05-31 | Siebel Systems, Inc. | Method, apparatus, and system for implementing a framework to support a web-based application |
US7146617B2 (en) | 2001-09-29 | 2006-12-05 | Siebel Systems, Inc. | Method, apparatus, and system for implementing view caching in a framework to support web-based applications |
US8359335B2 (en) | 2001-09-29 | 2013-01-22 | Siebel Systems, Inc. | Computing system and method to implicitly commit unsaved data for a world wide web application |
US7962565B2 (en) | 2001-09-29 | 2011-06-14 | Siebel Systems, Inc. | Method, apparatus and system for a mobile web client |
US7289949B2 (en) | 2001-10-09 | 2007-10-30 | Right Now Technologies, Inc. | Method for routing electronic correspondence based on the level and type of emotion contained therein |
US7062502B1 (en) | 2001-12-28 | 2006-06-13 | Kesler John N | Automated generation of dynamic data entry user interface for relational database management systems |
US6804330B1 (en) | 2002-01-04 | 2004-10-12 | Siebel Systems, Inc. | Method and system for accessing CRM data via voice |
US7058890B2 (en) | 2002-02-13 | 2006-06-06 | Siebel Systems, Inc. | Method and system for enabling connectivity to a data system |
US7672853B2 (en) | 2002-03-29 | 2010-03-02 | Siebel Systems, Inc. | User interface for processing requests for approval |
US7131071B2 (en) | 2002-03-29 | 2006-10-31 | Siebel Systems, Inc. | Defining an approval process for requests for approval |
US6850949B2 (en) | 2002-06-03 | 2005-02-01 | Right Now Technologies, Inc. | System and method for generating a dynamic interface via a communications network |
US7437720B2 (en) | 2002-06-27 | 2008-10-14 | Siebel Systems, Inc. | Efficient high-interactivity user interface for client-server applications |
US7594181B2 (en) | 2002-06-27 | 2009-09-22 | Siebel Systems, Inc. | Prototyping graphical user interfaces |
US8639542B2 (en) | 2002-06-27 | 2014-01-28 | Siebel Systems, Inc. | Method and apparatus to facilitate development of a customer-specific business process model |
US20040010489A1 (en) | 2002-07-12 | 2004-01-15 | Rightnow Technologies, Inc. | Method for providing search-specific web pages in a network computing environment |
US7251787B2 (en) | 2002-08-28 | 2007-07-31 | Siebel Systems, Inc. | Method and apparatus for an integrated process modeller |
US9448860B2 (en) | 2003-03-21 | 2016-09-20 | Oracle America, Inc. | Method and architecture for providing data-change alerts to external applications via a push service |
US7904340B2 (en) | 2003-03-24 | 2011-03-08 | Siebel Systems, Inc. | Methods and computer-readable medium for defining a product model |
WO2004086198A2 (en) | 2003-03-24 | 2004-10-07 | Siebel Systems, Inc. | Common common object |
WO2004086197A2 (en) | 2003-03-24 | 2004-10-07 | Siebel Systems, Inc. | Custom common object |
US8762415B2 (en) | 2003-03-25 | 2014-06-24 | Siebel Systems, Inc. | Modeling of order data |
US7685515B2 (en) | 2003-04-04 | 2010-03-23 | Netsuite, Inc. | Facilitating data manipulation in a browser-based user interface of an enterprise business application |
US7412455B2 (en) | 2003-04-30 | 2008-08-12 | Dillon David M | Software framework that facilitates design and implementation of database applications |
US7620655B2 (en) | 2003-05-07 | 2009-11-17 | Enecto Ab | Method, device and computer program product for identifying visitors of websites |
US7409336B2 (en) | 2003-06-19 | 2008-08-05 | Siebel Systems, Inc. | Method and system for searching data based on identified subset of categories and relevance-scored text representation-category combinations |
US20040260659A1 (en) | 2003-06-23 | 2004-12-23 | Len Chan | Function space reservation system |
US7237227B2 (en) | 2003-06-30 | 2007-06-26 | Siebel Systems, Inc. | Application user interface template with free-form layout |
US7694314B2 (en) | 2003-08-28 | 2010-04-06 | Siebel Systems, Inc. | Universal application network architecture |
US7441246B2 (en) * | 2004-03-19 | 2008-10-21 | Microsoft Corporation | Configurable collection of computer related metric data |
US8607322B2 (en) | 2004-07-21 | 2013-12-10 | International Business Machines Corporation | Method and system for federated provisioning |
US7289976B2 (en) | 2004-12-23 | 2007-10-30 | Microsoft Corporation | Easy-to-use data report specification |
US8209308B2 (en) | 2006-05-01 | 2012-06-26 | Rueben Steven L | Method for presentation of revisions of an electronic document |
US9135228B2 (en) | 2006-05-01 | 2015-09-15 | Domo, Inc. | Presentation of document history in a web browsing application |
US8566301B2 (en) | 2006-05-01 | 2013-10-22 | Steven L. Rueben | Document revisions in a collaborative computing environment |
US7779475B2 (en) | 2006-07-31 | 2010-08-17 | Petnote Llc | Software-based method for gaining privacy by affecting the screen of a computing device |
US7730478B2 (en) | 2006-10-04 | 2010-06-01 | Salesforce.Com, Inc. | Method and system for allowing access to developed applications via a multi-tenant on-demand database service |
US8082301B2 (en) | 2006-11-10 | 2011-12-20 | Virtual Agility, Inc. | System for supporting collaborative activity |
US8954500B2 (en) | 2008-01-04 | 2015-02-10 | Yahoo! Inc. | Identifying and employing social network relationships |
US8583615B2 (en) | 2007-08-31 | 2013-11-12 | Yahoo! Inc. | System and method for generating a playlist from a mood gradient |
US20090100342A1 (en) | 2007-10-12 | 2009-04-16 | Gabriel Jakobson | Method and system for presenting address and mapping information |
US8504945B2 (en) | 2008-02-01 | 2013-08-06 | Gabriel Jakobson | Method and system for associating content with map zoom function |
US8490025B2 (en) | 2008-02-01 | 2013-07-16 | Gabriel Jakobson | Displaying content associated with electronic mapping systems |
US8014943B2 (en) | 2008-05-08 | 2011-09-06 | Gabriel Jakobson | Method and system for displaying social networking navigation information |
US8032297B2 (en) | 2008-05-08 | 2011-10-04 | Gabriel Jakobson | Method and system for displaying navigation information on an electronic map |
CA2665669A1 (en) | 2008-05-09 | 2009-11-09 | Research In Motion Limited | Method for cell selection in a radio access network |
US8646103B2 (en) | 2008-06-30 | 2014-02-04 | Gabriel Jakobson | Method and system for securing online identities |
US8510664B2 (en) | 2008-09-06 | 2013-08-13 | Steven L. Rueben | Method and system for displaying email thread information |
US8661056B1 (en) | 2008-11-03 | 2014-02-25 | Salesforce.Com, Inc. | System, method and computer program product for publicly providing web content of a tenant using a multi-tenant on-demand database service |
US8510045B2 (en) | 2009-12-22 | 2013-08-13 | Steven L. Rueben | Digital maps displaying search-resulting points-of-interest in user delimited regions |
US8925041B2 (en) | 2010-04-01 | 2014-12-30 | Salesforce.Com, Inc. | System, method and computer program product for performing one or more actions based on a determined access permissions for a plurality of users |
US8566654B2 (en) | 2010-08-13 | 2013-10-22 | Salesforce.Com, Inc. | Debugging site errors by an admin as a guest user in a multi-tenant database environment |
US8943002B2 (en) | 2012-02-10 | 2015-01-27 | Liveperson, Inc. | Analytics driven engagement |
US8756275B2 (en) | 2012-02-17 | 2014-06-17 | Zebedo | Variable speed collaborative web browsing system |
US8769004B2 (en) | 2012-02-17 | 2014-07-01 | Zebedo | Collaborative web browsing system integrated with social networks |
US8769017B2 (en) | 2012-02-17 | 2014-07-01 | Zebedo | Collaborative web browsing system having document object model element interaction detection |
US9507695B2 (en) * | 2014-04-14 | 2016-11-29 | International Business Machines Corporation | Risk-based test coverage and prioritization |
US9916224B2 (en) * | 2015-09-15 | 2018-03-13 | Linkedin Corporation | Integrating quality analysis with a code review tool |
US9955488B2 (en) * | 2016-03-31 | 2018-04-24 | Verizon Patent And Licensing Inc. | Modeling network performance and service quality in wireless networks |
-
2020
- 2020-05-04 US US15/929,476 patent/US11163731B1/en active Active
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11403326B2 (en) | 2020-12-03 | 2022-08-02 | International Business Machines Corporation | Message-based event grouping for a computing operation |
US11474892B2 (en) | 2020-12-03 | 2022-10-18 | International Business Machines Corporation | Graph-based log sequence anomaly detection and problem diagnosis |
US11513930B2 (en) | 2020-12-03 | 2022-11-29 | International Business Machines Corporation | Log-based status modeling and problem diagnosis for distributed applications |
US11599404B2 (en) * | 2020-12-03 | 2023-03-07 | International Business Machines Corporation | Correlation-based multi-source problem diagnosis |
US11797538B2 (en) | 2020-12-03 | 2023-10-24 | International Business Machines Corporation | Message correlation extraction for mainframe operation |
Also Published As
Publication number | Publication date |
---|---|
US11163731B1 (en) | 2021-11-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11163731B1 (en) | Autobuild log anomaly detection methods and systems | |
US11099973B2 (en) | Automated test case management systems and methods | |
US10803029B2 (en) | Generating javascript object notation (JSON) schema from JSON payloads | |
US10061578B2 (en) | System and method of configuring a data store for tracking and auditing real-time events across different software development tools in agile development environments | |
US11663033B2 (en) | Design-time information based on run-time artifacts in a distributed computing cluster | |
US10261890B2 (en) | Methods and systems for executing tests using grouped/filtered test classes during testing of an application | |
US8468391B2 (en) | Utilizing log event ontology to deliver user role specific solutions for problem determination | |
US9298588B2 (en) | Tracing system for application and module tracing | |
US20220156053A1 (en) | Software dependency management | |
US8875120B2 (en) | Methods and apparatus for providing software bug-fix notifications for networked computing systems | |
US20170075794A1 (en) | Methods and systems for computing code coverage using grouped/filtered source classes during testing of an application | |
US20170324803A1 (en) | Automated testing of perceptible web page elements | |
US11599539B2 (en) | Column lineage and metadata propagation | |
US10565089B2 (en) | Identification of code features potentially associated with code behavior | |
US9256520B2 (en) | Using linked data to determine package quality | |
US20150066869A1 (en) | Module Database with Tracing Options | |
US20200249921A1 (en) | Structured development for web application frameworks | |
AU2019253836A1 (en) | Processing data utilizing a corpus | |
US9706005B2 (en) | Providing automatable units for infrastructure support | |
CN104636130A (en) | Method and system for generating event trees | |
US10713070B2 (en) | Systems and methods for capturing and visualizing user interactions across devices | |
US10116512B2 (en) | Service discovery and/or effort estimation in networked computing environments | |
CN112579578A (en) | Metadata-based data quality management method, device and system and server | |
US20220405065A1 (en) | Model Document Creation in Source Code Development Environments using Semantic-aware Detectable Action Impacts | |
US10606577B1 (en) | System and method for assuring customers during software deployment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SALESFORCE.COM, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:RIDDELL, KEVIN;REEL/FRAME:052565/0572 Effective date: 20200501 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |