WO2021206704A1

WO2021206704A1 - Generating reports from applications

Info

Publication number: WO2021206704A1
Application number: PCT/US2020/027191
Authority: WO
Inventors: Mauricio COUTINHO MORAES; Jhonny Marcos Acordi MERTZ; Carlos Alexandre DE ARAUJO LIMA
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2020-04-08
Filing date: 2020-04-08
Publication date: 2021-10-14

Abstract

An example system comprises application engines respectively comprising: code segments and execution location markers to return indicators identifying that a respective code segment has been executed, the application engines respectively associated with: an awareness builder engine to: store and retrieve the indicators; and a report builder engine to: receive a request for a report on execution of the code segments; communicate with a respective awareness builder engine to retrieve the indicators; and return the report including the indicators. The example system further comprises a report aggregator engine to: request and receive reports from report builder engines; aggregate the indicators in the reports; rank the indicators as aggregated; and output an aggregated report indicating a ranking of the indicators as aggregated.

Description

GENERATING REPORTS FROM APPLICATIONS

BACKGROUND

[0001] Validating and/or testing of applications generally occurs prior to deployment into production environments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Reference will now be made, by way of example only, to the accompanying drawings in which:

[0003] Figure 1 is a block diagram of an example system to generate reports from applications.

[0004] Figure 2 a block diagram of another example system to generate reports from applications.

[0005] Figure 3 is a flow diagram of an example method to generate reports from applications.

[0006] Figure 4 is a block diagram of an example computer-readable medium including instructions that causes a processor to generate reports from applications.

[0007] Figure 5 is an example of code segments and execution location markers.

[0008] Figure 6 is an example of indicators identifying that a respective code segment has been executed being generated in the example system of Figure 2.

[0009] Figure 7 is an example of reports being requested in the example system of Figure 2.

[0010] Figure 8 is an example of reports being generated in the example system of Figure 2.

[0011] Figure 9 is an example of aggregated reports being generated in the example system of Figure 2. DETAILED DESCRIPTION

[0012] Validating deployed applications in production environments is challenging with applications generally being tested prior to deployment, which may not give an indication of their behavior in the production environments (e.g. in real-time). For example, identification of behavior of an application, for example an application that corresponds to a website, may occur by running the application offline and testing the offline application under certain conditions. However, such an approach may be inefficient and may not represent behavior of the application in real-time and/or in production environments

[0013] As described herein, an application may be made “self-aware” by providing the application with execution location markers which return indicators identifying that respective code segments has been executed. For example, the execution location markers may return line numbers of an execution location marker when executed, which may indicate that code segments associated with execution location marker were executed (e.g. code segments in a same branch of code of the application as the execution location marker). Similarly, not receiving an indicator for an execution location marker (e.g. in branch of the application that is “supposed” to run) may indicate a failure of the code segment (e.g. as the execution location marker following a code segment has not executed due to failure of the code segment).

[0014] However, an application may be made self-aware in any suitable manner and/or the execution location markers may return any suitable information which identifies that a respective code segment has been executed. Hence, such self- aware applications may dynamically monitor themselves based on the information returned by the execution location markers, which may be stored in association with the application using an awareness builder engine which, for example, may be component of an application manager engine and/or an application programming interface (API) associated with the application.

[0015] The application is further associated with a report builder engine (e.g. which may also be a component of the application manager engine and/or API) which may be queried by an external report aggregator engine, and which may provide a report of the behavior of the application to the report aggregator engine by communicating with the awareness builder engine to obtain the information returned by the execution location markers. Such a report generally includes indicators identifying that a respective code segment has been executed, such as associated line numbers thereof.

[0016] The report aggregator engine is generally to query many deployments of the application to receive respective reports from the applications and/or report builder engines thereof, and aggregate and rank the indicators in the reports to generate and output an aggregated report. The report aggregator engine may further communicate with a source code manager engine to receive information identifying issues and/or behaviors associated with respective code segments of the applications, and the ranking of the indicators may be used to generate importance scores for the issues and/or behaviors, and/or rank the issues and/or behaviors.

[0017] An aspect of the present specification provides a system comprising: application engines respectively comprising: code segments and execution location markers to return indicators identifying that a respective code segment has been executed, the application engines respectively associated with: an awareness builder engine to: store and retrieve the indicators; and a report builder engine to: receive a request for a report on execution of the code segments; communicate with a respective awareness builder engine to retrieve the indicators; and return the report including the indicators; and a report aggregator engine to: request and receive reports from report builder engines; aggregate the indicators in the reports; rank the indicators as aggregated; and output an aggregated report indicating a ranking of the indicators as aggregated.

[0018] Another aspect of the present specification provides a method comprising: receiving, by a computing device, reports from applications including indicators identifying respective code segments that have been executed at the applications; determining, by the computing device, a ranking of the indicators; receiving, by the computing device, from a source code manager computing device, information identifying issues associated with the respective code segments; determining, at the computing device, importance scores of the issues based on the ranking of the indicators; and outputting an aggregated report indicating the importance scores of the issues.

[0019] Another aspect of the present specification provides a non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to: execute application modules comprising code segments and execution location markers which return indicators identifying respective code segments that have been executed; execute awareness builder modules to: store and retrieve the indicators; execute report builder modules to: receive a request for a report on execution of the code segments; communicate with a respective awareness builder module to retrieve the indicators; and return the report including the indicators; execute a report receiver module to: receive reports from the report builder modules including the indicators identifying the respective code segments that have been executed at the application modules; execute a behavior determining module to: determine the respective code segments associated with given behaviors of the application modules, the given behaviors including respective given code segments of the application modules; execute a ranking module to: rank the given behaviors based on respective numbers of times the respective given code segments were executed by the application modules as indicated by the reports; and execute an output module to: output an aggregated report indicating the ranking of the given behaviors.

[0020] Figure 1 is a block diagram of an example system 100 to generate reports from applications. The system 100 may include application engines 101- 1....101-N, interchangeably referred to herein, collectively, as the applications engines 101 and, generically, as an application engine 101. As depicted there are an integer number “N” of application engines 101 , representing “N” deployments of a respective application associated with the application engines 101 , for example in a container-orchestrated environment and/or any other suitable environment. Furthermore, the number “N” of the application engines 101 may change, for example as demand for the associated application changes. While not depicted, the system 100 may include an engine (e.g. a container-orchestrated environment engine) to manage the number “N” of the application engines 101 as the demand changes. However, the number “N” of the application engines 101 may be managed in any suitable manner.

[0021] As used herein, the term “engine” refers to hardware (e.g., a processor, such as a central processing unit (CPU) an integrated circuit or other circuitry) or a combination of hardware and software (e.g., programming such as machine- or processor-executable instructions, commands, or code such as firmware, a device driver, programming, object code, etc. as stored on hardware). Hardware includes a hardware element with no software elements such as an application specific integrated circuit (ASIC), a Field Programmable Gate Array (FPGA), etc. A combination of hardware and software includes software hosted at hardware (e.g., a software module that is stored at a processor-readable memory such as random access memory (RAM), a hard disk or solid-state drive, resistive memory, or optical media such as a digital versatile disc (DVD), and/or implemented or interpreted by a processor), or hardware and software hosted at hardware.

[0022] For example, an application engine 101 may comprise hardware or a combination of software and hardware for implementing an application, such as a website, and the like. In some examples, an application engine 101 may comprise a portion of a server which hosts the application and/or the website. However, an application engine 101 may comprise hardware or a combination of software and hardware which correspond to any suitable application.

[0023] While details of only the application engine 101-1 are depicted, it is understood that the other application engines 101 are generally similar and/or the same as the application engine 101-1 (including the application engine 101- N).

[0024] In particular, as shown by the application engine101-1 , an application engine 101 comprises code segments 103 and execution location markers 105. While not depicted, it is understood that the other application engines 101 (such as the application engine 101-N) also include respective code segments 103 and execution location markers 105

[0025] As will be described in more detail below, the code segments 103 may comprise portions and/or segments of code and/or source code and/or executable instructions, and the like, of an application executed when the application engine 101 is implemented, and the like.

[0026] As will be described in more detail below, the execution location markers 105 may generally comprise functions and/or engines and/or executable instructions, and the like, which return indicators identifying that a respective code segment 103 has been executed, which generally indicate behavior of the code segments 103. While the execution location markers 105 will be described in more detail below, in some examples the indicators returned by the execution location markers 105 may comprise: a line number of an application executed when the application engine 101 is implemented, and the like, at which an execution location marker 105 is located.

[0027] Furthermore, the execution location markers 105 generally return indicators identifying behavior of the code segments 103 in response to receiving a service request, and the like, to process the application engine 101 , for example in a production and/or real-time environment. In particular, it is understood that processing an application engine 101 may include controlling the hardware and/or software of the application engine 101 to implement and/or process an application and/or executable instructions of the application engine 101 which includes the code segments 103.

[0028] For example, a service request may be received via a communications network and the like, for example from a requesting device and/or a client device, and the like. The service request may be to implement and/or process the application engine 101 (and/or an application thereof). The service request may comprise a HyperText Transfer Protocol (HTTP) request transmitted to a server implementing the application engine 101 , for example to implement a website. An application engine 101 receives the service request and may responsively implement the code segments 103 and the execution location markers 105, and return a service response to the requesting device and/or a client device. Hereafter the terms “service request” and “service response” will be used to respectively indicate a request to process an application engine 101 , and a response to the request to process an application engine 101 ; and the terms “request” and “response” will be used to indicate a request for report from a report builder engine, and a response to the request for the report, described in more detail below.

[0029] In some examples, a service request may generally be received via an application manager engine which may comprise an API of the application engine 101 , and a service response may be transmitted to the requesting device via the application manager engine. Put another way, an application manager engine may generally receive the service request and pass the service request to the application engine 101. The application engine 101 may generate a service response to the service request that includes the indicators identifying that a respective code segment 103 has been executed; an awareness builder engine (described below) may remove and store the indicators from the service response, prior to the service response being transmitted to the requesting device via the application manager engine.

[0030] Hence, the execution location markers 105 may return indicators identifying that a respective code segment 103 has been executed, for example at a run-time of the application engine 101 and/or in real-time and/or in a production environment. In particular, the indicators indicate behavior of the application engine 101 and/or the code segments 103 at a run-time and/or in real-time and/or in a production environment.

[0031] As depicted the application engine 101-1 is associated with an awareness builder engine 109-1 and a report builder engine 111-1. Similarly, the application engine 101-N is associated with an awareness builder engine 109-N and a report builder engine 111 -N. The awareness builder engines 109-1...109- N are interchangeably referred to herein, collectively, as the awareness builder engines 109 and, generically, as an awareness builder engine 109. Similarly, the report builder engines 111-1...111 -N are interchangeably referred to herein, collectively, as the report builder engines 111 and, generically, as a report builder engine 111 . In general, the application engines 101 of the system 100 are respectively associated with awareness builder engine 109 and a report builder engine 111 .

[0032] While not depicted, an awareness builder engine 109 and a report builder engine 111 associated with an application engine 101 may be components of an application manager engine associated with the application engine 101 and/or components of an API for the application engine 101. Hence, in general, service requests and requests for reports may both be received via the application manager engine and/or API, but service requests may be ignored by the report builder engine 111 while requests for reports are handled by the report builder engine 111 (e.g. and not an application engine 101 ). Hereafter, operation of an awareness builder engine 109, a report builder engine 111 , code segments 103 and execution location markers 105 will be described with reference to the application engine 101-1 , however it is understood that such operation applies to other deployments of the application engines 101.

[0033] An awareness builder engine 109 (e.g. such as the awareness builder engine 109-1) is to store and retrieve the indicators generated by the execution location markers 105. For example, an awareness builder engine 109 may receive the indicators from the execution location markers 105 in a service response and store the indicators locally, for example at a memory and/or database associated with the awareness builder engine 109 and/or the application engine 101-1 and/or an associated application manager engine. In some examples, the indicators may be included in a header of a service response, generated by the application engine 101 as received from the application engine 101 . For example, a service request may comprise an HTTP request, and a service response may comprise an HTTP response, with the indicators in an HTTP header thereof. However, the indicators may be included in the service response in any suitable manner.

[0034] A report builder engine 111 (e.g. such as the report builder engine 111-1) is generally to receive a request for a report on execution of the code segments 103; communicate with the respective awareness builder engine 109 (e.g. such as the awareness builder engine 109-1) to retrieve the indicators; and return the report including the indicators.

[0035] The indicators and/or the report may comprise any suitable indication identifying that a respective code segment 103 has been executed. In some examples, the indicators may include location indications of respective execution location markers associated with respective code segments 103. For example, such location indications may indicate an associated line number of a code segment 103 within the application engine 101 ; in such examples, the location indications may indicate that a code segment 103 which preceded or followed an associated execution location marker 105 was executed.

[0036] Furthermore, the indicators may include and/or indicate respective numbers of times that respective code segments 103 (e.g. and/or respective execution location markers 105) were executed. Put another way, the indicators may include and/or indicate respective numbers of times that respective code segments 103 (e.g. and/or respective execution location markers 105) were executed for a given time period while the application engine 101-1 is deployed in a production environment. In some examples, the indicators may include a time stamp of when an execution location marker 105, and hence a respective code segment 103, was executed, which can enable filtering of the indicators by time, such that numbers of times that respective code segments 103 (e.g. and/or respective execution location markers 105) were executed may be filtered based on a given time period. However, the numbers of times that respective code segments 103 (e.g. and/or respective execution location markers 105) were executed may comprise a running count.

[0037] As depicted, the system 100 further comprises a report aggregator engine 120, which may comprise a portion of a server which hosts the report aggregator engine 120. The report aggregator engine 120 is to: request and receive reports from the report builder engines 111 ; aggregate the indicators in the reports; rank the indicators as aggregated; and output an aggregated report indicating a ranking of the indicators as aggregated. Hence it is understood that the report aggregator engine 120 is in communication with at least the report builder engines 111 via any suitable combination of wired and/or wireless communication links. Communication links in the present specification are depicted as double-ended arrows between components.

[0038] Hence, for example, the report aggregator engine 120 may request reports from the application engines 101 , for example periodically and/or on demand (e.g. via a user operating the report aggregator engine 120 to request an aggregated report), by transmitting the requests for the reports via the report builder engines 111 , which receive and/or intercept the requests for the reports without sending the requests to the reports to the application engines 101 themselves. The report builder engines 111 generate respective reports, as described above, and transmit the reports to the report aggregator engine 120. The report aggregator engine 120 receives the reports, and aggregates the information in the reports, for example by summing up the numbers of times that respective code segments 103 (e.g. and/or respective execution location markers 105) were executed, for example as represented by line numbers in the indications in the reports.

[0039] Hence, the report aggregator engine 120 may be further to rank the indicators as aggregated based on respective numbers of times respective code segments 103 were executed as indicated by the indicators, as aggregated. For example, first code segments 103 that were executed more than second code segments 103 may be ranked higher than the second code segments 103. Indeed, the ranking may comprise a listing of the code segments 103, and/or associated execution location markers 105, and/or associated line numbers thereof, in an order of a number of times they were respectively executed, and/or the ranking may comprise ordering of the code segments 103, and/or associated execution location markers 105, and/or associated line numbers thereof, according to a number of times they were respectively executed.

[0040] Put another way, the ranking may be based on a number of times that a respective code segment 103 was executed as indicated by the indicators, as aggregated. [0041] In yet further examples, while not depicted in Figure 1 , the report aggregator engine 120 may receive, from a source code manager engine and/or a source code manager computing device (e.g. the source code manager engine may comprise a portion of the source code manager computing device), information identifying issues associated with the respective code segments 103 and/or identify behaviors associated with the respective code segments 103. Indeed, in these examples, the report aggregator engine 120 may request reports from the application engines 101 in response to receiving information identifying issues and/or issues, associated with the respective code segments 103, from a source code manager engine and/or a source code manager computing device.

[0042] For example, a source code manager engine and/or a source code manager computing device may comprise a computing device, and the like, which tracks source code lines (e.g. code segments) which are associated with given features and/or given bugs and/or given bug fixes and/or given technical debts and/or given performance results and/or given security breaches, which may be used to identify issues and/or behaviors associated with the respective code segments 103. For example, an issue may comprise a bug fix, and the like (which may be represented by a “commit” at the source code manager engine and/or a source code manager computing device), associated with respective line numbers associated with the code segments 103 and a behavior may comprise a feature and/or a branch associated with respective line numbers associated with the code segments 103. Indeed, in some examples, an issue and a behavior described herein may be similar, in that each may indicate respective line numbers associated with the code segments 103, however an issue may comprise a specific issue with specific line numbers, such as a bug and/or a bug fix, while behaviors may be more generic and indicate specific line numbers for different behaviors and/or branches of an application engine 101 , as described in more detail below.

[0043] Put another way, information identifying issues, and/or behaviors, associated with the respective code segments 1-3, and the indicators of the reports identifying respective code segments 103 that have been executed at the application engines 101 , may comprise respective line numbers associated the respective code segments 103 (which may include, but are not limited to, line numbers of associated execution location markers 105).

[0044] The report aggregator engine 120 may coordinate the line numbers and/or code segments 103 of the information identifying issues, and/or behaviors, and the line numbers and/or code segments 103 in the reports, and determine importance scores of the issues and/or behaviors based on the ranking of the indicators as aggregated. For example, a particular issue and/or behavior associated with code segments 103 associated with that are higher ranked may be assigned a higher importance score than other issues and/or behaviors associated with code segments 103 associated with that are lower ranked.

[0045] Put another way, the report aggregator engine 120 may determine importance scores of the issues, and/or behaviors, based on the ranking of the indicators by: determining code segments 103 associated with an issue, and/or behavior, as indicated by the information (e.g. received from the source code manager engine and/or computing device); and determining positions (and/or an order) of the code segments 103 associated with the issue, and/or behavior, in the ranking of the indicators.

[0046] Hence, for example, the aggregated report generated and output by the report aggregator engine 120 may indicate: first respective code segments 103 that are most executed in the application engines 101 ; and second respective code segments 103 that are least executed in the application engines 101.

[0047] In yet further examples, the issues and/or behaviors identified by the information received from the source code manager engine and/or computing device may be received with a relevancy ranking. For example a first bug fix (and associated line numbers and/or code segments 103) may have a higher relevancy ranking than a second bug fix (and respective associated line numbers and/or code segments 103). For example, the first bug fix may be more important and/or more relevant due to a level of failure of an application fixed by the first bug fix, as compared to a level of failure of the application fixed by the second bug fix. In some examples, the relevancy ranking may be set manually at the source code manager engine and/or computing device, and/or the relevancy ranking may be set based on a failure code and/or error code associated with a failure. For example, a first bug fix and/or first issue and/or first behavior associated with a “403” “forbidden” error code may be assigned a higher relevancy than a second bug fix and/or second issue and/or second behavior associated with a “400” “bad request” error code, as a 403” “forbidden” error code may indicate a failure of the application, while a “400” “bad request” error code may be associated with a bad/invalid client request.

[0048] Hence, for example, when a ranking of code segments 103, and/or line numbers associated therewith, are the same and/or similar, such that the importance scores of associated issues and/or behaviors are the same and/or similar, a relevancy ranking may be used to increase or decrease the importance scores of the issues and/or behaviors. In particular, an importance score of a first issue and/or behavior may be increased, relative to a second issue and/or behavior, when an associated relevancy ranking of the first issue and/or behavior is higher than a respective relevancy ranking of the second issue and/or behavior.

[0049] Alternatively, issues and/or behaviors may be ranked based on the respective numbers of times that the respective given code segments 103 were executed by the application engines 101 , as indicated by the reports. Hence, for example, when a first issue and/or behavior includes code segments 103 that were executed a larger number of times than respective code segments 103 of a second issue and/or behavior, the first issue and/or behavior may be higher ranked than the second issue and/or behavior.

[0050] Furthermore the identifiers received in the reports may be received with information indicating a deployment of respective application engines 101. For example, the application engines 101 may be deployed geographically which may include, but is not limited to, different application engines 101 processing service requests from specific geographic regions (e.g. even when the different application engines 101 are co-located at a given server and/or group of servers). Similarly, the application engines 101 may be deployed by company, which may include, but is not limited to, different application engines 101 processing service requests from specific companies. As such, the report aggregator engine 120 may be further to: to rank given issues and/or given behaviors for different deployments of the application engines 101. Put another way, the rankings and/or importance scores and/or relevancy scores described herein may be sorted by deployment of associated application engines 101.

[0051] In yet further examples, the report aggregator engine 120 is further to rank given issues and/or given behaviors after waiting for a given time period to allow for the reports to be received for the given time period after booting of the application engines 101 or upgrading of the application engines 101. For example, a booting process and/or an upgrading process, may cause code segments 103 associated with booting and/or upgrading to execute which may not execute thereafter, and/or may execute thereafter, but less frequently than when booting and/or upgrading. As such, the report aggregator engine 120 may wait for the given time period before generating the aggregated report to allow code segments 103 to execute under normal operation of an application engine 101. In particular, if the report aggregator engine 120 does not wait for the given time period, then the code segments 103 associated with booting and/or upgrading may be higher ranked than code segments 103 which execute under normal operation (e.g. non-booting and/or non-upgrading processes).

[0052] However, in some examples, the given issues and/or given behaviors identified in the information from the source code manager engine and/or computing device may be associated with booting and/or upgrading, and hence the report aggregator engine 120 may generate the aggregated report within the given time period.

[0053] As described above, the report aggregator engine 120 generally outputs an aggregated report indicating a ranking of the indicators as aggregated and/or a ranking of given behaviors and/or a ranking of given issues and/or importance scores. Such an output generally includes controlling a physical hardware device to output the aggregated report and/or change a physical state. For example, the report aggregator engine 120 may be further to output the aggregated report using a communication interface and/or a printer and/or a computing device and/or a display device. Alternatively, the report aggregator engine 120 may be further to output the aggregated report by altering a memory to store the aggregated report. However, any suitable output of the aggregated report which causes a physical hardware device to output the aggregated report and/or change a physical state is within the scope of the present specification.

[0054] Attention is next directed to Figure 2 which is a block diagram of another example system 200 to generate reports from applications. The system 200 is substantially similar to the system 100, with like components having like numbers, but in a “200” series rather than a “100” series. For example, the system 200 may include application engines 201-1. .201-N, comprising code segments 203, execution location markers 205, awareness builder engines 209- 1...209-N, report builder engines 211-1...211-N and a report aggregator engine 220, that are respectively similar to the application engines 101 , the code segments 103, the execution location markers 105, the awareness builder engines 109, the report builder engines 111 , and the report aggregator engine 120, as described above.

[0055] The application engines 201-1.. .201-N, are interchangeably referred to herein, collectively, as the applications engines 201 and, generically, as an application engine 201. Similarly, the awareness builder engines 209-1 ...209-N are interchangeably referred to herein, collectively, as the awareness builder engines 209 and, generically, as an awareness builder engine 209. Similarly, the report builder engines 211-1...211-N are interchangeably referred to herein, collectively, as the report builder engines 211 and, generically, as a report builder engine 211.

[0056] In contrast to the system 100, in the system 200, the awareness builder engines 209 and the report builder engines 211 are components of respective application manager engines 230-1. .230-N, as described above. The application manager engines 230-1. .230-N are interchangeably referred to herein, collectively, as the application manager engines 230 and, generically, as an application manager engine 230. As described above, the application manager engines 230 may comprise, and/or be components of, a respective API associated with a respective application engine 201 , and handle and/or manage service requests, service responses, request for reports and/or transmission of reports.

[0057] As depicted, the application manager engines 230-1...230-N comprise respective memories 231-1 ...231-N which may be used by respective awareness builder engines 209 to store the indicators from the execution location markers 205 and from which the indicators may be retrieved to build a report by respective report builder engines 211. The memories 231-1...231-N are interchangeably referred to herein, collectively, as the memories 231 and, generically, as a memory 231.

[0058] In general, the awareness builder engines 209, the report builder engines 211 and the memories 231 may be components of respective application manager engines 230, such as an API for the application engines 201. However, in some examples, a memory 231 may be a components of a respective awareness builder engine 209.

[0059] A memory 231 may include a database and the like, which is local to a respective application manager engines 230. A memory 231 may store the indicators received with a response from the application engine 201. In particular, the memory 231 may store the indicators according to a domain specific language (DSL) format and/or in any other suitable format.

[0060] A report builder engine 211 and a respective awareness builder engine 209 may further be to communicate with each other to process a request for a report received at the report builder engine 211 from the report aggregator engine 220. For example, a memory 231 may be accessible via a respective awareness builder engine 209, for example within the application manager engines 230. When the report builder engine 211 receives a request for a report, the report builder engine 211 may retrieve the indicators stored at the memory 231 (e.g. via the awareness builder engine 209). In some examples, the request for the report may include criteria for searching the indicators stored at the memories 231 which may include, but is not limited to, a given time period during which code segments 203 were executed (e.g. in examples the indicators identifying that a respective code segment has been executed are stored with a time stamp, and the like), particular code segments 203 (e.g. associated line numbers) which were executed, and the like.

[0061] As depicted, the report aggregator engine 220 comprises a communication interface 235 (also referred to herein as the interface 235) which may include any suitable wired or wireless communication interface to communicate with other components of the system 200 via any suitable combination of wired and/or wireless communication links.

[0062] For example, the interface 235 may be used to communicate with the report builder engines 211 as described herein. The interface 235 may be used to output an aggregated report, as described herein, by transmitting the aggregated report to a printer 250 for printing and/or by controlling the printer to print the aggregated report. Similarly, the interface 235 may be used to output an aggregated report, as described herein, by transmitting the aggregated report to a memory 252, such as a database, to alter the memory 252 to store the aggregated report. Similarly, interface 235 may be used to output an aggregated report, as described herein, by transmitting the aggregated report to a computing device 254 for processing and/or for rendering at an associated display device 255.

[0063] As depicted, the interface 235 may be used to communicate with a source code manager engine 260 (e.g. a source code manager computing device) to receive the information identifying issues and/or behaviors associated with respective code segments 203, as described herein.

[0064] Referring to Figure 3, a flowchart of an example method 300 to implement a method to generate reports from applications is depicted. In order to assist in the explanation of method 300, it will be assumed that method 300 may be performed with the system 200, and at least partially by a computing device implementing the system 200 and/or a processor thereof. Indeed, the method 300 may be one way in which the system 200 may be configured. Furthermore, the following discussion of method 300 may lead to a further understanding of the system 200, and its various components. Furthermore, it is to be emphasized, that method 300 may not be performed in the exact sequence as shown, and various blocks may be performed in parallel rather than in sequence, or in a different sequence altogether. Furthermore, it is to be emphasized that the method 300 may alternatively be performed with the system 100, and at least partially by a computing device implementing the system 100 and/or a processor thereof

[0065] Beginning at a block 301 , a computing device receives reports from applications (e.g. the application engines 201) including indicators identifying respective code segments 203 that have been executed at the applications^. g. the application engines 201). As described herein, the reports may be received in response to a request and/or request for the reports, transmitted to the application engines 201 , which are processed by the report builder engines 211 . As previously described, the indicators identifying respective code segments 203 that have been executed at the applications (e.g. the application engines 201) may comprise respective line numbers associated with the respective code segments 203. For example, an associated line number per code segment 203 per execution may be included in the indicators; hence, for example, when a given code segment 203 is executed an integer “M” number of times, indicators may include an “M” number of instances of an associated line number of the given code segment 203.

[0066] At a block 303, the computing device determines a ranking of the indicators, as described herein. For example a ranking of the indicator may be based on a number of times that a respective code segment 203 was executed as indicated by the indicators received in the reports. Furthermore, the ranking may comprise ordering the indicators according a number of times that a respective code segment 203 was executed.

[0067] At a block 305, the computing device receives from a source code manager computing device (e.g. the source code manager engine 260), information identifying issues (and/or behavior) associated with the respective code segments 203. For example, as described herein, information identifying issue (and/or a behavior) may comprise information identifying line numbers associated with code segments 203 associated with an issue (and/or behavior)

[0068] At a block 307, the computing device determines importance scores of the issues (and/or behaviors) based on the ranking of the indicators. For example, as described herein, determining the importance scores of the issues (and/or behaviors) based on the ranking of the indicators may comprise: determining code segments 203 associated with an issue (and/or behavior) as indicated by the information received from the source code manager computing device (e.g. the source code manager engine 260); and determining positions (and/or an order) of the code segments 203 associated with the issue (and/or behavior) in the ranking of the indicators.

[0069] At a block 309, the computing device outputs an aggregated report indicating the importance scores of the issues. For example, the computing device may output the aggregated report via the interface 235 and/or via the printer 250 and/or via the memory 252, and/or via computing device 254 (and/or the display device 255).

[0070] Figure 4 is a block diagram of an example device 400 that includes a computer-readable medium 401 and a processor 402. The computer-readable medium 401 includes instructions that, when implemented by the processor 402, cause the processor 402 to generate reports from applications. The computer-readable medium 401 may be a non-transitory computer-readable medium, such as a volatile computer-readable medium (e.g., volatile RAM, a processor cache, a processor register, etc.), a non-volatile computer-readable medium (e.g., a magnetic storage device, an optical storage device, a paper storage device, flash memory, read-only memory, non-volatile RAM, etc.), and/or the like. The processor 402 may be a general-purpose processor or special purpose logic, such as a microprocessor (e.g., a central processing unit, a graphics processing unit, etc.), a digital signal processor, a microcontroller, an ASIC, an FPGA, a programmable array logic (PAL), a programmable logic array (PLA), a programmable logic device (PLD), etc. The computer-readable medium 401 or the processor 402 may be distributed among a plurality of computer- readable media or a plurality of processors.

[0071] The computer-readable medium 401 includes modules. As used herein, a “module” (in some examples referred to as a “software module”) is a set of instructions that when executed or interpreted by a processor or stored at a processor-readable medium realizes a component or performs a method.

[0072] The computer-readable medium 401 may include application modules 411 comprising respective code segments 413 and execution location markers 415 which return indicators identifying respective code segments 413 that have been executed. The application modules 411 may include instructions that, when executed, cause the processor 402 to: receive a service request from a requesting device and/or a client device; process the service request via the code segments 413 and execution location markers 415; generate a service response and indicators identifying that a respective code segment 414 has been executed (e.g. the identifiers generated by the execution location markers 415); and transmit the service response to the requesting device and/or a client device.

[0073] The computer-readable medium 401 may include an awareness builder modules 419. The awareness builder modules 419 may include instructions that, when executed, cause the processor 402 to store and retrieve the indicators (e.g. at the computer-readable medium 401 and/or a memory similar to a memory 231 ). For example, the awareness builder modules 419 may store the indicators received with a service response, as described above.

[0074] The computer-readable medium 401 may include report builder modules 421. The report builder modules 421 may include instructions that, when executed, cause the processor 402 to receive a request for a report on execution of the code segments 413; communicate with a respective awareness builder module 419 to retrieve the indicators generated by the execution location markers 415 (e.g. as stored at the computer-readable medium 401 and/or a memory similar to a memory 231); and return the report including the indicators.

[0075] The computer-readable medium 401 may include a report aggregator module 430 may include instructions that, when executed, cause the processor 402 to: receive the reports, generate an aggregated report, and output the aggregated report. In particular, as depicted, the report aggregator module 430 comprises modules 431, 432, 433, 434 to implement specific functionality of the report aggregator module 430. Indeed, in some examples, the report aggregator module 430 may be optional and the modules 431 , 432, 433, 434 may be provided without the report aggregator module 430. Alternatively, in some examples, the modules 431 , 432, 433, 434 may be optional and the specific functionality thereof may be provided at the report aggregator module 4430 (e.g. in a non-modular format).

[0076] As depicted, the computer-readable medium 401 may include the report receiver module 431 which may include instructions that, when executed, cause the processor 402 to: receive reports from the report builder modules 421 including the indicators identifying the respective code segments 413 that have been executed at the application modules 411.

[0077] As depicted, the computer-readable medium 401 may include the behavior determining module 432 which may include instructions that, when executed, cause the processor 402 to: determine the respective code segments 413 associated with given behaviors of the application modules 411 , the given behaviors including respective given code segments 413 of the application modules 411.

[0078] As depicted, the computer-readable medium 401 may include the ranking module 433 which may include instructions that, when executed, cause the processor 402 to: rank the given behaviors based on respective numbers of times the respective given code segments 413 were executed by the application modules 411 as indicated by the reports. In particular, the ranking module 433 may rank the given behaviors based on respective numbers of times the respective given code segments 413 were executed by the application modules 411 , as indicated by the reports, by summing the associated respective numbers. In some examples, the ranking module 433 may be further to rank the given behaviors for different deployments of the application modules 411 , for example by geography, company, and the like, as described above. In some examples, the ranking module 433 is further to rank the given behaviors after waiting for a given time period to allow for the reports to be received for the given time period after booting of the application modules 411 or upgrading of the application modules 411 , as described above.

[0079] As depicted, the computer-readable medium 401 may include the output module 434 which may include instructions that, when executed, cause the processor 402 to: output an aggregated report indicating the ranking of the given behaviors.

[0080] As depicted, the computer-readable medium 401 may further include a source code manager module 460 which may include instructions that, when executed, cause the processor 402 to: provide, to the behavior determining module 432 (and/or the report aggregator module 430), identifiers of the respective given code segments of the given behaviors.

[0081] It is further understood that the modules described with respect to Figure 4 may be adapted to rank issues, as described above, and/or adapted to assign importance scores to issues and/or behaviors as described above.

[0082] It is further understood that the modules described with respect to Figure 4 may be adapted to rank the indicators received in the reports and output an aggregated report indicating the ranking of the indicators.

[0083] An example of code segments 203 and execution locations markers 205 are next described with respect to Figure 5. In particular, Figure 5 depicts a code snippet 500 in Java-like code. Various functions, and other parts of the code snippet 500, are on various lines indicated by line numbering 501. The code snippet 500 may be an example of code segments 203 and execution locations markers 205 of the application engine 101 , the application engine 201 and/or an application module 411. While execution locations markers 205 are depicted in specific locations in the code snippet 500, the execution locations markers 205 may be located in any suitable location, and may be added manually and/or automatically. [0084] The code snippet 500 generally performs an authorization of a user based on a string “user” received, along with an integer “input”, for example in a request to process the code snippet 500 (e.g. a request to process the application engine 101 , the application engine 201 and/or the application module 411).

[0085] The code snippet 500 performs an authorization of the “user” for example by determining whether the “user” exists, for example in a database, and the like (e.g. via a function “!userExists(user)”). If the “user” does not exist, an error code “403” is returned. However, if the “user” exists, the code snippet 500 determines whether the “input” is valid via a function (e.g. via a function “!isValid(input)”). If the “input” is not valid, an error code “400” is returned. If the “input” is valid, a function “business logic(input)” is implemented and a code “200” is returned. Each of the depicted functions may correspond to a code segment 203, and/or branches of the code snippet 500 that include a depicted function may correspond to a code segment 203.

[0086] As depicted, the code snippet 500 further comprises execution location markers 205, for example a function “passedHere()” which may call a function which may return indicators identifying that a respective code segment has been executed, such as a line number of a function “passedHere()” being executed, and which may include a time stamp, as described above. Regardless, when a given instance of the function “passedHere()” is called, the indicators returned indicate that a code segment 203 which precedes the given instance of the function “passedHere()” was executed; conversely, when a given instance of the function “passedHere()” is not called, a code segment 203 which precedes the given instance of the function “passedHere()” was not executed, for example due to a failure of the code segment 203.

[0087] Regardless, the indicators returned by the function “passedHere()” are stored by a respective awareness builder engine 209 and/or a respective awareness builder module 419, for example at a memory 231 , and the like.

[0088] The line numbers 501 of the execution location markers 205 may be used to define given behaviors and/or issues of the code snippet 500. For example, the code snippet 500 may have the following behaviors:

[0089] -Behavior 1 : the string “user” does not exist such that the function “passedHereO” of line numbers 3, 5 are implemented. As such, Behavior 1 may be referred to as being composed of two steps. Furthermore, an associated code segment of Behavior 1 may be line numbers 2 to 6.

[0090] -Behavior 2: the string “user” does exist, but the “input” is invalid such that the function “passedHereO” of line numbers 3, 8 and 10 are implemented.

As such, Behavior 2 may be referred to as being composed of three steps. Furthermore, an associated code segment of Behavior 2 may be line numbers 2, 3, and 8 to 11.

[0091] -Behavior 3: the string “user” does exist, the “input” is valid such that the function “passedHereO” of line numbers 3, 8 and 13 and 15 are implemented.

As such, Behavior 3 may be referred to as being composed of four steps. Furthermore, an associated code segment of Behavior 3 may be line numbers 2, 3, 8, and 13 to 16.

[0092] Hence, for example, an issue and/or behavior of the code snippet 500 may be defined by the line numbers of the code snippet 500. For example, the source code manager engine 260 (and/or a source code manager computing device) may provide, to the report aggregator engine 220 a list of the above referenced behaviors the code snippet 500 with the associated line numbers and, optionally, with a respective relevancy score.

[0093] Attention is next directed to Figure 6 which depicts the awareness builder engines 209 of the system 200 storing respective indicators 601-1. .601-N identifying that respective code segments have been executed, for example in response to receiving a service request (not depicted) to process an application. The indicators 601-1...601 -N are interchangeably referred to herein, collectively, as the indicators 601 and, generically, as an indicator 601. The indicators 601 are generated by the execution location markers 205 and stored at the respective memories 231.

[0094] Attention is next directed to Figure 7 which depicts the report aggregator engine 220 of the system 200 transmitting a request 701 for reports to the report builder engines 211 , via the application manager engines 230. The report builder engines 211 intercept the request 701 (e.g. the request 701 is not passed to application engine 201); and, in response to receiving a request 701 , the report builder engines 211 communicate with respective awareness builder engines 209 to retrieve the respective indicators 601 , for example to generate respective reports therefrom.

[0095] Attention is next directed to Figure 8 which depicts the report builder engines 211 of the system 200 transmitting respective reports 801-1. .801-N to the report aggregator engine 220 in response to receiving the request 701. The reports 801-1. .801-N are interchangeably referred to herein, collectively, as the reports 801 and, generically, as a report 801. It is understood that the report builder engines 211 have generated the reports 801 from respective indicators 601

[0096] While not depicted in Figure 8, the reports 801 may include line numbers of the execution location markers 205 that have been executed, as described herein, and may indicate a number of times that respective execution location markers 205, and associated code segments 203, have been executed.

[0097] Also depicted in FIG. 8, the source code manager engine 260 provides (e.g. transmits), to the report aggregator engine 220, information 803 identifying of behaviors and/or issues associated with the code segments 203, and which are to be ranked and/or for which importance scores are to be generated. In some examples, the information 803 may be transmitted prior to the request 701 ; in some of these examples, the request 701 may be transmitted by the report aggregator engine 220 in response to receiving the information 803.

[0098] Attention is next directed to Figure 9 which depicts a portion of the system 200 that includes the report aggregator engine 220, the printer 250, the memory 252 and the computing device 254. While the other components of the system 200 are not depicted, they are nonetheless understood to be present.

[0099] Indicators of the reports 801 are also depicted in Figure 9. For example, in the reports 801 the indicators include the lines numbers of the execution location markers 205 of the code snippet 500, and a number of times that an execution location marker 205 of a respective line number was executed. For example, as depicted, for the report 801-1 , the execution location marker 205 of the line number 3 was executed 7500 times (e.g. at the application engine 201- 1). Similarly, for the report 801-1 , the execution location marker 205 of the line number 5 was executed 6000 times (e.g. at the application engine 201-1), etc. Similarly, for the report 801-N, the execution location marker 205 of the line number 3 was executed 9200 times (e.g. at the application engine 201 -N), etc.

[00100] As depicted, the report aggregator engine 220 is aggregating the reports 801 into an aggregated report 901 at which a total number of times that an execution location marker 205 of a respective line number was executed at both application engines 101 , as indicated by both reports 801-1 , 801-N. Hence, for example, the execution location marker 205 of the line number 3 was executed 16700 times (e.g. 7500 times at the application engine 201-1 and 9200 times at the application engine 201 -N).

[00101] Furthermore, at the aggregated report 901 , the report aggregator engine 220 has ranked the indicators (e.g. the line numbers) by ordering the indicators (e.g. the line numbers) according to a total number of times that the execution location marker 205 were executed. Hence, for example, as the execution location marker 205 of the line number 3 was executed more than the execution location marker 205 of the other line numbers, the line number 3 is listed first in the aggregated report 901. Similarly, as the execution location marker 205 of the line number 5 was executed a second largest number of times (e.g. 13,100 = 6000+7100), as compared to the execution location marker 205 of the other line numbers, the line number 5 is listed second in the aggregated report 901 , etc.

[00102] As depicted, the report aggregator engine 220 further generates a second aggregated report 902 based on the information 803. In particular, as depicted, the information 803 includes line numbers of the three behaviors described above with respect to the code snippet 500 (and which may also be referred to as issues). As depicted, the information 803 further includes a relevancy ranking of the behaviors. For example, Behavior 1 is ranked as having a Relevancy of Ί”, Behavior 2 is ranked as having a Relevancy of “2”, and Behavior 3 is ranked as having a Relevancy of “3”.

[00103] However, as depicted, the aggregated report 902 may be based on the line numbers of the behaviors and not the relevancy ranking.

[00104] For example, the report aggregator engine 220 may sum the total number of times that the execution location markers 205 of line numbers of the behaviors were executed, as indicated by the aggregated report 901. Hence, for example, for Behavior 1 , the number of times that the execution location markers 205 of line numbers 3 and 5 were executed are summed to “29,800” (e.g. 16,700+13,100). Similarly, for Behavior 2, the number of times that the execution location markers 205 of line numbers 3, 8 and 10 were executed are summed to “21 ,400” (e.g. 16,700+3,600+1 ,100). Similarly, for Behavior 3, the number of times that the execution location markers 205 of line numbers 3, 8,

13 and 15 were executed are summed to “24,300” (e.g. 16,700+3,600+2,500+2,500). For conciseness, in the aggregated report 902 a behavior and/or issue is referred to as “B/l”; hence, “B/l 1” refers to Behavior (and/or Issue) 1 of the information.

[00105] While not depicted, the report aggregator engine 220 may process the indicators in other ways. For example, as the line number “3” is common to all the behaviors, the numbers of times that execution location markers 205 at the line number 3 may be omitted from the sums of the aggregated report 902. Furthermore, the aggregated report 902 may be based on average numbers of times execution location markers 205 of line numbers using a number of steps of a behavior.

[00106] As depicted, however, at the aggregated report 902, the report aggregator engine 220 has ranked the behaviors based on a total number of times that the execution location markers 205 of associated line numbers were executed. Hence, for example, Behavior 1 is ranked first, as the execution location markers 205 at the line numbers 3 and 5 were executed the largest number of times, as compared to the execution location markers 205 of the other behaviors. As depicted, the Behavior 3 is ranked second, as the execution location markers 205 at the line numbers 3, 8, 13 and 15 were executed the second largest number of times. Hence, the Behavior 3 is ranked third and/or last.

[00107] As depicted, at the aggregated report 902, the report aggregator engine 220 has further assigned importance scores to the behaviors (and/or issues) based on the rankings of the indicators. For example, Behavior 1 has an importance score (e.g. “Score”) of 1 , Behavior 3 has an importance score (e.g. “Score”) of 2, and Behavior 2 has an importance score (e.g. “Score”) of 3.

[00108] However, as depicted, the report aggregator engine 220 has further generated a third aggregated report 903, also referred to as a weighted aggregated report, in which the relevancy rankings of the behaviors (and/or issues) are also used to rank the behaviors (and/or issues). For example, as Behavior 2 has a higher relevancy ranking than Behavior 3, the number of times that associated execution location markers 205 of Behavior 2 were executed may be higher weighed than the number of times that associated execution location markers 205 of Behavior 3. As such, in the aggregated report 903, Behavior 2 is ranked higher than Behavior 3, with respective importance scores increased and decreased accordingly.

[00109] As also depicted in Figure 9, the report aggregator engine 220 may output the report 901 and/or the report 902 and/or the report 903 to the printer 250 (e.g. where the reports 901 , 902, 903 are printed), and/or to the memory 252 (e.g. where the reports 901 , 902, 903 are saved), and/or to the computing device 254 (e.g. where the reports 901 , 902, 903 are processed and/or, as depicted, rendered at the display device 255).

[00110] It is further understood that the reports 901 , 902, 903 may indicate which code segments 203 “should” be tested first; for example, when a report 901 , 902, 903 indicates most executed code segments 203 (e.g. and/or associated line numbers), then it may be inferred that such most executed code segments 203 should be tested first. Indeed, the reports 901 , 902, 903 may be used to determine which bugs and/or performance results and/or security breaches should be fixed and/or addressed first (e.g. when given most executed code segments 203 in a production environment are associated with bugs and/or flaws and/or performance results and/or security breaches of an application) and/or which technical debts should be paid first (e.g. given technical debts, as indicated by the behaviors and/or issues, may be ranked as described above), and/or which performance results and/or are associated with bugs and/or flaws of an application). Furthermore, the reports 901 , 902, 903 may be used to determine which behaviors of an application are most important to users of the application (e.g. by determining, in a production environment, which behaviors are most used (e.g. as in the aggregated report 902).

[00111] It should be recognized that features and aspects of the various examples provided above may be combined into further examples that also fall within the scope of the present disclosure.

Claims

1. A system comprising: application engines respectively comprising: code segments and execution location markers to return indicators identifying that a respective code segment has been executed, the application engines respectively associated with: an awareness builder engine to: store and retrieve the indicators; and a report builder engine to: receive a request for a report on execution of the code segments; communicate with a respective awareness builder engine to retrieve the indicators; and return the report including the indicators; and a report aggregator engine to: request and receive reports from report builder engines; aggregate the indicators in the reports; rank the indicators as aggregated; and output an aggregated report indicating a ranking of the indicators as aggregated.

2. The system of claim 1 , wherein the indicators include respective numbers of times respective code segments were executed.

3. The system of claim 1 , wherein the report aggregator engine is further to rank the indicators as aggregated based on respective numbers of times respective code segments were executed as indicated by the indicators.

4. The system of claim 1 , wherein the aggregated report further indicates: first respective code segments that are most executed in the application engines; and second respective code segments that are least executed in the application engines.

5. The system of claim 1 , wherein the report aggregator engine is further to output the aggregated report using a communication interface or a printer.

6. A method comprising: receiving, by a computing device, reports from applications including indicators identifying respective code segments that have been executed at the applications; determining, by the computing device, a ranking of the indicators; receiving, by the computing device, from a source code manager computing device, information identifying issues associated with the respective code segments; determining, at the computing device, importance scores of the issues based on the ranking of the indicators; and outputting an aggregated report indicating the importance scores of the issues.

7. The method of claim 6, wherein the information identifying the issues associated with the respective code segments, and the indicators identifying respective code segments that have been executed at the applications, comprise respective line numbers associated with the respective code segments.

8. The method of claim 6, wherein the ranking is based on a number of times that a respective code segment was executed as indicated by the indicators.

9. The method of claim 6, wherein the determining the importance scores of the issues based on the ranking of the indicators comprises: determining code segments associated with an issue as indicated by the information; and determining positions of the code segments associated with the issue in the ranking of the indicators, the indicators identifying respective code segments that have been executed at the applications.

10. The method of claim 6, wherein the determining, at the computing device, importance scores of the issues is further based on a relevancy ranking of the issues as indicated by the information.

11. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to: execute application modules comprising code segments and execution location markers which return indicators identifying respective code segments that have been executed; execute awareness builder modules to: store and retrieve the indicators; execute report builder modules to: receive a request for a report on execution of the code segments; communicate with a respective awareness builder module to retrieve the indicators; and return the report including the indicators; execute a report receiver module to: receive reports from the report builder modules including the indicators identifying the respective code segments that have been executed at the application modules; execute a behavior determining module to: determine the respective code segments associated with given behaviors of the application modules, the given behaviors including respective given code segments of the application modules; execute a ranking module to: rank the given behaviors based on respective numbers of times the respective given code segments were executed by the application modules as indicated by the reports; and execute an output module to: output an aggregated report indicating the ranking of the given behaviors.

12. The non-transitory computer-readable medium of claim 11 , further comprising a source code manager module to provide, to the behavior determining module, identifiers of the respective given code segments of the given behaviors.

13. The non-transitory computer-readable medium of claim 11 , wherein the ranking module ranks the given behaviors based on respective numbers of times the respective given code segments were executed by the application modules, as indicated by the reports, by summing the respective numbers.

14. The non-transitory computer-readable medium of claim 11 , wherein the ranking module is further to rank the given behaviors for different deployments of the application modules.

15. The non-transitory computer-readable medium of claim 11 , wherein the ranking module is further to rank the given behaviors after waiting for a given time period to allow for the reports to be received for the given time period after booting of the application modules or upgrading of the application modules.