WO2020027705A1

WO2020027705A1 - Methods for tracking and optimizing of performance in organizations using artificial intelligence

Info

Publication number: WO2020027705A1
Application number: PCT/SA2018/050022
Authority: WO
Inventors: Ibraheem ALHASHIM; Ali ALBUSALEH
Original assignee: Alhashim Ibraheem
Priority date: 2018-07-30
Filing date: 2018-07-30
Publication date: 2020-02-06

Abstract

An artificial intelligence system to help enhance the performance of organizations by tracking everyday work activities and periodically and automatically producing rich reporting on these activities along with recommendations that can identify and help mitigate inefficiencies. This is achieved by capturing large amounts of signals representing user activities and running a series of activity predicting models and recommendation systems, using artificial neural networks, that can characterize the nature of the activity along with useful details such as duration and relevance.

Description

Methods for Tracking and Optimizing of Performance in Organizations Using Artificial Intelligence

An artificial intelligence system to enhance the performance of staff and organizations by tracking everyday work activities and periodically producing rich reporting along with recommendations that can help increase overall efficiency. The goals of the system are: to allow for evidence-based management, perform secure and privacy-aware tracking, allow for more fair and unbiased decision making, be very lightweight and non-obstructive to everyday work, all of which contributes towards increased overall productivity.

The present invention relates generally to using artificial intelligence systems and methods for tracking and evaluating everyday work activities performed using digital devices.

The ubiquitous use of computers and other digital devices in modern organizations has been a transformational human achievement. We are able to process documents and produce work-related articles at an unprecedented speed. However, from small to large organizations it can be very difficult to track the different pieces of work-related artifacts produced by a large number of employees. Being able to track what work is performed by which entity and the time and effort it took can be invaluable indicators on how to better manage a particular employee, a team, or an entire department. Performance indicators or key performance indicators (KPIs) are currently employed to evaluate the success of a particular activity or organization. Such indicators often rely on self-reported evidence of an activity by its performers. However, these existing practices can be greatly inefficient and fundamentally suffer from problematic tendencies of exaggerations of own work or omissions of significant deficiencies in the output.

Commercially available monitoring systems provide a basic set of reporting on user activity (e.g., timeline charts and standard tables showing basic statistics on a typical measure such as files or requests processed). However, these systems suffer from the issues outlined above. A human agent still has to go over these reports and compile more meaningful summaries, conclusions, and recommended actions. Such slow feedback cycles can hinder the efforts of process optimization. Furthermore, for large organizations, this process might not be feasible considering the scale of the organization and the number of tasks and employees involved in the numerous daily activities. This same problem is faced across many different fields and is often referred to as “big data”. Recent efforts that tackle such problems are increasingly trending towards applying methods from artificial intelligence, more specifically machine learning methods and deep learning. Artificial intelligence systems are able to handle large amounts of data more efficiently and for some applications are able to detect patterns more proficiently than any human would.

This specification describes a system for tracking and optimizing of the process and recommending performance and process enhancing suggestions with the help of artificial intelligence. The system takes as input multiple activities performed by a human operator of digital devices, which are assets of their organization, where these activities (or signals) are sent to a central artificial intelligence system, which we will refer to as the artificial intelligence (AI) engine, for processing and produce valuable insights for both management and staff.

The AI engine, typically installed on premises, receives periodic human initiated activities directly via a secure and encrypted channel between initiating device and the device hosting the AI engine. The AI engine converts these incoming activity signals to a specific internal representation and the raw data is then discarded in order to ensure better privacy protections and efficient storage of the large amounts of incoming signals. The AI engine applies multiple inference steps using existing trained models and accumulates the results and generates a periodic detailed report and a list of recommendations that can cover activities initiated by a user spanning a full workday or week. These reports also include suggested recommended actions inferred using other specialized recommendation models as part of the AI engine.

These reports are made available to authorized entities (following industry standards in authentication) for analysis and consideration. If one of the actions recommended by the system is performed, the AI engine will take that into consideration and further self-asses the long-term effects of the recommendations in its ongoing learning process of its internal models concerned with long-term effects of applied recommendations and measuring the performance of the entities that approved them. The system’s lifecycle is illustrated in the accompanying figures.

In summary, this invention comprises of multiple software and hardware components that together enable a system in which signals generated from devices such as computers, mobile devices, cameras and other digital devices, used in an organization during working hours, are securely sent to an on-premise AI engine that generates reports, insights, and recommendations by analyzing activities performed by the initiating entities. More specific details and examples are provided in their appropriate sections.

This invention is concerned with the problem of tracking and analyzing large amounts of signals generated by digital assets used by different entities in an organization in order to assess performance and help recommend productivity enhancing actions. The number of signals that can be collected from digital devices can be very large and it can be very challenging to discern its relevance through human effort. Furthermore, self-reporting is subject to all sort of human bias that could be mitigated when utilizing a fair observer. Another major problem is privacy concerns where direct monitoring of individuals by others might amount to breaking computer and privacy laws in some jurisdictions. Finally, the expected storage requirements for signals, such as regular computer screen capturing, might be very costly and in some organizations quite infeasible to achieve.

To summarize, due to the size and type of the data and signals regularly captured and the entailing requirements to store and meaningfully process such data for tracking, reporting, and process optimization applications, the problem is technically very challenging using prior solutions.

This invention utilizes recent advancements in artificial intelligence technologies to address the problems mentioned above.

The tracking step is made simple and general to many types of digital signals generated by the user (e.g., the signal’s timestamp, the current user’s login credentials, computer screens, keystroke logs, mouse movement, audio generated or recorded, a record of files manipulated, and many other manners of digital signals that can be recorded) by capturing a summarized compact representation that is periodically sent to the AI engine (typically a workstation connected using the organization’s local network). The size and breadth of the signals can be customized based on the organization’s hardware in order to ensure low resource utilization on both the machine generating these signals and on the network as a whole. The more signals the system is allowed to capture, the better the AI engine reports and recommendations would be. To address issues relating to security and privacy protection, the system uses end-to-end encryption between the device generating the signal and the AI engine where the required secret keys are only accessible to the software components operating on both ends (i.e., the signal generating device and the AI engine). Furthermore, all signals are immediately processed by the AI engine and are converted into a more abstract representation (or embedding) by applying an inference step using a pre-trained representation conversion model. This step ensures that sensitive materials (e.g. exact screenshots, keystrokes, audio clips, etc.) are only stored in an abstract representation that is only used for the reports and recommendations generation step performed by the AI engine, which in turn ensures a one-way process with no possible way to recover the original signals. The converted signal representations are also an order of magnitude more efficient to store, which in turn solves the storage challenge for large-scale tracking.

Artificial intelligence models residing in the AI engine accumulates and process thousands of signals throughout the regular course of day-to-day operations in an organization. The AI engine is made of several pre-trained models of which some are made to continue training as more signals and actions are captured. These models are designed and implemented using the latest machine learning methods (i.e., deep learning techniques). The AI engine is comprised of three key AI models: the signal conversion model, the report and recommendations generating model, and the self-assessment model. The signal conversion model is responsible for converting all incoming signals, generated by the digital assets used across the organization, into a compact and abstract representation. The reports and recommendations generating model takes as input the compact signals representations generated by the signal conversion model and infer a multitude of outputs that are compiled into a user-friendly report along with accompanying actions and recommendations. Finally, the self- assessment model captures which actions have been taken by management and their effect on the overall performance of the organization.

The life-cycle of the system described in this invention can be seen in the accompanying figures.

The present invention provides the advantage of having the ability to easily, securely, efficiently, and meaningfully track and monitor staff performance with minimal human bias leading to more evidence-based and balanced decision making across an organization, which in turn leads to enhanced overall performance driven by the strong evidence generated by the invention. The technical requirements to incorporate this invention in an existing environment is minimal (requiring the installation of non-obstructive software clients at the user’s devices, and a single specialized device hosting the AI engine and its key three AI models). Furthermore, the speed at which the tracking and report and recommendation is produced is unparalleled when compared to human-driven efforts that are currently employed (due to the use of computers that can handle much larger loads). Finally, the quality of the reports and recommendations produced would uncover patterns and correlations that might not be obvious to human observers when considering the captured signals (as has been increasing made apparent with the latest advances in AI technologies applied in many other fields).

The following illustrations represent an overview of the system proposed in this invention.

Fig.1

illustrates an overview of the life-cycle of the tracking and recommendations’ system operation. [1] Represents the digital signals sent to the AI engine via a secure network connection. [2] Represents the set of generated reports and recommendation that are inferred from the models that make up the AI engine. [3] Represents a user-friendly view of the generated reports and recommendations viewable only by the authorized entities. [4] Represents the special signals being sent to the AI engine which are generated when a system recommended action is being applied or approved.

The following description and various embodiments of methods with reference to the accompanying figures are provided to help in describing the invention as defined by the claims. Those skilled in the art will find the ideas presented in this invention to be implementable using hardware and software configurations that are commonly available and are in wide use. Embodiments are described herein by way of example for different embodiments and illustrative figures. Those skilled in the art will understand that embodiments are not limited to the embodiments or figures shown. The figures and their detailed descriptions are not intended to limit embodiments to one embodiment but to cover all modifications that maintain the core functionality and utility of the invention.

In some embodiment, the system may be designed to include a computer software that captures and relays user activity to the AI engine via a secure network connection. User activities may include a wide range of signals that are initiated by a user on a computer including and not limited to screenshots, keystrokes, mouse movement and clicking, audio signals generated or recorded via a microphone, other background processes running on the computer, CPU and memory utilization along with many other digital signals occurring during user activity.

According to one embodiment, the system’s designated AI engine is hosted on a well-equipped computer workstation that is connected to the organization’s network and is able to communicate with the capturing systems on the different computers and digital devices used by the users that are targeted for tracking. The AI engine comprises of three separate artificial intelligence trained models.

In one embodiment, the first AI engine model is the compact signal conversion model that is actively listening to incoming requests, which are encrypted and sent via and secure network protocols, whereas each signal is then converted by the model to a compact representation that has the advantages of being small-sized and may not be used to reconstruct the exact original signals which are especially useful for sensitive signals (e.g., screenshots, keystrokes).

According to one embodiment, the second AI engine model is the reports and recommendations generation model. This model takes as input a set of signals that have been converted into the compact signal representation and infers a set of descriptive labels that summarize what the set of input signals mean (e.g., the user has been using a spreadsheet software for some time, or a video file was playing). Along with these labels, the model outputs recommendations that best match the best practices following the organization’s set of policies. For example, if it is against policy to view non-work-related videos during work time, a reminder of the policy should be sent to the user.

According to at least some embodiments, the third AI engine model can be comprised of different models that assess the performance of both the recommended actions and the entities applying them. These models look at the long-term trends of actions and the decided counteractions, by the appropriate entities within the organization, and relates that to another global metric of the organizational or departmental performance. Examples and more detailed descriptions are listed below.

illustrates an example of the development and deployment of the artificial intelligence driven tracking and recommendation system of the present invention, according to at least some embodiments.

As shown, the system starts by recovering digital signals that reflect user interactions with a device within the organization. Such devices may include but are not limited to, desktop personal computers, laptops, tablet devices, mobile phones. In one embodiment, a basic custom built capturing software is installed on these devices in order to transmit user interaction signals to the next step in the system. In some embodiments, these signals may include, but are not limited to, the signal’s timestamp, the current user’s login credentials, computer screens, keystroke logs, mouse movement, audio generated or recorded, a record of files manipulated, and many other manners of digital signals that can be recorded. As shown in the figure, these signals are transmitted to the artificial intelligence engine (referred to as the AI engine). In some embodiment, if the devices are used offsite (e.g., offshore in oil and gas industries) with no means of connecting to the rest of the system, a scheduled offloading step can be used to mitigate the lack of connectivity.

In one embodiment, the AI engine is a well-equipped standalone workstation with suitable hardware that facilitates the learning and inference of AI models. A typical device may contain up to four modern graphics processing units (GPU) and should be connected, typically via ethernet, to the digital devices that have the signal capturing software mentioned above. The AI engine should also be equipped with large capacity storage and large size and fast random-access memory (RAM). The AI engine operates automatically by listening to network requests as they are received. In such an embodiment, the AI engine can concurrently run the three trained AI models. These models are implemented as artificial neural networks (e.g., implemented using the deep learning frameworks Tensorflow or PyTorch). These models are trained either with or without supervision, where the supervised variants are trained by providing examples of user activities and their expected outcomes such as the expected labels and possible recommendations. In one embodiment, one of the models converts signals to their compact privacy-protecting representation. This model takes as input the different incoming signals and produces a binary compact representation that is used in the succeeding steps. The second model is responsible for generating the reports and recommendations that are essential for assessing the current performance of the organization and suggesting possible improvements. This model takes the compact signal representations and output time-based labels and descriptions of a user’s activity. For example, when a user loads and plays a five minutes video file on their computer, several signals are periodically sent to the AI engine’s second model producing a report entry with a description similar to the following “the user played a video for five minutes”. If the organization’s policy on work computers does not permit viewing videos during work time, a recommendation is generated to notify the user of such policy (i.e., non-work-related videos are not allowed). Following the last example, an authorized entity viewing the final generated activity report (e.g., the daily activity report) may choose to follow through with the recommend action (i.e., sending a notification to the user) resulting in the action being carried out and at the same time sending a signal to the third model in the AI engine. According to one embodiment, the third model in the AI engine is trained to capture inputs from at least two separate sources, one being the approved recommended actions, and the other is overall performance indicators captured after a certain time following the time when a decision was made to apply the recommended action or not. In such an embodiment, the model assesses the quality of the recommended action and its effect on the overall performance and present the findings to the issuing entities.

This invention is applicable to many organizations with a reasonably well-connected network of digital assets (e.g., computers, printers, cameras, etc.). Such organizations include governmental entities (e.g., municipalities, hospitals, education centers, military establishments, etc.), corporate entities (e.g., banking, legal, consulting, media, industrial, etc.), and other small and medium size business and non-profit organizations.

Claims

A system for tracking and optimizing process and performance in organizations using artificial intelligence technologies, comprising of: a tracking method that captures user activity or signals and feeds them into an artificial intelligence system that generates reports and recommendations that can help enhance organizational performance via active reviews of performance and discovery of unexpected trends and possible string correlations.
The system as recited in claim 1, wherein the tracking method comprises: capturing one or more signals generated from digital devices by the tracked staff members in an organization.
The system as recited in claim 1, wherein the artificial intelligence system that generates reports and recommendations comprises of three learned models: a signal conversion model that converts signals to a compact representation to be used by other models in the artificial intelligence system; a report and recommendations generating model that generates detailed reports and presents recommendations based on the outputs; a self-assessment model that evaluates the performance changes due to human responses to the recommendations presented by the system.
The method as recited in claim 2, further comprising: capturing user interactions using digital devices such as computers and other connected digital devices and sending them to the artificial intelligence system.
The method as recited in claim 2, further comprising: non-obstructive capturing operation as a background service on the tracked system where users are not presented with any distracting messages or having any significant performance degradation on the tracked device.
The system as recited in claim 3, wherein the signal conversion model comprising: in response to a set of signals within a specified capture duration, the model computes a compact representation that is efficient to store and does not allow for exact recovery of the initial signal.
The system as recited in claim 3, wherein the reports and recommendation model comprising: upon request by authorized entities, the model generates a detailed review of the activities, concluded by applying the model on the processed compact signals representation, along with a set of actions and recommendations related to the predicted conclusions (e.g., an illegal software was used for a certain amount of time, therefore, a meeting with the tracked individual is warranted).
The system as recited in claim 3, wherein the self-assessment model comprising: in response to approved or ignored recommended actions the system records a new signal that is used for long-term performance evaluation of both the type of actions generated by the recommendation system and the approving entity.