RU2778208C1 - System for smart monitoring of user's behavior during interaction with content - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: system for smart monitoring of user’s behavior during the interaction with content contains a server and user devices connected to the server, in which primary data is collected on user devices by means of software and hardware means; behavioral events are extracted; data is aggregated, and a vector of values of event signs is formed; user’s disorders are controlled by combination of the vector of signs with conditions of user’s behavior events, predetermined on the user device; primary data and user’s disorders are stored in a server database. The server contains a module for video data validation for protection from photo- and video-forgery; a module for preliminary processing of data for a video-analytics module; a unit for control of events by the formed vector of signs; a unit for load control and data transmission. The server processes and stores biometrical data, providing cryptographic protection and reduction in load on the client.

EFFECT: increase in the quality of monitoring of user’s behavior.

1 cl, 2 dwg

Description

The invention relates to means for monitoring user behavior when interacting with content on a user device in online communication processes - during distance learning and conducting online exams, testing, determining professional competencies during interviews, conducting scientific, business events, to determine the attention and interest of users .

There is a known method for monitoring user behavior when interacting with content and a system for its implementation (RU 2673010 C1), selected as a prototype, in which primary data are collected from the user device, extract signs of user behavior events that are compared with conditions pre-set on the user device occurrence of user behavior events, and, in case of a match, transmit data on the occurrence of the event to the server and store the result in the database.

In this method, there is no protection against photo and video forgery with biometric user identification, data preprocessing, load balancing mechanisms. Also in this method, the processing and storage of part of the biometric data on the user's device leads to vulnerabilities and the possibility of unauthorized access to user data.

In addition, this method lacks the ability to configure system parameters for a specific user and the format of the discipline, does not provide the flexibility to introduce new solutions, functionality scalability, high reliability and fault tolerance.

These shortcomings are eliminated by the proposed solution.

The problem of monitoring user behavior when interacting with content on a user device is solved by developing a scalable software and hardware system that includes data preprocessing services to reduce the amount of user data on the server, intelligent noise filtering and increasing the information content of a compressed video signal, cryptographic protection of user data, and implementation of biometric user verification. with protection against photo and video forgery, load balancing using hardware and software methods in order to optimize the resources used, maximize the throughput of communication channels, reduce server response time.

The technical result consists in the presence of protection against photo and video forgery during user identification and verification, pre-processing of data for the video analytics module, load balancing, no processing and storage of biometric data on the user's device, the presence of system settings for a specific user and the discipline delivery format for process management communications through electronic user devices in order to improve the quality of monitoring user behavior when interacting with content, ensuring the flexibility of implementing new solutions, scalability of functionality, high reliability and fault tolerance.

The technical result is achieved by the fact that in the system for intelligent monitoring of user behavior when interacting with content, containing a server and user devices associated with the server, in which primary data is collected on each user device by means of software and hardware, behavioral events are extracted, data is aggregated, and event attribute values vector, a database is stored on the server and user violations are monitored by comparing the attribute vector with the conditions for the occurrence of user behavior events pre-set on the user device, on user devices in the data collection unit, the system parameters are configured for a specific user and the format for passing disciplines; the server contains a video data validation module to protect against photo and video forgery, which ensures the detection of software and physical face substitution during user identification and verification; a data preprocessing module for a video analytics module with noise filtering and image regularity analysis by comparing the brightness levels of each pixel of the input data to describe the neighborhood; a block for controlling the occurrence of events according to the generated feature vector; load control and data transfer block for load balancing between the client and the server, ensuring uniform load distribution and scaling of multiple clients and servers; the server processes and stores biometric data with cryptographic protection and reduces the load on the client; provided flexibility in the implementation of new solutions, scalability of functionality, high reliability and fault tolerance through the use of microservice architecture.

Primary data refers to data about user actions that are performed on the user device using its hardware and software, user biometric data, as well as data about the surrounding space. The collection of the listed data can be in the form of text, graphics, audio and video, including biometric data.

Behavioral events are understood as events about user actions in the system, including the connection of additional hardware, including through a Bluetooth connection, launching applications, search queries, opening files, changing the active window, scrolling, turning on / off the microphone / video camera on the device, as well as change in the direction of gaze, the angle of rotation of the user's head, the appearance of third-party faces in the frame, the appearance of extraneous noise. When collecting behavioral events, a vector of feature values is formed, where each event represents several dimensions of the vector, which is further compared with predefined conditions for the occurrence of user behavior events. The extracted feature vectors for each data type are sent to the GPU server, where the data is processed.

System configurable parameters are understood as user behavior events that need to be tracked in accordance with the monitoring objectives and which are selected before starting the online communication process in the data collection unit. For example, you can set the allowed angle of looking away, turning the head, or excluding monitoring of these behavioral events, allowing scrolling, etc. In accordance with the selected parameters, information about the occurrence of behavioral events enters the event control unit.

The processing of biometric data is carried out by a video analytics unit, in which, using computer vision algorithms, in particular, the gaze-estimation-adas-0002, facial-landmarks-35-adas-0002, head-pose-estimation-adas-0001 models, the user is identified and gaze direction analysis. Biometric data is stored on a dedicated server with cryptographic protection.

The process of identifying photo and video fraud, including the presence of software or physical face spoofing, is carried out by using SimpleNet basic neural networks. Images from the track are used to obtain four modalities (2 rank poolings and 2 optical streams), which are processed by independent SimleNet core networks and further aggregated by a fully connected layer. The use of this architecture allows using the proposed algorithm in real time.

Noise filtering, taking into account the subsequent application of object and event detection algorithms, is carried out by a data preprocessing module using a median filter, where preprocessing is performed based on image regularity analysis to search for important areas with an analysis of the neighborhoods of each pixel. In this case, the brightness levels of the pixel and its surroundings are compared.

Load balancing is carried out through a set of algorithms and methods that correspond to the OSI transport layer.

The use of a microservice architecture, in contrast to a monolithic architecture, where there is no isolation and distribution between the system modules, contributes to flexibility and scalability, and the use of orchestration allows you to control the start and stop of containers, group them into clusters and organize all the processes that make up the application, which contributes to high reliability and resiliency by managing container deployments and automating updates, health monitoring, and failover procedures.

The intelligent system works as follows.

The user device can be desktop computers, laptops, phones, tablets equipped with a webcam.

In FIG. 1 shows a functional diagram of an intelligent monitoring system, where the structural blocks of individual microservices are highlighted with a dotted line. It includes blocks for collecting data on ongoing processes on the user's device 1, video data collection 2, hardware connection control 3, data aggregation 4, load and data transfer control 5, data distribution 6, data preprocessing 7, video data validation 8, biometric verification 9, video analytics 10, event control 11, aggregation of processed data 12, database 13, summarizing and obtaining results 14.

Data collection is carried out using three blocks. Block 1 includes a module for monitoring running software processes and applications on the user's device 15 and a module for monitoring mouse events 16. This block collects data on mouse clicks and scrolls when interacting with content, and also controls running processes on the user's device and passes the results to the module aggregation module 20. Block 2, which includes a webcam connection module 17, implements image capture from a user device based on video data collection tools (webcam) and desktop snapshots, and transmits the captured images to a data aggregation module 20. Block 3 includes includes modules for collecting data on Bluetooth connections 18 and network control 19 to check the stability of the connection with the server and the Internet, and transmits the results to the data aggregation module 20. The data aggregation module 20 extracts a feature vector for each type of data received from the user device in as a result of the operation of blocks 1, 2, 3, and sends a vect op signs on the GPU server through the data transfer module 21.

Then the data is distributed by module 22 to the event control unit 11 and the data preprocessing module 23. Module 22 is implemented using hardware-software methods and reduces the load on the communication channel with a decrease in video data processing time.

The event control unit 11 of user behavior when working with technical devices is carried out using the module for analyzing running software tools 24, the module for analyzing Bluetooth connections 25, designed to study and monitor traffic and analyze connections to the user device, the mouse event control module 26, which allows you to control clicks and scrolling on the page depending on the content.

Module 23 implements data pre-processing, performing intelligent noise filtering, taking into account the subsequent application of object and event detection algorithms, as well as compressing video signal data without losing important information. Next, the data is transmitted to the video validation module 27.

The video data validation module 27 analyzes video based on the 3d-reconstruction algorithm of a face in a photo and estimates the depth map of local descriptors, detects photo and video forgeries, including the presence of software (photo and video overlays) and physical face substitution, detecting forgery in real time . Upon successful validation, the frame will enter the biometric verification unit 9, which includes a face search module 28 in the image and a biometric verification module 29, which, in accordance with the identification number, performs user verification. User verification is performed at the very beginning of the system operation and then after a set period of time At. Verification is performed by extracting high-level facial features and comparing them with the available biometric data in the database.

After successful verification, the data is transferred to the video analytics unit 10, which includes a cryptographic protection module 30 and a module for analyzing non-verbal signs of user behavior 31 (gaze aversion, head rotation, emotions) using the Open VINO™ Toolkit library and pre-trained deep models from the Open Model Zoo repository, giving a gain in data processing speed.

The gaze-estimation-adas-0002 deep learning model receives as input the eyes detected using the facial-landmarks-35-adas-0002 model and the orientation of the head along the local xyz axes, forming an unnormalized gaze direction vector at the output. In addition, the head position angles along the xyz axes are formed using the head-pose-estimation-adas-0001 model.

The processed data from the event control unit 11 for analyzing user behavior when working with technical devices and the video analytics unit 10, which analyzes non-verbal signs of behavior, are sent to the processed data aggregation module 32, which controls the totality of all groups of events and records all detected signs of events in the database 33. Summing up and obtaining results module 34 allows you to view the results of monitoring the user's behavior on the user device when interacting with content, generating a report with full statistics (types, quantities and time of detection of signs of events) on detected violations in the form of interactive graphic diagrams in the interface part of the system monitoring FIG. 2, as well as in the form of a generated Excel report.

In addition, the system for monitoring user behavior in the event of violations of its interaction with content in real time displays all information about violations in the form of interactive graphic diagrams on the screen of the examiner (lecturer).

The proposed intelligent system for monitoring user behavior when interacting with content, implemented as microservices, meets the requirements:

1. Settings for communication processes through electronic user devices for a specific user and the format of passing the discipline, and can be used in applied areas of online work.

2. Scaling and building up the system architecture, which is required in modern realities, when a number of processes in educational and other organizations are transferred to a remote mode of operation using multimedia platforms for video conferencing.

3. Increases the efficiency and productivity of the processes of monitoring, analysis, control and implementation of effective management of communication remote processes.

4. Provides an increase in the reliability of decision-making when monitoring user behavior when interacting with content by expanding the control functionality and the ability to adjust system parameters both for a specific user and a specific technical process.

5. Improves system stability through load balancing.

6. Provides biometric user verification with protection against photo and video forgery, as well as cryptographic data protection to prevent unauthorized access to user data on the server.

Claims (1)

A system for intelligent monitoring of user behavior when interacting with content, containing a server and user devices connected to the server, in which primary data is collected on each user device by means of software and hardware, including a module for monitoring running software processes and applications, a module for monitoring mouse events, a module connecting to a webcam with capturing images of desktop snapshots, a module for collecting data on Bluetooth connections and monitoring the network to check the stability of the connection with the server and the Internet; extraction of behavioral events; aggregation of data and formation of a vector of event signs values; control of user violations by comparing the feature vector with pre-set conditions on the user device for the occurrence of user behavior events; storage of primary data and violations of the user in the server database, characterized in that on the user devices in the data collection unit, the system parameters are configured for a specific user, including, in addition to the biometric user verification, data on the user's behavioral events and the discipline delivery format; the server contains a video data validation module to protect against photo and video forgery, which ensures the detection of software and physical face substitution during user identification and verification; a data preprocessing module for a video analytics module with noise filtering and image regularity analysis by comparing the brightness levels of each pixel of the input data to describe the neighborhood; a block for controlling the occurrence of events according to the generated feature vector; load control and data transfer block for load balancing between the client and the server, ensuring uniform load distribution and scaling of multiple clients and servers; the server processes and stores biometric data with cryptographic protection and reduces the load on the client.

Images