UA141317U - AUTOMATED METHOD FOR DETERMINING THE SCOPE OF INFORMATION RECOGNITION - Google Patents

Abstract

An automated method of determining the amount of information recognition by the subject when providing multimedia information includes providing multimedia information, recording and processing data on the movement of the subject's eyes using a video capture device and using an artificial neural network to determine the amount of recognition by the subject of multimedia information.

Description

The useful model belongs to technologies that involve means and methods of tracking the movement of subjects' eyes and refers to an automated way of determining the volume of reading or recognition by the subject of specified areas when providing multimedia information. A useful model allows you to establish whether the subject was familiarized with the visual information presented to him, in particular textual information and/or pictorial information, in what volume, and whether this volume was sufficient to confirm the subject's familiarization with the content of the text fragments of the provided multimedia information, whether the subject has read the presented information, in particular whether the subject has read the textual information.

A useful model allows you to objectively establish whether the specified subject has read the text information and in what volume, and whether this corresponds to the specified volume, which is established as sufficient for the subject's perception of the content of the presented text fragments in the provided multimedia information.

Eye tracking technologies (oculography, eye tracking) involve determining certain characteristics of eye movement, in particular, determining the point at which the gaze is directed, or the coordinates of the point of intersection of the optical axis of the eye (eyeball) with the plane of the specified object, in particular the plane of the device on which the /a certain visual stimulus is presented.

The use of eye tracking systems (eye tracking or oculography) in various fields, and in particular for determining the subject's reading, is well known.

There is a known method of confirming the reading of an electronic message by a user using eye position tracking technology when the user reads an electronic message containing many lines on an electronic device, in which the number of lines of the message viewed by the user is determined based on the characteristics of the eye position, in particular the determination of the coordinates of the eye position, and it is determined whether the user read the message by comparing the number of lines viewed by eye to a threshold number of lines based on the plurality of lines of the electronic message. In addition, it is determined whether the user was surprised while viewing an electronic message by comparing the duration of time spent by the eye while viewing an electronic message with the expected duration of time for displaying the provided electronic message (05 2016/0094705 AT, 31.03.2016).

A known method and system for testing a user's language knowledge that allows for the automatic determination of language proficiency using the technology of tracking the user's gaze while reading a text sample, using a camera (or eye tracker) connected to a processor and made with the possibility of recording eye fixation time for each word of the text sample presented to the user, and subsequent recording of the saccade time for each pair of words on which the user's gaze moves between fixations, followed by a comparison of the user's gaze parameters with the parameters of the gaze model of at least one reader having a certain the known level of language proficiency, and the subsequent generation of the user's language proficiency assessment based on the results of the comparison (05 2019/0080623 А! 03/14/2019).

05 2012/0237084 AT, 20.09.2012) describes a method and system for identifying text and its position in visual media content, for example, a document displayed on a computer monitor or other display device, allowing to determine whether sub the subject interacted with the text, the level of the subject's interaction with the text, for example, whether the subject looked at the text, whether the subject read the text, whether the subject perceived the text and understood it, or other types of interaction of the subject with the text. The determination is based on data obtained from the eye tracking device, and the emotional response detection system can be used separately or together. The method provides identification of a part of the visual media content that makes up the text, determining the position of the identified text of the visual media content, collecting data from the subject's eyes while viewing the visual media content, regarding pupil size, blinking and gaze coordinates, generating gaze patterns of the subject based on the collected eye data, determining whether the subject interacts with the identified text based on at least a determined position of the identified text and the generated gaze pattern, determining the level of interaction of the subject with the identified text, in particular, determining whether the subject has read or familiarized the subject with the identified text, and determining the emotional response of the subject to the identified text.

Known methods and systems involve the use of specialized equipment for tracking eye movements, which cannot always be provided outside of laboratory conditions and research institutions, and especially in everyday life. When studying eye movements, even with the involvement of specially adapted equipment, there is a question of data selection for further processing and interpretation. For example, in the case of covert attention observed in eye movement recordings, when the gaze trace and fixation points pass by the real object of attention with only a small fixation time, there are difficulties in establishing an unambiguous relationship between the results of eye movement research and the cognitive process of a specific subject

Although various eye tracking systems and their applications are known, there are still certain problems that people face when reading, and everyone solves these problems in a unique way: rereading complex parts of text, looking for unknown words, writing down details to remember, etc. . Automating these processes with the involvement of publicly available and commonly used equipment, for example, to monitor human reading, assess speed, distinguish eye movements and behavioral responses during reading from other activities, comment on fragments of text that are difficult to read or whose content was not recognized, etc. and., are functions that can be extremely useful for people who have to process huge amounts of text every day.

Currently, there are no solutions for standardized verification and confirmation of the process of processing visual information, and in particular reading, with high accuracy based not on specialized hardware, but on any video devices or web cameras involving the principle of tracking eye movements.

The problem, which is solved within this useful model, lies in areas where decision-making significantly depends on whether the subject (person) has actually familiarized himself with the presented information, in particular, read a certain document. A useful model can be applied in such fields as education, legal practice, public sectors, etc.

The purpose of the useful model was to provide the possibility of automated determination and evaluation of the subject's recognition of information and the amount of reading with high accuracy when viewing multimedia information on a variety of devices, such as a personal computer or laptop, tablet, smartphone, virtual screen or any other appropriate display device or medium. The useful model allows in real time to check the subject's familiarization with the presented information, to determine its scope, to determine the subject's recognition of text sections, tables, images, etc. when providing multimedia information and the amount of information, the content of which was recognized by the subject,

In particular, checking the reading of text parts in the presented multimedia information or reading any electronic document.

The useful model is based on the task of providing an automated way of determining with high accuracy the amount of information recognition by the subject when providing multimedia information, in particular the amount of information suitable for reading, even better determining the amount of read information, using widespread electronic devices suitable for displaying multimedia information , video recording, processing, storage and transmission of data.

The task is solved by the fact that the automated method of determining the scope of information recognition by the subject when providing multimedia information, which involves the provision of multimedia information, recording and processing of data on the movement of the subject's eyes using a video device and determining the scope of recognition by the subject of the provided multimedia information, according to the useful model, the method is carried out using a system that includes: - user hardware and software complex (APK-K), in which a video device is connected or integrated, a multimedia information display module, a data reception and transmission module, a video data recording module from a video device , a data storage module, a data processing module, and - a remote hardware and software complex of interaction with the user (APK-VC), in which a multimedia information display module, a data reception and transmission module, a data storage module, a subject verification module are combined or integrated -user, processing module yes them, which is suitable for the work and training of an artificial neural network, and the method includes the following stages: - in the APC-K, calibration is carried out taking into account the individual anatomical parameters of the eyes when providing the subject with moving visual stimuli and tracking tasks; upload multimedia information to the remote APC-VK, classify fragments into structural elements according to features - text, image, table, etc., transfer the classified information from the remote APC-VK to the APC-K and present it to the user; - according to the data of the video stream from the video device on the APC-K, the data of eye movement and/or gaze are recorded and, taking into account behavioral reactions in the form of at least 60 eye blinks and/or head movements, and/or changes in the coordinates of pre-marked points on the face, individual gaze points are determined at each moment of time in the coordinates of the plane of the multimedia information display module of the APC-K during the provision of information, and transmit the received information to the remote APC-VK in which: - establish the correspondence between individual points of view and the fragment of multimedia information provided at this time, - with the application an artificial neural network forms the data of a set of views and establishes the probability of the relation of each individual view of the subject to reading; - aggregate data from a set of views into a coordinate system of classified structural elements of the provided multimedia information; - calculate by the number in each structural element the density of all views and the density of views defined as reading, and set appropriate threshold values for the calculated densities that delimit statistical noise; - calculate the ratio of the overlapping area of the area, where the data density of all views and the data density of views defined as reading exceeds the threshold values, to the total area of all structural elements defined in the provided multimedia information, and establish the amount of recognized information by the subject by which the information is determined as read

In a further aspect of the utility model, the multimedia information presented to the subject is text fragments without images and/or tables or with images and/or tables in known electronic document presentation formats or video information containing text fragments and/or images , and/or tables.

In yet another aspect of the utility model, the method includes an additional step in which, before providing multimedia information to the subject, the identity of the subject is verified based on visual data about the subject, which is entered into the remote APC-VC.

In another aspect of the utility model, the method includes verification of the identity of the subject, which is carried out before each subsequent stage of providing the subject with multimedia information or an electronic document.

In yet another aspect of the utility model, the method involves calibrating the eye parameters according to the determined behavioral responses before each step of providing the subject with multimedia

From information or an electronic document.

In a further aspect of the utility model, the device for displaying multimedia information or, for example, an electronic document is, in particular, a television, a computer monitor, a smartphone display, a tablet display, or any other device designed to display multimedia information.

In yet another aspect of the utility model, the applied video recording device is, in particular, a web camera or a video camera connected or integrated into the APC-K, in particular a computer, smartphone, tablet, or any video device made with the ability reception and transmission of video data.

In the next aspect of the useful model, the number of transitions from the specified text to accompanying materials, in particular tables and images, and the time spent on such transitions are determined when providing multimedia information, and optionally presented on the multimedia information display module.

In yet another aspect, the useful model forms a map of the data density of views and/or the density of data views defined as reads and/or the overlap area of the region where the data density of all views and the data density of views defined as reads exceeds threshold values, and necessarily present on the multimedia information display module.

In yet another aspect of the utility model, the percentage of viewed structural elements out of the total number of structural elements in the presentation of multimedia information, in which the view was not defined as reading, is calculated and optionally displayed on the multimedia display module.

In yet another aspect of the useful model, the percentage of structural elements in which the view was determined to be read is calculated out of the total number of structural elements when providing multimedia information, and optionally displayed on the multimedia information display module.

In a further aspect of the utility model, the ratio of the volume of viewed structural elements, in which the view was not determined as reading, to the volume of structural elements, in which the view was determined as reading, when providing multimedia information, is calculated, and optionally presented on the multimedia display module information

In the next aspect of the useful model, after the completion of the session of providing multimedia information or its fragment, a reading verification certificate is formed in APK-VC, in particular as an electronic document-reading verification certificate, and optionally stored in the data storage module.

In the next aspect of the utility model, the read verification certificate is uploaded to the APK-K.

In the next aspect, the invention provides that all received data of the subject is stored in the data storage module APC-VK and optionally APC-K.

Fig. 1 - Automated system for determining the amount of information recognition, where (1) APC-K, (10) a video recording device, (20) a multimedia information display module, (30) a data reception and transmission module, (50) a recording module from a video recording device of video stream data ) coordinates of the plane of the multimedia information display module; (2) APC-VC, (21) multimedia information display module, (31) data reception and transmission module, (41) data storage module, (61) user subject identity verification module, (71), data processing module , suitable for artificial neural network operation and training, (81) artificial neural network operation and training unit, (91) multimedia information fragment classification unit into structural elements, (101) video stream data processing unit from the APK-K video recording device, (111) block for establishing the correspondence between individual points of view and a fragment of multimedia information provided at this time, (121) block for establishing the probability of the relation of each individual view of the subject to reading, (131) block for forming the data of a set of views and aggregating the data of a set of views into a coordinate system of classified structural elements of the provided multimedia information, (141) block for calculating the number in each structural element of data density of all views and data density of views defined as reading, and setting the corresponding threshold values for the calculated densities, (151) a unit for calculating the ratio of the overlapping area of the area where the data density of all views and the data density of the views defined as reading exceeds the threshold values to the total area of all structural elements defined in provided multimedia information, (161) block for setting the amount of information recognized by the subject, (171) block for setting the amount of text information read by the subject, (181) block for providing statistical information and generating reports.

From Fig. 2 - Indicators in time series.

Fig. C - Filtering using VROSU, EROSMU.

Fig. 4 - Filtering only eye movements on the screen.

Fig. 5 - Convolutional Neural Network architecture with 113,006 training parameters.

Fig. 6 - Classification of each group of fixations (observations grouped by EROSIV).

Detailed description of the useful model

Available methods and systems for tracking eye movements for various purposes involve the use of specialized video devices - eyetrackers. As a rule, attention is focused on determining the exact coordinate positions of the smallest structural elements of the text, in particular words, even better letters, and on tracking trajectories and basic movement patterns. There are certain limitations to the use of eyetrackers depending on the method on which the system is based, in particular dark pupil or light pupil, and the significant influence on the obtained data of certain behavioral reactions.

The known methods do not offer methods for remote processing of eye tracking data obtained from non-specialized eye tracking devices, in particular, using a standard web camera, which can be defined for example as web tracking, or any video device attached or integrated in any electronic device for personal use, suitable for receiving, processing, transmitting and, optionally, saving data, where such a method has a high accuracy of determination and is easily accessible to the user.

For the purposes of determining the amount of information recognized and/or read by the subject without the involvement of specialized equipment, this useful model proposes the combination in a defined way into an automated system of hardware and software configured to perform certain functions, as described here. Also, for the implementation of a useful model, the use of an artificial neural network is provided, with the help of which training at the initial stage, template models for determining the density of gaze in the defined structural elements of multimedia information, in particular an electronic document, were created, which later made it possible to achieve 97 95 accuracy in determining the amount of recognition of the provided multimedia information and determining the volume of read information.

The described method allows the user subject to track and store all materials for bo-reading and associated reading metadata, to search and filter content by reading template parameters (for example, the time and speed of fixation allow to single out the area of interest within the text); receive various analytical annotations after processing, receive real-time help options when familiarizing with the content of multimedia information, including reading (for example, automated translation). The method automatically tracks eye movements and correlates them with information such as text on the screen, allowing for real-time processing of reading patterns and storing all metadata for later analysis.

To solve the problem of computer-aided reading verification in the claimed utility model, it was necessary to convert a common off-the-shelf video device, in particular a web camera, into an eye tracking device. For these needs, machine learning was applied to build a system for eye gaze tracking and recognition of specific eye movements during reading, which, together with the introduction of new elements and improvement of known ones, as well as due to new connections and a defined sequence of actions, made it possible to effectively solve the problem of reading verification .

The implementation of a useful model involves the use of conventional video devices, in particular, connected, built-in or integrated into any electronic device, for example, a tablet, smartphone, laptop, personal computer or other suitable, which requires the creation of an adapted hardware and software package (APK), where important there is also the selection of data for video recording, the method of video recording, the method of their processing, transmission and the selection of criteria for interpretation.

At the beginning, work was carried out on the analysis of the subject's visual activity, where reading/non-reading was classified according to the gaze, regressions, interline transitions (5vmeer5) and saccades were detected, and a set of individual gazes was mapped on the structural elements of the multimedia information/document. As a suitable means for this was the Isocrema program as described, Vopota? V., Spatsyv M., MeIpuspik U. Maspnipe I eatipa Tesnpidnev ip Veadipu Tgaskipd. My achievements |(IZTOVMI OOK)I / A. Vopota?, M. Spatsv, M. MeIpuspyk. - Veepimeog

Asadetu apa VsheO iarbe rappegei m/yy o OayaKooi arch. - Osi 19, 2018 perzv//tediit.sot/edVeepNimeogagoMr/tasnipe-Ieagtipd-iespppidpevz-ip-teadipd-igasKipa-teei-uotsg- aioguisok-bas7bbavazzZa| for the creation of which mainly open source libraries were used

With a code.

The details of all patents, published applications and references cited in this specification are incorporated by reference in their entirety.

Data collection

When collecting data to track human activity, a tracker was used, for example

SaeRoiipi eu, which made it possible to obtain gaze coordinates with an error angle of no more than 1-1.5 degrees after calibration. During each session, the SaeRoipi Appaiuziv program records video and on-screen images along with eye movement tabular data.

The entire data set consisted of 2 parts: 51 read time series and 85 non-read time series. Each research participant had to perform the following actions: read a 2-minute text; find specific information and things in the pictures, watch a 3-minute video.

The obtained time series consists of several columns (Fig. 2):

EROSH, EROHU - screen coordinates, relative to screen resolution, algorithm A

VROSH, VROHU - screen coordinates, relative to screen resolution, algorithm B

EROIO - fixation identifier

FPOGD - eye fixation duration

BPOGV, FPOGV - validity of information

VKIO - blink identifier

Data preprocessing and feature selection

In the course of our research, we discovered that the tracked coordinates may not be perfect. Blinking, moving the head, changing lighting - all these factors interrupt or spoil the flow of data. Therefore, additional steps were taken that could alleviate the situation, smoothing the movement of the gaze was adopted as the main option, although the value of this approach has limits. Smoothing eliminates one important feature - microsaccades. Saccades are rapid, simultaneous movements of both eyes between two fixation points. Microsaccades are movements within a single fixation that respond to how users fixate their gaze. Although smoothing is not an ideal choice when it comes to saccades, it helps to approximate word detection.

This is what filtering meant for these research purposes (Fig. 3, 4): filtering using VROSU, EROSMU, filtering only the movements of the gaze on the screen.

In order to easily manipulate the data set and the training/gtest models, a time sequence processing window with a width of 100 values (which roughly corresponds to the average time to read one plane on A4 paper) was chosen. This resulted in splitting all data sets into 24,568 read series and 14,288 "no-read" series of multiple time series with a length of 100 values, allowing for 90 95 overlap.

Read/non-read classification

Next, three main methods of time series classification were used. Three groups of features were created: detection of a linear trend for EROSH, seasonality and general characteristics, but the selected features were not informative and did not describe the data well.

Next, a convolutional neural network was applied. The functions ah, au were used. After some tuning, the optimal architecture was found with 113,006 training parameters, see Fig. 5. This model gave 96 95 accuracy on a subset of tests and was subsequently selected as the baseline model for further research.

Reading clustering models

The main task was to classify each group of fixations (observations grouped by EROSIV) as one of three main patterns: saccades, interline transitions, regressions (Fig. 6).

The main obstacle encountered at this stage was the labeling of the data set, as 60 Hz data is inherently difficult to label. This also proved to be a problem for clustering. Some of the minor issues that have been addressed are high similarity between regressions and line transitions, and scrolling freezes (5sgoiIipo) that were outliers. To exclude fixations during scrolling, a reading classification algorithm was used.

The entire dataset was filtered from spikes to saccades only. To obtain saccade data, points were grouped by fixation identifier (EROSION) and only the last observation from each group was selected. As a result, all identified saccades were divided into minimum/maximum values with a naive algorithm along the horizontal axis, and minimum saccades were divided into groups of interline transitions and regressions. To achieve the required results, we used clustering using the K-means method on the three main ones

From the functions of saccades: the projection of interline transitions along the horizontal axis, the angle of interline transitions along the horizontal axis, and the difference from the previous saccade.

One of the examples of automated processing of multimedia information is the creation of a complete text document from a video in the following sequence: identification of static frames in the video (with a threshold deviation between the shots), separation of sheets (text fragments) from the frame, then combined sheets into a single "panoramic" images, separating text into a displayed document, identifying points of interest.

As a result, it was possible to obtain a machine learning model capable of predicting with 97 95 accuracy whether the user has read/not read the text within 1.6 seconds of recording.

Some of the main findings of the study are that an algorithm that relies on the predictions of a prior machine learning (ML) model can account for gaze movement (regressions, interline transitions (interline transitions) and saccades) and calculate relative reading speed; an algorithm that provides information about reader's interest, provides a weight factor for a given word, which may be important for the reader.

Scientists who study human behavior such as eye tracking can uncover previously blocked areas in health technology and business. As evidenced by this algorithm, it is possible to make real-time predictions based on gaze-tracking technologies, and perhaps go beyond that with more scientific applications that no one thought possible before.

Determining the general parameters of reading without clearly separating that part of the text information that was actually read does not allow to predict or establish with high accuracy the amount of recognized content of text information among all multimedia or all text information, the difficulty of its assimilation, separation of particularly important content fragments, etc.

With the involvement of the method described above for the best implementation of the method of determining the amount of information recognition by the subject when providing multimedia information, in particular, determining the amount of reading by the subject of structural elements of multimedia information or an electronic document defined as text, an automated system was applied.

The selected parameters for targeted training of the artificial neural network allow the following determination of the probability of the relation of each individual view of the subject to reading. 60 At the same time, according to the spatial and temporal characteristics of eye movements, taking into account behavioral reactions, it is possible to divide sections of text fragments of multimedia information into groups of "careful reading", "review or familiarization reading", "scanning reading" and "scrolling" and separate them for further analysis areas of text fragments highlighted as "careful reading" or proper reading.

The automated method of determining the amount of information recognition by the subject is carried out as follows.

Using the eye calibration unit (90) in the APC-K (1), calibration of the eye parameters is carried out taking into account the individual anatomical parameters of the eyes when providing the subject with moving visual stimuli and tracking tasks. Calibration can be carried out both once and several times - or repeatedly as needed, in particular with a certain frequency or before each session of providing multimedia information to the subject.

The connection of the subject's interaction with multimedia information is confirmed by verifying the subject's identity in module (61), which is carried out before providing the subject with multimedia information on the basis of visual data about the subject that is entered into the remote

APC-VC, once or several - or repeatedly, for example with a certain frequency or before each subsequent provision of multimedia information to the subject, or after interrupting the viewing session for a predetermined time before each session of providing multimedia information after such an interruption.

Identification/verification of the subject's identity can be done by entering data about the subject into the database of the data storage module of the corresponding device in advance or when the subject first applies to the APC-K.O. In subsequent sessions of the subject's application to multimedia information about which provides for determining the scope of recognition and/or reading in any specified mode, compares the subject's parameters with those available in the database during the entire period of the subject's information processing session or sessions. When authorizing in the system on the first visit, in order to simplify the identification/verification mechanism and reduce the time for identification, a system of logging into the system using existing accounts, in particular social network accounts, etc., including audio and/or photo and/or video recorded data may be applied object, in particular its appearance, such as facial features, etc.

After calibration, or before it, or during calibration, using the data reception and transmission module, which is suitable for receiving and transmitting information (31), upload multimedia information to the remote APC-VK (2) and use the fragment classification unit (91) to classify its fragments into structural elements by characteristics - text, image, table, etc. (all other information other than text, images, or tables).

Next, the classified information is transferred from the remote APC-VC (2) to the APC-K (1) and provided to the subject on the multimedia information display module (20).

Multimedia information can be displayed on any suitable display medium, such as a TV screen, a computer or laptop monitor, a smartphone display, a tablet display, etc.

Using the video recording device (10) of the video stream, the recording module (50) of the APC-K (1) records eye movement and/or gaze data taking into account behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in coordinates beforehand marked points on the face, and in the data processing unit of the video stream (80) of the data processing module (70) with the help of the individual point determination unit (100), individual points of view are determined at each moment of time in the coordinates of the plane of the multimedia information display module (20)

APC-K (1) during the provision of multimedia information.

It is important that any video device capable of receiving and transmitting video data is used as a video recording device, in particular a web camera or a video camera connected or integrated into the APC-K (1), in particular it can be used connected to a personal electronic device (personal computer, phone, smartphone, tablet or other) photo and/or video camera or present, built-in to said device, or otherwise.

The received information is transmitted in any known way (wired or wireless) with the use of data reception and transmission modules (30) and (31) to the remote APC-

VK (2) and then to the data processing module (71).

With the use of the work and training unit of the artificial neural network (81) and the matching unit (111), the correspondence between the individual points of view and the currently provided piece of multimedia information is detected, then with the use of the artificial neural network of the unit (81) they form the given set of views and with the help of the probability setting block (121), the probability of the relation of each individual view of the subject to reading is determined.

The specified data of the set of views are aggregated into the coordinate system of the classified structural elements of the provided multimedia information in the block of formation and aggregation of the set of view data (131).

Further, in the data processing module (71), the data density of all views and the data density of views defined as reading are calculated in the calculation unit (151) according to the number in each structural element, and appropriate threshold values are set for the calculated densities that delimit statistical noise.

Additionally, in the data processing module (71), optionally in a separate block associated with blocks (81)-(181), determine the number of transitions from the specified text to accompanying materials, in particular tables and/or images, and the time that is spent on such transitions that are taken into account when determining behavioral reactions when reading and "not reading".

Then, in the area overlap area ratio calculation block (151), the ratio of the overlap area of the area where the data density of all views and the data density of views defined as reading exceeds the threshold values to the total area of all structural elements defined in the provided multimedia information is determined, and set the amount of information recognized by the subject using block (161) of setting the amount of information recognized by the subject, after which the read information is determined using block (171) of setting the amount of text information read by the subject.

Multimedia information to be processed by the subject is any type of information suitable for display on various devices and may contain text or its fragments (in particular, for example, inscriptions, numbers), tables, images, text fragments without images and/ or tables or with images and/or tables in known electronic document presentation formats, for example, is an electronic document, an electronic book (for example, a production document, educational material, electronic testing material, entertainment material, etc.) and/or video information containing text fragments and/or images and/or tables. After processing in the block (91) the multimedia information loaded in the APC-VC, it is classified by

From the separation from the entire content of fragments containing text features, which are text fragments, tables (with text elements and numbers) and images, in particular, related to text and/or table fragments.

Multimedia information is presented to the subject in a conventional format, but preliminary classification allows to focus on behavioral responses and eye movements in relation to target parts of multimedia information.

The remote APC-VC additionally provides the ability to present the results of data processing regarding the amount of recognition and/or reading of multimedia information on the user's display device (20) in a variety of ways. The scope and type of provision of this data may be determined by the operator of the remote APC-VC and/or the owner of this multimedia information and/or the user or otherwise.

In particular, in the block of providing statistical information (181), data on the subject's behavior during reading is generated, in particular the amount of information read, reading speed, the complexity of information perception, data on various aspects of eye movement, and further on the basis of these data, the a report, in particular a certificate of reading verification, in particular as an electronic document, after the completion of each session or series of sessions of providing multimedia information or its fragment.

In particular, the multimedia information display module (20) and/or (21) provides a generated report and/or map in any visually acceptable form, and/or diagram, and/or graph or other, of views density and/or data density of views defined as reading and/or the overlap area of the area where the data density of all views and the data density of views defined as reading exceeds the threshold values, and/or the calculated percentage of viewed structural elements out of the total number of structural elements when providing multimedia information, in in which the gaze was not defined as reading, and/or the calculated percentage of structural elements in which the gaze data was defined as reading, from the total number of structural elements in the provision of multimedia information, and/or the calculated ratio of the volume of viewed structural elements in which the gaze was not was defined as a read, to the extent of the structural elements in which the view data was defined as a read, provided and multimedia information, etc.

Such visualization of the results of recognition and/or reading of multimedia information and the speed of these processes can occur both after the end of the session of providing multimedia information, and during the session after the end of viewing certain defined fragments of multimedia information in real time. This is an additional tool for controlling the process of recognition and/or reading for the user and/or the owner of multimedia information and/or the operator of the separate APC-VC and develops into a number of additional capabilities of the method and system described here.

Data exchange between APC-K and APC-VK is carried out using wired or wireless transmission, while setting up APC-VK allows you to expand the system and carry out such data exchange with multiple APC-K, and also ensures a reduction of the functional and software load on APC-K , all received subject data are stored in the data storage module (41)

APC-VC and optionally part of the data, in particular reports and/or maps and/or certificates - in APC-

K.

The application of a machine learning approach makes the claimed method of checking the familiarity with multimedia information and checking the reading easily applicable and standard on many devices. Thanks to the declared method and system in real time using any means of video recording, it is possible to detect with 97 95 accuracy whether a person is reading at the moment and calculate the amount of information actually read.

The data obtained in this way regarding the amount and/or speed of recognition and reading of information allow to expand the possibilities of using the methods and/or systems described here, based on eye tracking technology, for various purposes.

Thus, taking into account the possibility of clear identification/verification of the person of the subject in connection with specific multimedia information, for example, an electronic document that he views, recognizes and reads, there is a possibility of independent automated confirmation of familiarization with the document by this person and confirmation of the amount of information read, which allows with a high degree of probability to assert the depth of cognitive processing of information by a specific subject. This is useful in various educational processes, in the assessment of certain professional skills, assessment of remote interaction between personnel within the same enterprise or between different structures, in particular the public sector, in legal practice, etc. In addition, the relationship between the person of the subject, the specific electronic document with which he got acquainted, and the amount of content read in this document

Zo allows for automated and easily accessible certification of a read document, in particular by creating a document signing certificate, which can be used in many industries.

While the described embodiments have been based on preferred embodiments, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the embodiments covered by the appended utility model formula.

Claims (15)

USEFUL MODEL FORMULA 1. Automated method of determining the amount of information recognition by the subject when providing multimedia information, which involves providing multimedia information, recording and processing data on the subject's eye movements using a video recording device and determining the amount of recognition of the provided multimedia information by the subject, which differs in that , that the method is carried out using a system that includes: - user hardware and software complex (APK-K), in which a video recording device, a multimedia information display module, a data reception and transmission module, a recording module from a video recording device of video stream data are combined or integrated, a data storage module, an automated data processing module, and - a remote hardware and software complex of interaction with the user (APK-VK), in which a module for visual provision of multimedia information, a data reception and transmission module, a data storage module, a sub-verification module are combined or integrated user object, fashion l data processing, which is suitable for the work and training of an artificial neural network, and the method includes the following stages: - in the APC-K calibration of eye parameters is carried out while providing the subject with test information in the form of moving visual stimuli; - download multimedia information to a remote APK-VK and classify its fragments into structural elements according to features - text, image, table, etc.; - transfer classified information from the remote APC-VC to APC-K and provide it to the subject; - receive a video stream with the help of a video recording device, according to the data of which the data of eye movements and/or gaze are recorded in the APC-K recording module, taking into account behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in the coordinates of pre-marked points on on the face, determine separate points of view at each moment of time in the coordinates of the plane of the multimedia information display module of the APC-K during the provision of multimedia information; transmit the received information to the remote APC-VC, in which: - establish correspondence between individual points of view and the currently provided fragment of multimedia information; - with the use of an artificial neural network, data sets of views are formed and the probability of the relationship of each individual view of the subject to reading is established; - aggregate the indicated data of a set of views into a coordinate system of classified structural elements of the provided multimedia information; - calculate the data density of all views and the data density of views defined as reading by the number in each structural element, and set appropriate threshold values for the calculated densities that delimit statistical noise; - calculate the ratio of the overlapping area of the area, where the data density of all views and the data density of views defined as reading exceeds the threshold values, to the total area of all structural elements defined in the provided multimedia information, and establish the amount of recognized information by the subject, by which it is determined information as read. 2. The method according to claim 1, which is characterized by the fact that the multimedia information provided to the subject is an electronic document, text fragments without images and/or tables or with images and/or tables in known formats for presenting electronic documents and/or video information containing text fragments and/or images and/or tables. C. The method according to claim 1 or 2, which differs in that it includes an additional step, in which, with the help of a verification module, before providing multimedia information to the subject, the identity of the subject is verified on the basis of visual data about the subject, which are entered into the remote APC -VK. 4. The method according to claim 3, which differs in that the subject's identity is verified before each subsequent provision of multimedia information or an electronic document to the subject. 5. The method according to any one of claims 1-4, which is characterized by the fact that the calibration of eye parameters and determined behavioral reactions is carried out before each session of providing multimedia information to the subject Zo. 6. The method according to any one of claims 1-5, which is characterized by the fact that the multimedia information display module is, in particular, a TV screen, a computer monitor, a smartphone display, a tablet display or any other device made of possibility of providing multimedia information. 7. The method according to any one of claims 1-6, which is characterized by the fact that the video recording device is, in particular, a web camera or a video camera that is connected or integrated in the APC-K, in particular in a computer, smartphone, tablet or any video device designed to receive and transmit video data. 8. The method according to any one of claims 1-7, which is characterized by forming a map of the density of views and/or the data density of views defined as reading and/or the overlap area of the region where the data density of all views and the data density of views defined as reading exceeds the threshold values, and does not necessarily provide multimedia information on the display module. 9. The method according to any one of claims 1-8, which is characterized by the fact that the percentage of viewed structural elements is calculated from the total number of structural elements when providing multimedia information, in which the view was not defined as reading, and optionally provided on the module display of multimedia information. 10. The method according to any one of claims 1-8, which is characterized by the fact that the percentage of structural elements in which the gaze data was determined as reading is calculated from the total number of structural elements when providing multimedia information, and is optionally provided on the module display of multimedia information. 11. The method according to any one of claims 9-10, which is characterized by calculating the ratio of the volume of viewed structural elements in which the view was not determined as reading to the volume of structural elements in which the view data was determined as reading, when provision of multimedia information and optionally provide multimedia information display on the module. 12. The method according to any one of claims 1-11, which is characterized by the fact that after the completion of the session of providing multimedia information or its fragment in the data processing module of the APC-VK, a reading verification certificate is formed, in particular as an electronic document, and optionally 60 stored in the data storage module. 13. The method according to claim 12, which differs in that the reading verification certificate is uploaded to the APK-K. 14. The method according to any one of claims 1-13, which differs in that all received data of the subject is stored in the data storage module of the APC-VK and optionally in the APC-K. Automatic recognition system of all information and DANCE SAVC-VK that » ! C "05 so y / y Gi network and NEK Sh.! . . And and 5 at 1, vi 039 MORE and more! x i STILL g | And и и , ив шЩ po) you and You still SHE still SHO ; TEO OIV BROSH ROV PROCESS YAROVO WITH YOUR COMMUNICATION IN VVOBU VBOSU BK AS WRKSHYA sha NN and aloe she dressed ma 095 23 1 claim 28953 omeyu her o 8a 2 2 ni vza eh MAV 0 2 R o30300 3966 I have 0 re Bo Nam Z GU 56 I i chv3Io 335 OZ v 026357 Ko o oo en yiv GU KI - VROBSHX, ERSKUU "coordinates of the screen, relative to the resolution of the screen, zagoryaim A " VROSHX, VRU - coordinates of the screen, relative to the resolution of the screen, vyagorstm V - EROSTNI - idemtidnkator of fixation i FPLOGD - duration of fikotz and achei i BOG. FPOGV - validity of formation 6 VC - idenzatsinkatov panmatsnya Fig. 2 in the paradise of data pt Z ; There are o Reflected data from I: ze V. Ha y CE Zach MI SCHI SHNEK EE NI IZ i KI I. her EI EE» Ne KK EE EE KK NN ZHE NE KK KA | ОО and А TA наче коках Ziyao shcha MTU - Fig. WITH Hansality of temporal "eras Beh Z Chatvynya ooraz5 ЧО хажёс Ne чвтанны оаюЗо VE . PI o PEJO shchih TH 0lobgo o SHE OVU Z SCHO 0 ZheKoevoe povo SHE SH LAK oo SHE Z po Mo a ih - aloe her same To VODNE nie OM de She s ooo 00 SS BONN in MKK SHE Eh o.2obo5 o o s no IS . . by Z PEN in X Koch KE ZO OH b.od00o - KT M 0 2 JAK 5 vVo0 NO ak Fig. 4 of these things. nk Shchove, MYUU of the most catfish; sei (Mene. 100, b. shot 1: Makhrovve Z shum. | (eve Yu.M pah razvnaiy | MiIkhROVNNYAII fetr - t y i ery: | (Mele 50. 36) with. shu: | (ove, 50, 36) siviYA U ST front sem r OmMore TA Y r of yarok (Mem. TT ah, ROOVNU 2. MahROIVTSO. fen early, sob: r (Mien 35.72) and myself. ХХ. copy of Z SOIKIV nene then rmak r Move. 223 And sovy Me 25) mah fchayscheny A OMakhRasVIhiYu eru ron r Men, KA Y believe: from ave, 15, 14) own Clearly - 7 rik Z Mome. 15, ZAY) ! I | Mope 122158 ivakh rosiivziy A Makhkoovvo Yu s - pe t woman and willingly itet eto echeyue cheno y; rats | bove o. Shmaneu 1 BIANeD. Ref. t YaMete. TAY N and Sri Pope, 1724) and there is. Yuelee and t i ok ro (chere, Fig. 5 4 corrective saccade from the reverse side. . A of the inter-line transition / INTER-LINE disturbances is T, dynamic fixation of the short-circuit flight: ; pl g z shi regressive |i R schyi and nipple and this is January АХ and tevlamiya that progressive saccade At я- very short (50 me) saccade fixations She zov 1s t Os The main characteristics of eye movements during reading, known since the times of /auv! Fig. 6