UA141318U - AUTOMATED METHOD OF VERIFICATION OF ELECTRONIC DOCUMENT ACQUISITION - Google Patents

Abstract

Automated method of verification of acquaintance of the subject with the electronic document, includes preliminary establishment of the volume of information recognition, in particular the volume of the read document sufficient for formation of the certificate of signing of the electronic document by the verified subject, verification of the subject on the basis of previously entered (APC) visual data about the subject at the beginning of the session of providing the subject with an electronic document and during each subsequent session of providing the same document to the same subject, automated determination of the scope of information recognition, in particular the volume of the document read by the verified subject providing an electronic document for all sessions of providing the same document in the agro-industrial complex using eye tracking technology and using an artificial neural network, and if the subject confirms the end of the session / sessions of acquaintance with the document and agrees to sign and if determined The amount of information recognized by the verified subject, in particular the volume of the read document, is equal to or greater than the previously set volume of the document signing certificate.

Description

The utility model belongs to technologies that involve means and methods of tracking the movement of subjects' eyes and refers to an automated way of verifying the subject's familiarization with an electronic document. This useful model makes it possible to establish whether the subject was familiarized with the visual information presented to him, in particular textual information and/or pictorial information, in what volume, and whether this volume was sufficient to confirm the subject's familiarization with the content of the text fragments of the provided multimedia information , whether the subject has read the presented information, in particular whether the subject has read the text information, in particular the information of the electronic document.

A useful model allows you to objectively establish whether the specified subject has read the text information and in what volume, and whether this corresponds to the specified volume, which is established as sufficient for the subject's perception of the content of the presented text fragments in the provided multimedia information.

In addition, the useful model refers to the method of verification of familiarization with the electronic document, in particular, involving the method and/or system of automated determination of the reading volume or recognition by the subject of specified areas when providing multimedia information.

Eye tracking technologies (oculography, eye tracking) include the determination of certain characteristics of eye movement, in particular, the determination of the point to which the gaze is directed, or the coordinates of the point of intersection of the optical axis of the eye (eyeball) with the plane of the specified object, in particular the plane of the device on which a certain visual stimulus is provided/presented.

The use of eye tracking systems (eye tracking or oculography) in various fields, and in particular for determining the subject's reading, is well known.

In (5 2012/0237084 A1, 20.09.2012, a method and system for identifying text and its position in visual media content, such as a document displayed on a computer monitor or other display device, allowing to determine whether the subject interacted with the text, the level of interaction of the subject with the text, for example, whether the subject looked at the text, whether the subject read the text, whether the subject perceived the text and understood it, or other types of interaction of the subject with the text. based on data obtained from an eye tracking device, and an emotional response detection system may be applied separately or together.The method includes identifying a part of visual media content that constitutes text, determining the position of the identified text of the visual media content, collecting data from the eyes of sub subject while viewing visual media content, regarding pupil size, blinking and gaze coordinates, generating subject gaze patterns based on collected eye data hey, determining whether the subject interacts with the identified text based on at least the determined position of the identified text and the generated gaze pattern, determining the level of interaction of the subject with the identified text, in particular determining whether the subject has read or familiarized himself with the identified text, and determining the subject's emotional response to the identified text.

There is a known method of confirming the reading of an electronic message by the user (using the technology of tracking the position of the eyes during the reading by the user on an electronic device of an electronic message containing many lines, in which the number of lines of the message viewed by the user is determined based on the signs of the position of the eyes, in particular, the determination of the coordinates of the position of the eyes, and whether the user has read the message by comparing the number of lines viewed by the eyes against a threshold number of lines based on the number of lines in the email.In addition, whether the user was surprised while viewing the email is determined by comparing the amount of time the eye spent on the email message, with the expected duration of time for displaying the provided electronic message (05 2016/0094705 A1, 31.03.2016).

Known methods include the use of specialized equipment to track eye movements, which cannot always be provided outside of laboratory settings and research facilities, and especially in everyday life. When studying eye movements, even with the involvement of specially adapted equipment, there is a question of data selection for further processing and interpretation. For example, in the case of covert attention observed in eye movement recordings, when the gaze trace and fixation points pass by the real object of attention with only a small fixation time, there are difficulties in establishing an unambiguous relationship between the results of eye movement research and the cognitive process of a specific subject

Although various eye tracking systems and their applications are known, there are still certain problems that people face when reading, and everyone solves these problems in a unique way: rereading complex parts of the text, looking for unknown words, writing down details to remember etc. Automating these processes with the involvement of publicly available and commonly used equipment, for example, to monitor human reading, assess speed, distinguish eye movements and behavioral responses during reading from other activities, comment on fragments of text that are difficult to read or whose content was not recognized, etc. and., are functions that can be extremely useful for people who have to process huge amounts of text every day.

Currently, there are no solutions for standardized verification and confirmation of the process of processing visual information, and in particular reading, with high accuracy based not on specialized hardware, but on any video devices or web cameras involving the principle of tracking eye movements.

The problem, which is solved within this useful model, lies in areas where decision-making significantly depends on whether the subject (person) has actually familiarized himself with the presented information, in particular, read a certain document. The presented useful model allows its application in such fields as education, legal practice, public sectors, etc.

The purpose of this useful model was to provide an automated way to verify the subject's familiarity with an electronic document, and to estimate the extent of the subject's recognition of information and the extent of reading with high accuracy when viewing multimedia information on various devices, such as a personal computer or laptop, tablet , smartphone, virtual screen or any other suitable display device or medium.

The useful model allows in real time to check the subject's familiarization with the presented information, to determine its scope, to determine the subject's recognition of text sections, tables, images and other when providing multimedia information, and the amount of information, the content of which was recognized by the subject , in particular, check the reading of text parts in the presented multimedia information or the reading of any electronic document.

The useful model is based on the task of providing an automated way of verifying the subject's familiarization with an electronic document with high volume accuracy

From the recognition of information by the subject when providing multimedia information, in particular the amount of readable information, it is even better to determine the amount of read information, using widely distributed electronic devices suitable for displaying multimedia information, video recording, processing, storage and transmission of data, as well as to ensure an automated method of signing electronic documents upon establishing that the verified subject has read the specified volume of the specified document.

In the first aspect, the useful model refers to an automated method of verifying the subject's familiarization with an electronic document, which includes preliminary establishment of the scope of information recognition, in particular the scope of the read document, which is sufficient for the formation of a certificate of signing of the electronic document by the verified subject, verification of the subject's identity based on visual data about the subject previously entered into the hardware and software complex (APK) at the beginning of the session of providing the subject with an electronic document and during each subsequent session of providing the same document to the same subject, determining the amount of information recognition, in particular, the amount of information read of the document by a verified subject when providing an electronic document for all sessions of providing the same document using eye tracking technology on the APC and using an artificial neural network, and in the event that the subject confirms the completion of the session/sessions of familiarization with the document and agrees signing, and if the specified volume of information recognized by the verified subject, in particular the volume of the read document, is equal to or greater than the pre-set volume, a document signing certificate is formed.

Determining the scope of information recognition, in particular the scope of the read document by a verified subject in the method of signing an electronic document, can be carried out using an automated method of determining the scope of information recognition, while the method is carried out using a system that includes: - user hardware and software complex (APK- K), in which a video device, a multimedia information display module, a data reception and transmission module, a recording module with a video device of video stream data, a data storage module, a data processing module are combined or integrated, and - a remote hardware and software complex of interaction with the user (APK-VK ), in which a multimedia information display module, a data reception and transmission module, a data storage module, a user subject verification module, a data processing module suitable for the operation and training of an artificial neural network are combined or integrated, and the method includes the following steps : - calibration is carried out at the APC-K taking into account the individual anatomical parameters of the eyes when providing the subject with moving visual stimuli and tracking tasks; - download multimedia information to the remote APC-VK, classify fragments into structural elements according to features - text, image, table, etc., transfer the classified information from the remote APC-VK to the APC-K and present it to the user; - according to the data of the video stream from the video device on the APC-K, they record eye movement and/or gaze data and, taking into account behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in the coordinates of pre-marked points on the face, determine individual gaze points in each moment of time in the coordinates of the plane of the multimedia information display module of the APC-K during the provision of information, and transmit the received information to the remote APC-VK in which: - establish the correspondence between individual points of view and the fragment of multimedia information provided at this time, - with the use of artificial the neural network forms data from a set of views and establishes the probability of the relation of each individual view of the subject to reading; - aggregate these sets of views into a coordinate system of classified structural elements of the provided multimedia information; - calculate by the number in each structural element the density of all views and the density of views defined as reading, and set appropriate threshold values for the calculated densities that delimit statistical noise; - calculate the ratio of the overlapping area of the area, where the data density of all views and the data density of views defined as reading exceeds the threshold values, to the total area of all structural elements defined in the provided multimedia information, and establish the amount of recognized information by the subject, by which it is determined information as read.

In another aspect, the model is useful in the way of signing an electronic document

To determine the amount of information recognition, in particular, the amount of a document read by a verified subject, is carried out using an automated system to determine the amount of information recognition, which includes: a) the hardware and software complex (APK) of the user (APK-K), which is connected or to which integrated: i) a video recording device, i) a multimedia information display module, i) a data reception and transmission module, suitable for receiving and transmitting the received information to a remote user interface (APK-VK), i) a recording module from a data video recording device video stream related to eye movement and/or gaze, and behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in the coordinates of pre-marked points on the face, m) data storage module, mi) data processing module, including : - video stream data processing unit from the video recording device APK-K, - eye parameter calibration unit when providing multimedia information, - you unit the value of individual points of view at each moment of time in the coordinates of the plane of the multimedia information display module during the provision of information, and b) a remote user interface (APK-VK), in which the following are connected or integrated: i) a multimedia information display module, and ) data reception and transmission module suitable for reception and transmission of received and processed information on the APC-K, i) data storage module, i) user subject identity verification module, m) data processing module suitable for work and training of an artificial neural network, which includes: - a block of work and training of an artificial neural network, - a block of classification of fragments of multimedia information into structural elements according to 60 features of text, images, tables and others,

- a unit for processing video stream data from the video recording device of the APK-K, - a unit for establishing correspondence between individual points of view and the currently provided fragment of multimedia information; - a block for establishing the probability of the relation of each individual view of the subject to reading, - a block for forming data of a set of views and aggregating data of a set of views into a coordinate system of classified structural elements of the provided multimedia information, - a block for calculating the number of data density of all views and density in each structural element data views defined as reading, and setting appropriate threshold values for the calculated densities delimiting statistical noise - a unit for calculating the ratio of the overlap area of the region where the data density of all views and the data density of views defined as reading exceeds the threshold values to the total area of all structural elements defined in the provided multimedia information, - a block for setting the volume of information recognized by the subject, - a block for setting the volume of textual information read by the subject - a block for providing statistical information and generating reports.

List of figures

Fig. 1 - Automated system for determining the amount of information recognition, where (1) APC-K, (10) a video recording device, (20) a multimedia information display module, (30) a data reception and transmission module, (50) a recording module from a video recording device of video stream data ) coordinates of the plane of the multimedia information display module; (2) APC-VC, (21) multimedia information display module, (31) data reception and transmission module, (41) data storage module, (61) user subject identity verification module, (71) data processing module, suitable for operation and training of an artificial neural network, (81) unit for operation and training of an artificial neural network, (91) unit for classification of multimedia information fragments into structural elements, (101) unit for processing video stream data from the video recording device APK-K,

From (111) a block for establishing the correspondence between individual points of view and a fragment of multimedia information provided at this time, (121) a block for establishing the probability of the relationship of each individual view of the subject to reading, (131) a block for forming the data of a set of views and aggregating the data of a set of views in the coordinate system of the classified structural elements of the provided multimedia information, (141) the block for calculating the data density of all views and the data density of the views defined as reading by the amount in each structural element, and setting the corresponding threshold values for the calculated densities, (151) the block for calculating the ratio of the overlapping area of the area , where the data density of all views and the data density of views defined as reading exceeds the threshold values, to the total area of all structural elements defined in the provided multimedia information, (161) unit for setting the volume of information recognized by the subject, (171) unit for setting the volume read sub' object of textual information, (181) block of providing statistical information and generating reports.

Fig. 2 - Indicators in time series.

Fig. C - Filtering using VROSU, EROSMU.

Fig. 4 - Filtering only eye movements on the screen.

Fig. 5 - Convolutional Neural Network architecture with 113,006 training parameters.

Fig. 6 - Classification of each group of fixations (observations grouped by EROSIV).

Available methods and systems for tracking eye movements for various purposes involve the use of specialized video devices - eyetrackers. As a rule, attention is focused on determining the exact coordinate positions of the smallest structural elements of the text, in particular words, even better letters, and on tracking trajectories and basic movement patterns. There are certain limitations to the use of eyetrackers depending on the method on which the system is based, in particular dark pupil or light pupil, and the significant influence on the obtained data of certain behavioral reactions.

The known methods do not offer methods for remote processing of eye tracking data obtained from non-specialized eye tracking devices, in particular, using a standard web camera, which can be defined for example as web tracking, or any video device attached or integrated to any personal electronic device suitable for receiving, processing, transmitting and,

it is not necessary to save data, where this method has a high accuracy of determination and is easily accessible to the user.

For the purposes of determining the amount of information recognized and/or read by the subject without the involvement of specialized equipment, this useful model proposes to combine an automated system of hardware and software configured to perform certain functions, the use of an artificial neural network (using the training of which template models were created at the initial stage determining the density of gaze in certain structural elements of multimedia information, in particular an electronic document, which later allowed to achieve 96 95 accuracy in determining the amount of recognition of the provided multimedia information and determining the amount of read information), and a certain sequence of their application, which together provided a tool for verifying the familiarization of sub object with an electronic document.

In addition, in the process of familiarization with an electronic document, the subject has the opportunity to track and store all reading materials and associated reading metadata, to search and filter the content by reading template parameters (for example, the time and speed of fixation allow to single out an area of interest within the text); receive various analytical annotations after processing, receive real-time help options when familiarizing with the content of multimedia information, including reading (for example, automated translation). The system automatically tracks eye movements and matches them with information, such as text on the screen, allowing it to process reading patterns in real time and store all metadata for later analysis.

To solve the problem of computer-aided reading verification in the claimed utility model, it was necessary to convert a common off-the-shelf video device, in particular a web camera, into an eye tracking device. For these needs, machine learning was applied to build a system for eye gaze tracking and recognition of specific eye movements during reading, which, together with the introduction of new elements and improvement of known ones, as well as due to new connections and a defined sequence of actions, made it possible to effectively solve the problem of reading verification .

Implementation of a useful model includes the use of conventional video devices, in particular

From connected, built-in or integrated into any electronic devices, for example, a tablet, smartphone, laptop, personal computer or other suitable, which requires the creation of an adapted hardware and software complex (APC), where the selection of data for video recording is also important, the method video recording, the method of their processing, transmission and selection of criteria for interpretation.

At the beginning, work was carried out on the analysis of the subject's visual activity, where reading/non-reading was classified according to the gaze, regressions, interline transitions (5meyer5) and saccades were detected, and a set of individual gazes was mapped on structural elements of multimedia information/document. As a suitable tool for this, the program was used (in particular, as described, Vopota; V. Spaib5 M., MeiIpuspik U. Maspipe I eatipd

Tesppidnez ip Veadipu TgasKipd. Meei usyg (IZTOVMI OOK)I / AV. Vopyota?, M. Snatsv, U. MeIpusnyk. -

Veepimeogyi Asadetu apa AvOiare rapypegei om/yy OayaNKooi iar5. - Osi 19, 2018 porz//teaiiyt.sot/FVeepimeogagoMir/taspipe-Ieagtipd-Iespppidpez-ip-geadipa-igasKipd-teei-uoig-aioguisok-bas7bbavazZa), for the creation of which mostly open source libraries were used.

The details of all patents, published applications and references cited in this specification are incorporated by reference in their entirety.

Data collection

When collecting data to track human activity, a tracker was used, for example

SaeRoiipi eu, which made it possible to obtain gaze coordinates with an error angle of no more than 1-1.5 degrees after calibration. During each session, the SaeRoipi Appaiuziv program records video and on-screen images along with eye movement tabular data.

The entire data set consisted of 2 parts: 51 reading time series and 85 "non-reading" time series. Each research participant had to perform the following actions: read a 2-minute text; find specific information and things in the pictures, watch a 3-minute video.

The obtained time series consists of several columns (Fig. 2): - ЕROSХ, ЕРОУ - screen coordinates, relative to the screen resolution, algorithm A - ВROSХ, VROSХ - screen coordinates, relative to the screen resolution, algorithm B - ЕРОСИбВ - fixation identifier - FPOGD - the duration of fixation of the eyes because - BPOGV, FPOGV - the validity of the information

- VKIO - flashing identifier

Data preprocessing and feature selection

In the course of our research, we discovered that the tracked coordinates may not be perfect. Blinking, moving the head, changing lighting - all these factors interrupt or spoil the flow of data. Therefore, additional steps were taken that could alleviate the situation, smoothing the movement of the gaze was adopted as the main option, although the value of this approach has limits. Smoothing eliminates one important feature - microsaccades. Saccades are rapid, simultaneous movements of both eyes between two fixation points. Microsaccades are movements within a single fixation that respond to how users fixate their gaze. Although smoothing is not an ideal choice when it comes to saccades, it helps to approximate word detection.

This is what filtering meant for these research purposes (Figs. 3, 4): - filtering using VROSM, EROSMU, - filtering only eye movements on the screen.

To easily manipulate the data set and the training/gesture models, a time sequence processing window of 100 values width (which roughly corresponds to the average time to read one plane on A4 paper) was chosen. This resulted in splitting all data sets into 24,568 read series and 14,288 "non-read" series of multiple time series with a length of 100 values, allowing for 90 95 overlap.

Classification of reading/non-reading.

Next, three main methods of time series classification were used. Three sets of features were created: linear trend detection for EROSH, seasonality, and general characteristics, but the selected features were not informative and did not describe the data well.

Next, a convolutional neural network was applied. The functions ah, au were used. After some tuning, the optimal architecture was found with 113,006 training parameters, see Fig. 5. This model gave 96 95 accuracy on a subset of tests and was subsequently selected as the baseline model for further research.

Reading clustering models

The main task was to classify each group of fixations (observations grouped by EROSION) as one of three main patterns: saccades, interline transitions, regressions (Fig. 6).

The main obstacle encountered at this stage was the labeling of the data set, as 60 Hz data is inherently difficult to label. This also proved to be a problem for clustering. Some of the minor issues that have been addressed are high similarity between regressions and line transitions, and scrolling freezes (5sgoiIipo) that were outliers. To exclude fixations during scrolling, a reading classification algorithm was used.

The entire dataset was filtered from spikes to saccades only. To obtain saccade data, points were grouped by fixation identifier (EROSION) and only the last observation from each group was selected. As a result, all identified saccades were divided into minimum/maximum values with a naive algorithm along the horizontal axis, and minimum saccades were divided into groups of interline transitions and regressions. To achieve the necessary results, we used clustering using the K-means method on three main functions of saccades: the projection of interline transitions along the horizontal axis, the angle of interline transitions along the horizontal axis, and the difference from the previous saccade.

One of the examples of automated processing of multimedia information is the creation of a complete text document from a video in the following sequence: identification of static frames in the video (with a threshold deviation between the shots), separation of sheets (text fragments) from the frame, subsequent merging of sheets into a single "panoramic" image , separating text into a displayed document, identifying points of interest.

As a result, it was possible to obtain a machine learning model capable of predicting with 97 95 accuracy whether the user has read/not read the text within 1.6 seconds of recording.

Some of the main findings of the study show that an algorithm based on the predictions of a previous machine learning (ML) model can account for gaze movement (regressions, interline transitions (zmeerz) and saccades) and calculate relative reading speed; an algorithm that provides information about the reader's interest provides a weighting factor for a given word that may be important to the reader.

Scientists who study human behavior such as eye tracking can uncover previously blocked areas in health technology and business. As justified by this algorithm, because it is possible to make real-time predictions based on gaze-tracking technologies, and perhaps go beyond this with more scientific applications that no one thought possible before.

Determining the general parameters of reading without clearly separating that part of the text information that was actually read does not allow to predict or establish with high accuracy the amount of recognized content of text information among all multimedia or all text information, the difficulty of its assimilation, separation of particularly important content fragments, etc.

With the involvement of the above, an automated system was created for better implementation of the determination of the scope of information recognition by the subject when providing multimedia information, in particular determination of the scope of reading by the subject of structural elements of multimedia information or an electronic document defined as text. The selected parameters for targeted training of the artificial neural network allow the following determination of the probability of the relation of each individual view of the subject to reading. At the same time, according to the spatial and temporal characteristics of eye movements, taking into account behavioral reactions, it is possible to divide sections of text fragments of multimedia information into groups of "careful reading", "review or familiarization reading", "scanning reading" and "scrolling" and to separate sections for further analysis text fragments highlighted as "careful reading" or own reading.

The automated method of determining the amount of information recognition by the subject is carried out as follows.

Using the eye calibration unit (90) in the APC-K (1), calibration of the eye parameters is carried out taking into account the individual anatomical parameters of the eyes when providing the subject with moving visual stimuli and tracking tasks. Calibration can be carried out both once and several times as needed, in particular with a certain frequency or before each session of providing multimedia information to the subject.

The connection of the subject's interaction with multimedia information is confirmed by verifying the subject's identity in module (61), which is carried out before providing the subject with multimedia information on the basis of visual data about the subject that is entered into the remote

APC-VK, one-time or several- or repeatedly, for example with a certain periodicity or

Before each subsequent provision of multimedia information to the subject, or after interrupting the viewing session for a predetermined time before each session of providing multimedia information after such an interruption.

Identification/verification of the subject's identity can be done by entering data about the subject into the database of the data storage module of the corresponding device in advance or when the subject first applies to the APC-K.O. In subsequent sessions of the subject's application to multimedia information about which provides for determining the scope of recognition and/or reading in any specified mode, compares the subject's parameters with those available in the database during the entire period of the subject's information processing session or sessions. During authorization in the system at the first application, in order to simplify the identification/verification mechanism and reduce the time for identification, a system of logging into the system using existing accounts, in particular social network accounts, etc., including audio and or photo and/or video recorded data may be applied object, in particular its appearance, such as facial features, etc.

After calibration, or before it, or during calibration, using the data reception and transmission module, suitable for receiving and transmitting information (31), upload multimedia information to the remote APC-VK (2) and classify it using the fragment classification unit (91) fragments into structural elements according to features - text, image, table, etc. (all other information other than text, images, or tables).

Next, the classified information is transferred from the remote APC-VC (2) to the APC-K (1) and provided to the subject on the multimedia information display module (20).

Multimedia information can be displayed on any suitable display medium, such as a TV screen, a computer or laptop monitor, a smartphone display, a tablet display, etc.

Using the video recording device (10) of the video stream, the recording module (50) of the APC-K (1) records eye movement and/or gaze data taking into account behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in coordinates beforehand marked points on the face, and in the data processing unit of the video stream (80) of the data processing module (70) with the help of the individual point determination unit (100), individual points of view are determined at each moment of time in the coordinates of the plane of the multimedia information display module (20)

APC-K (1) during the provision of multimedia information.

It is important that any video device capable of receiving and transmitting video data is used as a video recording device, in particular a web camera or a video camera connected or integrated into the APC-K (1), in particular it can be used connected to a personal electronic device (personal computer, phone, smartphone, tablet or other) photo and/or video camera or present, built-in to said device, or otherwise.

The received information is transmitted in any known way (wired or wireless) with the use of data reception and transmission modules (30) and (31) to the remote APC-

VK (2) and then to the data processing module (71).

With the use of the work and learning unit of the artificial neural network (81) and the matching unit (111), the correspondence between the individual points of view and the currently provided piece of multimedia information is detected, then with the use of the artificial neural network of the unit (81) the data sets of views are formed and with the help of the probability setting block (121), the probability of the relationship of each individual view of the subject to reading is determined.

The specified data of the set of views are aggregated into the coordinate system of the classified structural elements of the provided multimedia information in the block of formation and aggregation of the set of view data (131).

Further, in the data processing module (71), the data density of all views and the data density of views defined as reading are calculated in the calculation unit (151) according to the number in each structural element, and appropriate threshold values are set for the calculated densities that delimit statistical noise.

Additionally, in the data processing module (71), optionally in a separate block associated with blocks (81)-(181), determine the number of transitions from the specified text to accompanying materials, in particular tables and/or images, and the time that is spent on such transitions that are taken into account when determining behavioral reactions when reading and "not reading".

Then, in the area overlap area ratio calculation block (151), the ratio of the overlap area of the area where the data density of all views and the data density of views defined as reading exceeds the threshold values to the total area of all structural elements defined in the provided multimedia information is determined, and set the amount of information recognized by the subject using block (161) of setting the amount of information recognized by the subject, after which the read information is determined using block (171) of setting the amount of text information read by the subject.

Multimedia information to be processed by the subject is any type of information suitable for display on various devices and may contain text or its fragments (in particular, for example, inscriptions, numbers), tables, images, text fragments without images and/ or tables or with images and/or tables in known electronic document presentation formats, for example, is an electronic document, an electronic book (for example, a production document, educational material, electronic testing material, entertainment material, etc.) and/or video information containing text fragments and/or images and/or tables. After processing in block (91), the multimedia information loaded in the APK-VK is classified by separating from the entire volume of content fragments containing text features, which are text fragments, tables (with text elements and numbers) and images, in particular, that associated with text and/or tabular fragments.

Multimedia information is presented to the subject in a conventional format, but preliminary classification allows to focus on behavioral responses and eye movements in relation to target parts of multimedia information.

The remote APC-VC also provides the ability to present the results of data processing regarding the amount of recognition and/or reading of multimedia information on the user's display device (20) in a variety of ways. The scope and type of provision of this data may be determined by the operator of the remote APC-VC and/or the owner of this multimedia information and/or the user or otherwise.

In particular, in the block for providing statistical information (181), data on the behavior of the subject during reading is generated, in particular, the volume of information read, reading speed, the complexity of information perception, data on various aspects of eye movement, and further, on the basis of this data, the data is generated a report, in particular a proof of reading certificate, in particular as an electronic document,

at the end of each session or series of sessions of providing multimedia information or its fragment.

In particular, the multimedia information display module (20) and/or (21) provides a generated report and/or map in any visually acceptable form, and/or diagram, and/or graph or other, of views density and/or data density of views defined as reading and/or the overlap area of the area where the data density of all views and the data density of views defined as reading exceeds the threshold values, and/or the calculated percentage of viewed structural elements out of the total number of structural elements when providing multimedia information, in in which the gaze was not defined as reading, and/or the calculated percentage of structural elements in which the gaze data was defined as reading, out of the total number of structural elements in the provision of multimedia information, and/or the calculated ratio of the volume of viewed structural elements in which the gaze was not defined as read, to the extent of structural elements in which the view data was defined as read, when provided multimedia information, etc.

Such visualization of the results of recognition and/or reading of multimedia information and the speed of these processes can occur both at the end of the session of providing multimedia information and during the session at the end of viewing certain defined fragments of multimedia information in real time. This is an additional tool for controlling the process of recognition and/or reading for the user and/or the owner of multimedia information and/or the operator of the separate APC-VC and develops into a number of additional capabilities of the method and system described here.

The application of a machine learning approach makes the claimed method of checking the familiarity with multimedia information and checking the reading easily applicable and standard on many devices. Thanks to the declared method and system in real time using any means of video recording, it is possible to detect with 97 95 accuracy whether a person is reading at the moment and calculate the amount of information actually read.

The data obtained in this way regarding the volume and/or speed of recognition and reading of information allow to expand the possibilities of using the methods and/or systems described here, based on the technology of tracking the movement of eyes, for various purposes.

So, taking into account the possibility of clear identification/verification of the subject's identity in connection with specific multimedia information, for example, an electronic document that he views, recognizes and reads, there is a possibility of independent automated confirmation of familiarization with the document by this person and confirmation of the amount of information read , which allows us to state with a high degree of probability the depth of cognitive processing of information by a specific subject. This is useful in various educational processes, in the assessment of certain professional skills, assessment of remote interaction between personnel within the same enterprise or between different structures, in particular the public sector, in legal practice, etc. The connection between the person of the subject, the specific electronic document that he has read, and the amount of content read in this document allows for automated and easily accessible certification of the read document, in particular by creating a document signing certificate, which can be applied in many industries.

Data exchange between APC-K and APC-VK is carried out using wired or wireless transmission, while setting up APC-VK allows you to expand the system and carry out such data exchange with multiple APC-K, and also ensures a reduction of the functional and software load on APC-K , all received subject data are stored in the data storage module (41)

APC-VC and optionally part of the data, in particular reports and/or maps and/or certificates - in APC-

K.

While the described embodiments have been based on preferred embodiments, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the embodiments covered by the appended utility model formula.

Claims (2)

USEFUL MODEL FORMULA 1. An automated method of verification of the subject's familiarization with the electronic document, which differs in that it includes preliminary establishment of the amount of information recognition, in particular the volume of the read document, which is sufficient for the formation of a certificate of signing of the electronic document by the verified subject, verification of the subject's identity based on visual data about the subject previously entered into the hardware and software complex (APC) at the beginning of the session of providing the subject with an electronic document and during each subsequent 60 sessions of providing the same document to the same subject, automated determination of the scope of information recognition, in particular the volume of the document read by the verified subject when providing an electronic document for all sessions of providing the same document in the APC system using eye tracking technology and using an artificial neural network, and in the event that the subject confirms the completion of the familiarization session/sessions with the document and p approves the signing and if the determined volume of information recognized by the verified subject, in particular the volume of the read document, which is equal to or greater than the preset volume, forms a document signing certificate. 2. The method according to claim 1, which differs in that the determination of the scope of information recognition, in particular the scope of the read document by the verified subject, is carried out using a method in which a system is used, which includes: user hardware and software complex (APK-K), in which is connected or integrated with a video recording device, a multimedia information display module, a data reception and transmission module, a recording module from a video recording device of video stream data, a data storage module, an automated data processing module, and a remote hardware and software complex of interaction with the user (APK-VK), in which a module for visual presentation of multimedia information, a module for receiving and transmitting data, a module for storing data, a module for verification of a user subject, a module for processing data suitable for the operation and training of an artificial neural network are combined or integrated, and the method includes the following steps: APC-K performs the calibration of the eye parameters when the subject is given a test eye information in the form of moving visual stimuli; upload multimedia information to a remote APK-VK and classify its fragments into structural elements according to features - text, image, table, etc.; transfer classified information from the remote APC-VC to the APC-K and provide it to the subject; receive a video stream with the help of a video recording device, according to which data of eye movements and/or gaze are recorded in the APC-K recording module, taking into account behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in the coordinates of pre-marked points on the face, determine separate points of view at each moment of time in the coordinates of the plane of the display module of the multimedia information of the APC-K during the provision of multimedia information; transmit the received information to the remote APC-VC, in which: they establish correspondence between individual points of view and the fragment of multimedia information provided at this time; with the use of an artificial neural network, data sets of views are formed and the probability of the relation of each individual view of the subject to reading is established; aggregate the specified data of a set of views into a coordinate system of classified structural elements of the provided multimedia information; calculate by the number in each structural element the data density of all views and the data density of views defined as reading, and set appropriate threshold values for the calculated densities that delimit statistical noise; calculate the ratio of the overlapping area of the area where the data density of all views and the data density of views defined as reading exceeds the threshold values to the total area of all structural elements defined in the provided multimedia information, and set the amount of recognized information by the subject by which the information is determined as read Q. The method according to claim 2 or 3, which differs in that the determination of the scope of information recognition, in particular the scope of the document read by the verified subject, is carried out using a system that includes: a) the hardware and software complex (APK) of the user (APK-K ), in which the following are connected or integrated: i) a video recording device, i) a multimedia information display module, i) a data reception and transmission module, suitable for receiving and transmitting the received information to a remote user interface (APK-VK), i) a module for recording from a video recording device video stream data related to eye movements and/or gaze, and behavioral reactions in the form of at least eye blinking and/or head movements, and/or changes in the coordinates of pre-marked points on the face, m) a data storage module, mi) data processing module, which includes: 60 video stream data processing unit from the video recording device APK-K, eye parameters calibration unit when providing multimedia information, a unit for determining individual points of view at each moment of time in the coordinates of the multimedia information display module plane during information provision, and b) a remote user interaction API (API-VK) in which connected or in which integrated: i) multimedia information display module, i) data reception and transmission module suitable for receiving and transmitting received and processed information on the APC-K, i) data storage module, i) user subject identity verification module, m ) a data processing module suitable for the operation and training of an artificial neural network, which includes: a unit for operation and training of an artificial neural network, a unit for classifying fragments of multimedia information into structural elements based on the characteristics of text, images, tables, etc., a unit for processing video stream data from the device video recording of the APC-K, a block for establishing correspondence between individual points of view and the currently provided fragments entom of multimedia information; the unit for establishing the probability of the relationship of each individual view of the subject to reading, the unit for forming the data of a set of views and aggregating the data of a set of views into the coordinate system of the classified structural elements of the provided multimedia information, the unit for calculating the number in each structural element of the data density of all views and the data density of the defined views as a read, and setting appropriate threshold values for the calculated densities delimiting the statistical noise, a block for calculating the ratio of the overlap area of the region where the data density of all views and the data density of the views defined as a read exceeds the threshold values to the total area of all structural elements defined in the provided multimedia information, the block for setting the amount of information recognized by the subject, the block for setting the amount of textual information read by the subject, the block for providing statistical information and generating reports. APC eat ApK-Vy S MN i vk shen ki 3) SCHHI ! Network! p and I and others | also Head) Ivo 90) I | , E | ! and ! ! ! | and 1 E Fig. 1 TMESOTTM EN NINKOFYAYA BROBH DUTTER PROCESSING BROBO RRO ROB VROSH VROSU VROSU VK VKOSHK VKRMM fot v ob bis nebez ro Nyuv 6 v v Z nlayu omab oo Zb 0360 2nd bob o398 1 a what 2 2 nav omav oyu neo 06 and 1 oyu 1 th that 20 Z liu omeyu ZK OB 28 o azi Ne 1 in 50 Ru and ZO og OZ MOB OB a o 363 B 1 8 in Ky i VROTXH, EROSGU - coordinates of the screen, relative to the resolution of the reduction vlgopim A x VROPH, VROETU - coordinates screen subtracted from the separate giltness of the screen, the algorithm IN 6 ARMOR - identifier of fication and FPOGD - zrizhaliet eye fixations 8 BLOGS, FIG - obruptovanit information - VZHY - goes afihator of infused Figs. 2 Moving information display: zeyuk frud dyakIHh p Z Я Z i, «а т ЕВ "й | tet Filtered press шки КА НЕНЯ ШИ ЕК НЕ В ЖЕК; ЕК А У ЕВКВ ща BOTH IN MU MOVO ME IV ro. 5 BOM KK OH M TE UZH KK: 358 KN KV Z ze 5 I NN ENN z oh i. KZ 3 O Z Z I U BV Z E EE KN NK Her 1 and I vz No OUR Eh - shi shchi - puta pk pah pyv Virgin Mass Fig. C Duration of frequent series where IOM Readings 0ldo3a. Not itching 000030 V. . A SCHA i oo0025 in: | With p. х УМХ ШЭ Х. АХ оз 5 ШИ ШК штова2о ОЗ ШК . With and I about. O.O. r ЖЕЛЖШШЕНИИНИМЕЙН МИХ КЕ y; and p.oooto! oh oh ; voucher -0 ON and 2000 4900 8000 BO 10000 32005 Fig. 4 In the battle of VaI 1 er Z) bMevde, 198, 3) sopHiYA HO SODY. zennnnknreanekhnalnnkkadnnknaknnuchnkh oshshcrobye r (I am 10.) And ions. and More, MK Z mi: rosy z. MakHKOOVIVI district - -- M om: r bMape, 50. KO (Mene, 50, Z svysii Mchch O i ee ZT su i shen: | (Krve, xO, T i nah roobveii 5 Mahrovi - - i t (Mane . UM I SOMY: candle Z (Mene. 13 2 I I and oshrn in Monet. 33. 2553 tah rosiy g mahroonniaTO - - ROB y (Mole, v, 255) o o pl t zhlyutvtyutyutyut mya m myaichna iryb | (Move, 5. 288) Give 1: DVK rennya vtttntnntnnetnntya iy ro (Move 172) nama: More 1725) fors g doromyh began 7 svemny: Z KMene, 3728) I shryb Monet, 173) ska GK pephe one pi novels Sh ori iZole. 2 Fig. 5 4th corrective saccade: after the reverse 74 interline transition / Interline transitions 3 t and 1 and pnVamicny. - ; divamichnii Ffiksatsi / pet I flight ron: and write I ron Z zv | She from 7 is regressive. From their U and v a i sho progressive sakkala Shk i- very short (50 me) fixations of the sakhada Sh-- t-! you and I , same shk i E i : I -U, la 0s The main characteristics of eye movements during reading, known from the numbers /ауа! Fig. 6