RU2541922C2 - Method and device to determine look direction - Google Patents

Abstract

FIELD: measurement equipment.

SUBSTANCE: invention relates to the field of I&C. The method consists in setting mutual location of devices that detect images of the object of interest, which perform consecutive or synchronous detection of the object of interest in different spectral ranges. They produce an image of a whole object with a certain background with the help of one detecting device having sensitivity in the remote infrared range, on this image they separate the object itself from the background, and also separate individual segments of the object and localise them on this image. Similar segments of the object are localised on images produced from other detecting devices, which are sensitive in ordinary visible and/or near infrared rage, or regardless of the first detecting device they align these detecting devices in space relative to the first detecting device so that they detect images of object segments localised on images from the first detecting device. They calculate orientation of object segments and direction of the look line.

EFFECT: reduction of noise introduced by external sources, at the stage of image registration.

17 cl, 3 dwg

Description

The invention relates to a control and measuring technique and can be used to measure the dynamics of eye movement in the process of performing cognitive tasks by a person, as well as to implement attention-sensitive interfaces, eye-brain-computer interfaces, in systems that communicate between people with motor impairments functions.

State of the art

The technique relates to eye movement recording systems. The mechanisms of perception and active processing of visual information have been widely studied in the framework of applied cognitive sciences for more than fifty years [5, 4]. In the course of these studies, stable relationships between the dynamics of eye movements and the parameters of perception of visual information by a person (the so-called depth of attention) were described [4, 5, 6]. These patterns made it possible to implement methods for measuring visual attention, as well as recognizing user intentions based on an analysis of oculomotor activity [4, 5, 8, 14]. This opens up the prospect of creating a number of unique devices that provide an interface for human interaction with technical systems through gaze. The scope of such systems is, for example, remote control of autonomous robotic systems and technological processes. Measurement of visual attention makes it possible to predict the ability of the user to respond in a timely manner to changes in the space surrounding him [4]. The purpose of such systems is to take into account the human factor (for example, assessing the state of an operator involved in complex technological processes, a driver of a vehicle, etc.). The introduction of such systems in vehicles is a particularly urgent task due to the large number of victims of car accidents in Russia [7].

Despite the considerable amount of accumulated knowledge in the field of cognitive sciences, the creation of devices that would fully realize the above possibilities presents a difficult technical problem. Traditional methods of recording eye movements are contact, which limits their use in a wide range of applied problems. The sensors are mounted in such systems directly on the subject’s face (for example, in the electrooculography method) [22]. Many years of experience using traditional methods allowed us to formulate the basic requirements for such systems in general:

1. The possibility of non-contact, remote registration of the movement of the pupil;

2. Maximum freedom for the user's natural movements;

3. The use of passive registration methods (without involving special highlighting or user training).

The latter requirement is due to the need in some cases to conduct long-term observations of a person, while the presence of illumination imposes significant time limitations on such measurements (due to the accumulation of radiation dose). Partial achievement of the above requirements became possible only with the advent of detection devices based on matrix photodetectors (CCD, CMOS), where the main emphasis is on image processing. They include the detecting device itself (one or more), as well as a personal computer that provides real-time processing of the received images. Image processing algorithms at the same time implement complex multi-stage mathematical methods, among which the face and eyes of users in the image are usually sequentially searched, the pupil is searched, its outline, the user's spatial position is calculated and the line of sight is calculated. To date, several commercial systems are available that have the architecture described above [9, 10]. Despite significant advantages over traditional contact methods, they still have a number of significant disadvantages. The accuracy of measurements in real conditions depends on changes in the gradient and the level of external illumination. The technical capabilities of a number of detecting devices make it possible to partially compensate for these conditions. In the process of registering images, adaptive adjustment of the exposure time is performed depending on the exposure of the photosensitive matrix (tent mechanism-English shutter). This avoids saturation of the elements of the matrix photodetector and maintains the image contrast when changing the level of illumination. However, it is not always possible to fully compensate for external factors and maintain constant contrast with the help of a tent mechanism. This greatly affects the operation of image processing algorithms, which are extremely sensitive to image contrast. Today, this is the main factor affecting the operation of systems. An example is the registration of the face and eyes of the driver in night traffic conditions, where the headlights of oncoming cars create uneven differences in illumination. The intensity of oncoming traffic can lead to the inability to carry out these measurements. An increase in the contrast of images containing the user's face and eyes in low light conditions or during sharp changes in some systems is achieved by illuminating the face with low-intensity radiation in the near infrared range (700-950 nm). This range is not perceived by the human eye, but corresponds to the spectral sensitivity range of CCD-based devices. However, in the field, the corresponding component of external powerful light sources (for example, headlights of oncoming cars, sunlight) also contains the corresponding component, which affects the noise of the video data.

The solution proposed in the framework of this application provides a significant reduction in the influence of external illumination and the gradient of illumination from external sources at the stage of image registration. This is achieved by expanding the spectral range of registration of user images. It is proposed to register in the visible (spectral range <1000 nm) and far infrared (wavelength ≥3000 nm). The low noise level in far-infrared images is explained by the absence of a corresponding spectral component in most external lighting sources surrounding a person in everyday life (in the same sources where it is present, its effect on image quality is negligible). Images in the indicated ranges are recorded sequentially or simultaneously. Comparison of image data allows you to accurately determine the area of the face and eyes of the user, regardless of the position and intensity of external sources. Until recently, the technical implementation of such systems was impractical due to the imperfection of matrix infrared photodetectors. Basically, these were devices providing an acceptable level of signal / noise at low temperature. As a cooler, liquid nitrogen or helium was used. In this regard, the binding of these devices consisted of a cooling system (a microbolometer with gas circulating inside) and did not allow miniaturization of the device. However, recent advances in the development of these technologies and the emergence of uncooled miniature matrices for this range [19] made it possible to use them in a wide range of problems. The absence of the need for additional cooling devices allowed the creation of miniature portable imaging devices in the far infrared range. The combination of far-infrared images and the usual visible range at the registration and processing stage in aitracking systems can significantly reduce the influence of external noise and, as a result, increase the measurement accuracy.

Overview of analogues

A technical solution is known [20], in which the object is illuminated by several radiation sources that form a light pattern on the surface of the object. As an object, the user's eye is considered. The images of the object are recorded, also detecting the reflection of the light pattern. The object is detected by processing the obtained images containing the light pattern and the image of the object. The position of the pattern determines the spatial position of the object. The main disadvantage of the system is the presence of active illumination, as well as the detection of images in the spectral range of the emitter. Registration is performed in the visible and near-IR range (in the case of a far-IR emitter, thermal damage to the object itself can be caused, and therefore, with regard to the described problem, the use of such radiation sources does not make sense). In addition, the presence of external illumination in the indicated range can lead to noise of the signal and a significant decrease in the accuracy of measurements. The main difference of the proposed solution described above is the absence of any additional radiation sources. Image registration is performed not only in the visible, but also in the far infrared range. The absence of external sources of noise in the far infrared range in most cases provides an acceptable match in the signal / noise level in the resulting images. This leads to higher image contrast and more stable processing algorithms.

A solution is known [21], in which the object is registered by detecting devices. The object is highlighted, which is considered the user's face by several low-intensity sources of the near infrared range. Object images are detected using detection devices. The resulting images are binarized and analyzed by simultaneously scanning the image in the vertical and horizontal directions. A search is made for bright pixels in the image, which corresponds to the area of the object. Based on the bright areas in the image, the object is localized. The authors do not indicate the spectral range of the radiation source, however, as noted above, only the visible and near infrared ranges (≤1000 nm) are of practical importance. Image registration is also carried out in the spectral range of the emitter (to obtain the contrast of bright pixels against dark ones). The disadvantage of the system is still the possible noise of images by external sources, which can reduce the contrast of the signal reflected from the object. The proposed solution does not have any additional emitters. Image registration is performed in several spectral ranges, in particular, in the far IR. In this case, the contrast of the object is ensured by the object's own radiation, does not depend on external sources of radiation, and is a constant value.

A solution is known [11], in which the images of the object are detected and its orientation is calculated. It is proposed to increase the accuracy of calculations by eliminating the delay at the stage of transmitting the registered image of the object to the processing device. This solution can increase the stability of the systems, in particular when switching the tent mechanism of the detecting device. However, the proposed solution does not allow filtering out the noise and localizing the object itself directly from its images. Thus, the essence of the present invention is to eliminate the video transmission delay described above, while the present invention provides a solution to reduce noise in the step of registering images while simplifying the processing of video data. Such a combination is possible due to the shift of the image registration part to the far infrared range.

Also known is the solution [12], in which the registration of images of the object is carried out, the processing of these images with the subsequent localization of the object in the images. The main disadvantage of this solution is that the detection is in the visible range. This leads to the noise of the received images from external exposure. The proposed solution within the framework of this application shifts part of the measurements to the far infrared range, where the number of noise sources is much smaller than in the visible range. This allows you to achieve a relatively low noise level at the stage of image registration.

Also known is the solution [13], in which the object is illuminated by radiation sources. The radiation is formed into a pattern. The object is detected by detecting devices, followed by image data processing. The object is localized on the images with the subsequent installation of the spatial position of the object. The main disadvantage of the system is the detection of images in the visible range, as well as the presence of additional illumination, also produced in the visible spectral range. In the presence of external noise illumination, the spectral range of which coincides with the spectral range of the additional radiation source, the contrast of the pattern will be low. Similarly, the contrast of the object will also be low for stable processing algorithms. The main difference between this system and the proposed solution is the active exposure of the face and eyes of the user. In the present invention, registration is performed in a passive mode. In the far infrared range, the user's face itself is a source of thermal radiation.

Closest to the proposed method for registering the face and eyes of the user is the invention [1]. The paper describes a method for determining the spatial position of the head, as well as its orientation. The method is based on methods for reconstructing the spatial structure of an object from a series of ordinary two-dimensional images taken from various angles (using methods of epipolar geometry) [2]. The method consists in installing several detecting devices oriented at different angles with respect to a certain area of space. The registration of the user's head inside this space is performed. Then, for each image, a sequential determination of the anatomical features of the head is performed (features extraction), the features present on all images are searched, the configuration of the detection devices using epipolar geometry is calculated, and the head is reconstructed spatially with its spatial position calculated [2]. The main difference between this work and the present invention is the registration of all images in the visible range. As noted above, such systems are most sensitive to external exposure. In the present invention, the registration of images is spectrally spaced into the visible and far infrared ranges with various processing algorithms. In the proposed solution, registration is performed both in the far IR and in the visible spectral range. The position of the object of study and its individual segments is determined by the IR image using conventional threshold methods (binarization of the image with a fixed or floating threshold).

Closest to the proposed solution is [3]. The work proposes a system where the object is located on images obtained from a detecting device having sensitivity in the visible range by threshold processing. The difference proposed in this invention, the solution is to register the image in the far infrared range. This ensures a low level of noise when determining the face and eyes of the user in conditions of changing illumination. In addition, due to low noise levels (the sensitivity of detecting devices in the far IR range is about 0.1 degrees), the proposed method allows the use of simpler processing procedures. As such procedures, a simple image binarization is proposed.

Thus, in all the above solutions, the main problem remains the sensitivity of the systems to external noise exposure. The presence of such a flare significantly reduces the contrast of images (both in the case of active systems and passive), which does not allow measurements of the directivity of the eye and the dynamics of eye movements.

Thus, the technical problem to be solved is to ensure the reduction of noise introduced by external sources at the stage of image registration. There is an opportunity to effectively solve this problem at the stage of registration of images of the object. This can be achieved by recording part of the images in the spectral range, in which there are practically no external sources of radiation that introduce noise illumination into the images (as well as sharp short-term differences in illumination, the presence of a gradient of illumination, or intense illumination leading to the operation of the photodetector near the saturation zone). Separation of the signal registration by different spectral ranges allows to significantly simplify the processing algorithms for separating the object from the background and improve the localization of individual segments of the object.

The technical problem is solved by the proposed set of essential features.

A method for determining the directivity of the gaze, which consists in installing at least two pre-calibrated detecting devices for imaging an object in space (DU), at least one of which registers images in the far infrared with a spectral sensitivity range> = 3000 nm, and the rest images are recorded in the visible range of 400-1000 nm, while the remote control that records images in the IR range is fixed, all the remote control images are recorded in the region of space corresponding to the image for the object, the registration results are processed using BSOI, while first recording images of part of the remote control space with sensitivity in the far infrared range with a wavelength> 3000 nm and obtaining IR images of the object and some background behind it, after which the resulting IR image is processed, separating the object from the background and localizing the areas on the image corresponding to the image of the object, separate segments of the object are distinguished on the localized part of the image, also localizing them on the image, after which it is determined the coordinates of these segments are determined in the coordinate system attached to the image, the images of a part of the remote control space with sensitivity in the visible range of 400-1000 nm are recorded, the coordinates of the segments obtained during registration in the IR range are recalculated into the coordinates attached to images with remote control having sensitivity in the visible region, after which the corresponding segments of the object are localized in the images of the visible range, and the directivity of the line of sight is calculated for the case if the IR range is visible of the range are fixed relative to each other, or the coordinates of the segments obtained during registration in the IR range are recalculated into the orientation coordinates of the visible range remote control, after which the visible range remote control is oriented and the part of the space containing the images of the selected object segments in the IR images is recorded and based on the resulting images calculate the directivity of the line of sight for the case if the remote control of the IR range and the visible range are movable relative to each other.

Wherein:

- The remote control is calibrated by converting the coordinates of the position of the object and / or its segments in the images received from the remote control of the IR range into the coordinates of the position of the object on the images obtained from the remote control of the visible range, in case all the remote control are fixed relative to each other.

- The remote control is calibrated by converting the coordinates of the position of the object and / or its segments in the images received from the IR range into the orientation coordinates of the visible range relative to the IR range to register object segments localized on IR images if the visible range are mobile relative to the remote control IR range.

- image registration of part of the space in the infrared and visible range is carried out simultaneously.

- image registration of part of the space in the IR and visible range is carried out sequentially.

- first they process the IR image, separating the image from the background by analyzing the gradients of the thermal fields.

- after separating the object from the background, the position of the individual segments of the object in the image is determined using an analysis of the gradients of the thermal fields.

- after determining the position of the segments in the IR image, these segments are found in the visible range images using calibration data, for the case if all the remote controls are fixed relative to each other.

- after determining the position of the segments in the IR image, based on the calibration, the visible range remote control is oriented in such a way that they register a space region containing the images of the object segments localized on the infrared images, in case the visible range remote control is movable relative to the IR range;

- calculate the directivity of the line of sight based on images obtained from the visible range remote control, displaying the processing results on a visually perceptible medium.

- the face of the user is determined as an object, and the eyes of the user as segments.

The technical result is also achieved by a combination of features set forth below.

A device for determining the directivity of the gaze, including at least two means (DU) located in the space detecting images of an object, at least one of which registers images in the far IR range (spectral sensitivity range> = 3000 nm), and the rest record images in the visible range (spectral sensitivity range 400-1000 nm), while the remote control that records images in the IR range is fixed and all the remote control connected to the unit for collecting and processing information BSOI.

Wherein:

- The visible range remote controls are fixed.

- Remote controls of the visible range are mounted movably, while the actuator providing mobility is connected to the information collection and processing unit.

- Remote controls are set so that their fields of view coincide in the plane of movement of the object.

- The remote controls are set so that their field of view does not match the plane of movement of the object, while the field of view of the remote control of the IR range is greater than the field of view of the remote control of the visible range.

- BSOI collects images from all remote control simultaneously or sequentially, performs image processing, produces a control signal for the spatial orientation of the visible range remote control, calculates the directivity of the line of sight.

When calculating the direction of a person’s gaze, the pupil area and the iris of the eyes are contoured by methods [17] and, using the analysis of the projection of this contour, the direction of the line of sight is calculated, for example, similarly to [18].

The method and apparatus implementing this method are disclosed in the detailed description below, with reference to the accompanying figures:

Figure 1 - Relative location of the remote control, where: 1 and 3 - remote control of the visible range, 2 - remote control of the far IR range, 4 - actuators changing the orientation of the remote control, 5 - field of view 1 (P31), 6 - field of view 2 (P32), 7 - plane of movement of the object;

Figure 2 - Stages of processing an image obtained from a television remote control to identify the eye area: a) image of a person registered with a remote control having sensitivity in the far infrared range, with a cooler background, b) the result of threshold image processing with a dynamic threshold, c) points belonging to the boundary of the object (marked by cross markers) and an ellipse inscribed in the given points; d) the found eye areas according to the temperature gradient (taken out of the face image);

Figure 3 - Switching the remote control with a data acquisition and processing unit (BSOI), where: 8 - an image collection and processing unit, 9 - a remote control switching channel with a BSOI, 1 and 3 - visible range remote control, 2 - far infrared remote control.

Figure 1-Figure 3 and in the text used the following notation:

DU - detecting device;

BSOI - information collection and processing unit;

IR - infrared.

Method Description

1. An image remote control is installed that has a sensitivity range in the far IR range (spectral sensitivity range> 3000 nm). The position of the remote control is fixed in space. As these devices, for example, thermal imaging cameras FLIR (Sweden), CEDIP (France), etc. can be used;

2. Additionally, two or more remote controls are installed that have sensitivity in the usual visible range (400-1000 nm). In this case, conditions are provided technically under which the orientation of the visible range remote control relative to the infrared remote control is always known. As such remote control, television cameras based on CCD or CMOS silicon technology can be used, for example, Videoscan (Russia);

3. The remote control of the visible range can be fixedly installed relative to the remote control of the IR range, while their orientation is such that the field of view of the remote control of the infrared range and the field of view of the remote control of the visible range match the maximum in the plane of movement of the object, as shown in FIG.

3.1. Pre-calibrate all remote controls. For the case of a fixed arrangement of all the remote control units relative to each other, the purpose of calibration is to obtain a transformation of the coordinates of the position of the image element obtained from the remote control of the IR range into the position coordinates of a similar image element obtained from the remote control of the visible range.

3.2. The described calibration can be performed by placing a screen perpendicular to the optical axis of the infrared remote control, with markers printed on it, perceived by all the remote control simultaneously. In this case, the distance from the screen to all the remote controls, as well as the relative position of all the remote controls, is known. A screen image with markers applied by all the remote controllers is recorded.

3.3. By comparing the obtained images, a projection matrix is obtained:

β = A * γ, where β is the coordinate vector of the point in the image of the far infrared range, γ is the coordinate vector in the image of the visible range or, in the case of movable remote controls of the visible range, the orientation vector of the remote control from paragraph 2 relative to the remote control from paragraph 1. A is a matrix of mappings of coordinates β to coordinates γ. Such a procedure, for example, can be carried out by solving the system of equations β = A * γ for each marker according to the method described in [2].

3.4. After calibration, the images of the region of space are recorded using the IR remote control and the visible range remote control at the same time.

3.5. An operation is performed to separate the object from the background using an image obtained from the IR range. At this stage, the property of the object to emit thermal radiation is used. In this case, the background is considered on the image as a less bright area. Due to this, the search for an object can be carried out by analyzing the gradients of thermal fields in the image. The analysis may consist in finding the warmest areas of the image and extracting these areas from the general, colder background.

3.6. For example, such processing can occur by sequentially sorting through all the pixels of the image obtained from the IR range. Pixels with an intensity exceeding a certain threshold are selected as candidates for the area of the object. Pixels with an intensity below the threshold are remembered as belonging to the background image.

3.7. The threshold value can be selected dynamically depending on preliminary knowledge of the surface temperature of the human body. For example, it is known that usually the surface temperature of an object (e.g. skin) varies in the range of 18-26 degrees.

3.8. The threshold can be selected dynamically depending on the histogram of the image. The methods that can be used at this stage are given, for example, in [15].

3.9. Further, a combination of pixels with a high intensity value can be performed. Such a union in the field can be performed, for example, using the method of relationships [16]. At this point, pulsed noise can be cut off (individual high-intensity pixels surrounded by low-intensity pixels).

3.10. Further, the contouring of areas including pixels with high intensity can be performed. An analysis of the shape of these areas, for example, can be done by correlation analysis relative to the reference image of the face. Areas with a low degree of correlation can be attributed to the background of the image, while areas with a high correlation with the standard can be considered as the object itself (in our case, the user's face / faces) (Figure 2, image b)).

3.11. Geometrical figures can also be selected as a reference form for the analysis of the obtained image areas. For example, if it is known that an object (for example, a face) has an oval shape in most cases, then the object can be outlined by inscribing an oval into the outline of the area containing pixels with high intensity (Figure 2, image c)).

3.12. Next, the orientation of the object in the image plane is calculated (in our case, the orientation of the head). Such a calculation can, for example, be done by finding the maximum length between opposite sides within the area and the corresponding minimum length (which corresponds to the distance from the chin to the forehead and between the cheekbones, respectively). The orientation and position in the image coordinates is remembered.

3.13. Next, the individual segments of the object are determined. Segments are considered the user's eyes. Localization of segments in the image can, for example, be carried out by dividing the area of the image occupied by the object into two subregions. In our case, the eye area is sought in the upper subregion.

3.14. Segments can also be found by analyzing the temperature gradient. Thus, a priori known information about segment gradients can be used. In our case, the segments (for example, the eyes) have the lowest temperature in comparison with the adjacent areas, which is explained by the presence of the mucous surface of the sclera that shields the thermal radiation (Figure 2, image d).

3.15. The segments thus obtained are localized in the image obtained with the IR range.

3.16. The coordinates of the detected segments are determined on the image obtained with the IR range. Coordinates are attached to the image itself. These coordinates are remembered.

3.17. The coordinates of the segments detected in the image with the remote control from paragraph 1 are recalculated into the coordinates of the images obtained from the visible range remote control. Recalculation is performed using the matrix determined at the calibration stage described in clauses 3.1-3.3. The result is the coordinates of the segments in the coordinates attached to the remote control of the visible range.

3.18. Using the obtained coordinates on the image obtained from the visible range remote control, localize the image sections corresponding to the segments determined using the image with the infrared remote control.

3.19. Further on the images obtained with the visible range remote control, a further calculation of the directivity of the gaze is performed. The procedure can be performed, for example, in accordance with [18].

4. The remote control of the visible range can be set movably relative to the remote control of the infrared range, while their orientation is such that the field of view of the remote control of the infrared range and the field of view of the remote control of the visible range overlap, but do not match. In this case, the visual field of the remote control of the visible range is less than the field of view of the remote control of the infrared range.

The corresponding configuration is shown in Figure 1, position 5 for the IR range, Figure 1, position 6 for the visible range;

4.1. Similarly, item 3 calibrate. The purpose of calibration for the case of the movable orientation of the remote control of the visible range relative to the remote control of the infrared range is to obtain a transformation of the coordinates of the position of the image element of the space region obtained from the remote control of the infrared range, into the coordinates of rotation of the remote control of the visible range relative to the remote control of the infrared range, in which the region of space displayed on this pixel falls into the center of the image obtained from the visible range remote control;

4.2. Calibration is carried out, for example, as follows. A screen is mounted perpendicular to the optical axis of the IR remote control. The points that fall into the field of view of the IR range are determined (the corresponding field of view is shown in Figure 1, position 5), but lying on the border of the field of view of the IR range (in horizontal and vertical azimuths). The visible range remote control is pointed at these points so that the set points are in the center of the field of view of these remote control. Remember the rotation angles for each remote control of the visible range relative to the remote control of the IR range. Similarly, the rotation angles for the visible range remote control are determined for the point of intersection of the optical axis of the remote control of the infrared range with the plane of object movement (Figure 1, position 7).

4.3. All intermediate positions of the rotation angles for the visible range remote control relative to the infrared remote control are determined taking into account the positions defined in clause 4.2.

4.4. Image is recorded using the IR remote control.

4.5. Perform processing operations similarly to paragraphs 3.4.-3.16.

4.6. The coordinates of the object segments are recalculated into the angles of rotation of the remote control of the visible range relative to the remote control of the infrared range, taking into account the calibration data obtained in clause 4.2.

4.7. The visible range remote control is oriented using the data obtained in step 4.7. Register images.

4.8. Perform image processing and calculation of the line of sight similar to 3.19.

Device description

1. A device that implements the described method consists of a remote control registering images of a certain region of space in the far infrared range (> 3000 nm) (Figure 1, 2, position 2).

2. The device further includes one or more remote control devices that register images in the visible spectral range (400-1000 nm) (Figure 1, 2, positions 1 and 3),

3. All the remote control are connected to the unit for collecting and processing information, BSOI (Figure 3, position 8). BSOI provides processing of images received from all the remote control, and generates a control signal for actuators orienting the remote control of the visible range, if they are movable relative to the remote control of the IR range (Figure 1, position 4). The control signal, as well as the video signal from all the remote controls, is transmitted to the BSOI via the communication channel (Figure 3, position 9).

4. Remote control IR range, as well as remote control of the visible range are installed stationary relative to each other. The fields of view of the IR range and the visible range are the same (Figure 1, position 5).

4.1. Images from all remote control are registered simultaneously. In this case, all images are synchronously transmitted to the BSOI (Figure 3, position 8) via communication channels (Figure 3, position 9).

4.2. First of all, the processing of the image obtained from the remote control IR range (Figure 1, position 2). The treatment may, in particular, be carried out by the methods described in the method of the present invention. In the process of processing, the object is separated from the background with the subsequent determination of specific segments on this object.

4.3. The position of the detected segments is localized in the images obtained from the IR range. The coordinates of these objects are determined in the coordinate system attached to the image with the IR range.

4.4. The coordinates of the detected segments are recalculated from the coordinate system tied to the image from the remote control of the infrared range, to the coordinates tied to the image from the remote control of the visible range. The operation is performed in accordance with the calibration data obtained in step 3-5 of the method.

4.5. Localization of image segments linked to the coordinates of the image obtained from the visible range remote control is performed.

4.6. The direction of view is calculated similarly to that described in the method of the present device and similarly to [21]. The result is displayed on a visually perceptible medium.

5. Remote control of the visible range are set movably relative to the remote control of the IR range. In the latter case, the actuators orient the remote control of the visible range relative to the remote control of the IR range according to the control signal (Figure 1, position 4).

5.1. The field of view of the remote control of the visible range is smaller than the field of view of the remote control of the IR range (the field of view of the remote control of the visible range corresponds to Figure 1, position 6, while the field of view of the remote control of the infrared range corresponds to Figure 1, position 5).

5.2. The IR remote control images are recorded, the data is transmitted to the BSOI (Figure 3, position 8).

5.3. Perform image processing in BSOI similarly 4.2.-4.4.

5.4. Recalculate the coordinates of the image segments obtained from the remote control of the IR range in the orientation of the remote control of the visible range relative to the remote control of the IR range.

5.5. A control signal is generated and transmitted from the BSOI (Figure 3, position 8) to the actuators (Figure 1, position 4). The remote control is visible in the visible range.

5.6. The registration of images of segments of the remote control object of the visible range.

5.7. The direction of view is calculated similarly to that described in the method of the present device and similarly to [21]. The result is displayed on a visually perceptible medium.

Information sources

1. Patent US 8339459. Zhang Z., Sankaranarayanan A., Zhang Q., Liu Z., Cai Q., Multi camera head pose tracking, Microsoft Corporation (Redmond, WA).

2. Hartley R.I., Zisserman A. Multiple View Geometry in Computer Vision [Text] .- Cambridge University Press, 2004.

3. Patent US 8224108. Steinberg E., Prilutsky Y., Corcoran P., Bigioi P. Digital Image Processing Using Face Detection Information, Digital Optics Corporation.

4. Velichkovsky B.M., Comelissen F., Geusebroek J.-M., Graupner S.-Th., Hari R., Marsman J. B., Shevchik S. A., PannaschS. Measurement-related issues in investigation of active vision [Text] // Measurement with persons: Theory and methods / B. Berglund, G. B. Rossi, J. Townsend, L. Pendrill (Eds.). - London-New York: Taylor and Francis, 2011 .-- Pp. 281-300.

5. Pannasch S., Velichkovsky B.M. Distractor Effect and Saccade Amplitudes: Further Evidence on Different Modes of Processing in Free Exploration of Visual Images [Text] // Visual Cognition. -2009. - 17 (6/7). - Pp. 1109-1131.

6. Di Stasi LL, Renner R., Staehr P., Helmert JR, Velichkovsky BM, Canas JJ, Catena A., Pannasch S. Saccadic peak velocity is sensitive to variations in mental workload in complex environments [Text] // Aviation, Space, and Environmental Medicine, 2010. - 81 (4). - pp. 413-417.

7. Information on indicators of the state of road safety [Electronic resource] / State Traffic Safety Inspectorate of the Ministry of Internal Affairs of Russia. - Electron, given. - 2012 - Access mode: www.gibdd.ru/, free. - Zagl. from the screen. - Yaz. Russian

8. Majaranta P., Raiha K.-J. Twenty Years of Eye Typing: Systems and Design Issues [TeKCT] // Proceedings of ETRA 2002, Eye Tracking Research and Applications Symposium, 25-27, 2002.

9. SR Research Complete Eye Tracking Solutions [Electronic resource] / SR Research. - The electron. Dan. - Ontario, Canada, 2012. - Access mode: http://www.sr-research.com/free. - Zagl. from the screen. - Yaz. English

10. SMI SensoMotoric Instruments [Electronic resource] / SMI SensoMotoric Instruments. - The electron. Dan. - Teltow, Germany, 2013. - Access mode: http://www.smivision.com free. - Zagl. from the screen. - Yaz. English

11. Device for recording eye movement. A.S. USSR No. 1009418, class A61B 3/00 / Kurochkin S.L., Fomin B.M., Shif M.I. - publ. 1981.

12. A device for recording and analyzing the visual activity of the operator. A.S. USSR No. 1438690. Kurochkin S.L., Molodtsov V.A., Spetsakov A.I., Teterin N.V. - declared. 10/17/1986.

13. The method of determining the coordinates of the points of fixation of the gaze on the object of observation. A.S. USSR No. 1430027. Kurochkin S.L., Sidorov A.S., Fomin B.M., Shif M.I. - declared. 08/05/1986.

14. Velichkovsky B.M., Sprenger A., Pomplu, M., Unema P. Eye movements and encoding manipulations in two visual memory tasks [TeKCT] // Proceedings of the Psychonomic Society 38th Annual Meeting. November 20th-23rd - Philadelphia, USA, 1997 .-- P.73.

15. Kapur J.N., Sahoo P.K., Wong A.K. A new method for gray-level picture thresholding using the entropy of the histogram [Text] // Computer vision, graphics, and image processing. - 1985. -29 (3) - Pp. 273-285, 1, 2.5, 2.6.

16. Davies E. Machine Vision: Theory, Algorithms and Practicalities [Text]: - Academic Press, 1990 - Chap. 6.

17. Li D., Winfield D., Parkhurst D.J. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches [Text] // Computer Vision and Pattern Recognition - Workshops, 2005. CVPR Workshops, IEEE Computer Society Conference on, 25-25 June 2005. - pp. 79. - doi: 10.1109 / CVPR.2005.531.

18. Method and device for determining the spatial position of the eyes for calculating the line of sight [Text]: RF patent No. 2444275 / Shevchik SA, Velichkovsky BM, Velichkovsky BB, Vartanov AV - Publ. 09/10/10.

19. http://www.ulis-ir.com/

20. Method and installation for detecting and following an eye and the gaze direction therefore [Text]: patent US 7572008.

21. Eye position detecting apparatus [Text]: US patent 5218387.

22. Jin S., Lee K., and Hong K. "An Implementation of Multimodal Gaze Direction Recognition System using Image and EOG", 6th International Conference on Digital Content, Multimedia Technology and its Applications (IDC), pp.229-235 978-1-4244-7607-7, 2010.

Claims (17)

1. A method for determining the directivity of the gaze, consisting in the fact that at least two pre-calibrated detecting devices for detecting an object image in space (DU) are installed, at least one of which registers images in the far infrared range with a spectral sensitivity range> = 3000 nm, and the rest record the images in the visible range of 400-1000 nm, while the remote control that records images in the IR range is fixed, all the remote control records the image of the area of space corresponding to once an object, the registration results are processed using the BSOI information collection and processing unit, while first the images of a part of the remote control space with sensitivity in the far IR range with a wavelength> 3000 nm are first recorded and IR images of the object and some background behind it are obtained, after which they process the obtained IR image, separating the object from the background and localizing areas on the image corresponding to the image of the object, separate segments of the object are isolated on the localized part of the image, also localizing them in the image, after which the coordinates of these segments are determined in the coordinate system attached to the image, the images of part of the remote control space with sensitivity in the visible range of 400-1000 nm are recorded, the coordinates of the segments obtained during registration in the IR range are recalculated into the coordinates attached images with remote control having sensitivity in the visible region, after which the corresponding segments of the object are localized in the images of the visible range and the directivity of the line of sight is calculated For the case if the remote control of the IR range and the visible range are fixed relative to each other, or the coordinates of the segments obtained during registration in the IR range are recalculated into the orientation coordinates of the visible range remote control, then the visible range remote control is oriented and the part of the space containing the images of the selected segments is registered object on IR images, and on the basis of the obtained images calculate the directivity of the line of sight for the case if the remote control of the IR range and the visible range are movable relative to each other a. 2. The method according to claim 1, characterized in that the remote control is calibrated by converting the coordinates of the position of the object and / or its segments on the images obtained from the IR range, into the coordinates of the position of the object on the images obtained from the visible range, if all remote controls are fixed relative to each other. 3. The method according to claim 1, characterized in that the remote control is calibrated by converting the coordinates of the position of the object and / or its segments in the images obtained from the remote control of the IR range, into the orientation coordinates of the remote control of the visible range relative to the remote control of the IR range for recording segments of the object localized on IR images in the case that the visible range remote controls are movable relative to the IR range. 4. The method according to claim 2, characterized in that the image registration of part of the space in the infrared and visible range is carried out simultaneously. 5. The method according to claim 3, characterized in that the image registration of part of the space in the IR and visible range is carried out sequentially. 6. The method according to any one of claims 1 to 5, characterized in that the IR image is first processed, separating the image from the background by analyzing the gradients of the thermal fields. 7. The method according to claim 6, characterized in that after separating the object from the background, the position of individual segments of the object in the image is determined using an analysis of the gradients of the thermal fields. 8. The method according to claim 7, characterized in that after determining the position of the segments in the IR image, these segments are found in the visible range images using calibration data for the case if all the remote controls are fixed relative to each other. 9. The method according to claim 7, characterized in that after determining the position of the segments in the IR image, based on the calibration, the visible range remote control is oriented in such a way that they register a space region containing images of object segments localized in IR images, in case The visible range remote controls are movable relative to the IR range. 10. The method according to any one of claims 7, 8, characterized in that the directivity of the line of sight is calculated based on images obtained from the visible range remote control, displaying the processing results on a visually perceptible medium. 11. The method according to claim 3, characterized in that the face of the user is determined as the object, and the eyes of the user as segments. 12. A device for determining the directivity of the gaze, which includes at least two means (DU) located in the space detecting images of an object, at least one of which registers images in the far infrared range (spectral sensitivity range> = 3000 nm), and the rest register images in visible range (spectral sensitivity range 400-1000 nm), while the remote control that records images in the IR range is fixed, and all remote control are connected to the BSOI information collection and processing unit. 13. The device according to p. 12, characterized in that the visible range remote controls are fixed. 14. The device according to p. 12, characterized in that the visible range remote controls are mounted movably, while the actuator providing mobility is connected to the information collection and processing unit. 15. The device according to item 13, wherein the remote controls are installed so that their fields of view coincide in the plane of movement of the object. 16. The device according to 14, characterized in that the remote control is set so that their fields of view do not coincide in the plane of movement of the object, while the field of view of the remote control of the IR range is greater than the field of view of the remote control of the visible range. 17. The device according to p. 12, characterized in that the BSOI collects images from all the remote control simultaneously or sequentially, processes the images, produces a control signal for the spatial orientation of the visible range remote control, calculates the directivity of the line of sight.

Images