RU2712417C1 - Method and system for recognizing faces and constructing a route using augmented reality tool - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: this technical solution, in general, relates to computer engineering. Method includes the following steps: obtaining a video stream from one video stream source with a space area image; capturing video stream frames when one person's image is detected on them; determining the position and size of the face image on each captured frame; determining the image quality of the face image on the captured frames, on which the face image size is higher than the predetermined threshold values; selecting a face image with a maximum image quality of the face; calculating a vector characterizing features of the face image, based on the selected face image; comparing the calculated vector with the stored image vectors in the database (DB); based on the results of comparing the calculated vector with the stored vectors, assigning to the person an access parameter which determines the permission to be in the space region; displaying access parameter to image display device.

EFFECT: reduced time of people search in places of mass congestion of people or large multi-storey buildings, in which these people should not be located, and higher accuracy of persons identification.

11 cl, 4 dwg

Description

FIELD OF TECHNOLOGY

[001] This technical solution, in General, relates to the field of computer technology, and in particular to methods and systems for facial recognition using augmented reality glasses with an integrated camera or using a complex of systems that provide video surveillance.

BACKGROUND

[002] Currently, there is a problem of identifying criminals among a large crowd of people, for example, in public places or at public events. Security guards are not always able to identify the perpetrator without using additional equipment. This may be due to various factors, for example, visually a person’s appearance can change greatly over time due to the growth / shaving of a beard, changes in hair style or eye color, etc. There may also be a human factor - fatigue, which leads to dispersal of attention.

[003] Biometric facial identification systems make it possible to recognize a person’s face, despite the aforementioned possible changes in appearance. There are systems that include augmented reality glasses with a built-in camera, integrated with a face biometric identification system, but such a system allows you to solve the problem locally, only at the location of the security officer. In rooms with a large area, for example, in stadiums, such a solution is not effective. Analysis of the prior art allows us to conclude that inefficiency and, in some cases, the impossibility of using previous technologies, the disadvantages of which are solved by the present invention.

ESSENCE OF TECHNICAL SOLUTION

[004] A technical problem or task set in this technical solution is to create a reliable method for biometric facial recognition using augmented reality tools, as well as a system for implementing the method. An additional task is the visual display of classified people on the augmented reality glasses display and the construction of a route to the detected object using indoor navigation technologies.

[005] The technical result achieved by solving the above technical problem is to reduce the time people search in crowded places or large multi-storey buildings in which these people should not be, and to increase the accuracy of face identification. After detecting such people, the claimed system allows you to build a route to the indoor segment in which the fixed person is located, which reduces the time of detention.

[006] The specified technical result is achieved through the implementation of a method for recognizing faces in a video stream, performed by at least one computing device, comprising the steps of: receiving a video stream from at least one source of the video stream with an image of a region of space;

- capture frames of the video stream upon detection of images of at least one face on them;

- determine the position and size of the face image on each captured frame;

- determine the quality indicators of the image of the face on the captured frames, on which the size of the face image is higher than the predetermined threshold values;

- choose a face image with a maximum face image quality indicator;

- calculate a vector characterizing the features of the image of the face, based on the selected image of the face;

- comparing the calculated vector with the stored image vectors in the database (DB);

- based on the results of comparing the calculated vector with the stored vectors, an access parameter is defined for the person that determines the permission to be in the space region;

- outputting the access parameter to the image display device.

[007] In one particular embodiment of the method, the image display device is an augmented reality device, the access parameter being output in augmented reality mode.

[008] In another particular embodiment of the method, an area of interest is additionally set for incoming frames of the video stream, while only those frames of the video stream are captured in which the face image falls into the specified area of interest.

[009] In another particular embodiment of the method, after capturing frames of a video stream upon detection of images of at least one face, they additionally perform the step of:

- determine the presence of the image of a person in the frame, the image of the face of which was detected, and to determine the presence of the image of a person in the frame, determine the presence in the frame: image of the shoulders in the corresponding face image of the area or image of the human body.

[0010] In another particular embodiment of the method, facial image quality indicators are determined based on at least one face parameter:

- rotation of the head (in degrees) along each of the three axes of rotation, tilt up / down relative to the horizontal axis;

- tilt left / right relative to the vertical axis;

- rotation about a vertical axis.

[0011] In another particular embodiment of the method, steps are further performed in which:

- determine the location of the person with the assigned access parameter, indicating that the person does not have permission to be in the area of space;

- determine at least one augmented reality device located closest to the mentioned area of space;

- form a route from the region of space in which this device of augmented reality is located, to the region of space in which the said person is located;

- display the route on the augmented reality device in augmented reality mode.

[0012] In another particular embodiment of the method, steps are further performed in which:

- track the movement of the augmented reality device in real time;

- adjust the route in case of deviation of the trajectory of the augmented reality device from the previously formed route.

[0013] In another particular embodiment of the method, steps are further performed in which:

- upon reaching the augmented reality device, the area of the space in which the said person is located, analyzes in real time the incoming video stream from the camera of the augmented reality device to determine the presence of at least one face in the video stream frames;

- display in the specified area of the display of the augmented reality device in augmented reality mode information that the person in the field of vision of the employee with AR glasses does not have permission to be in the space in accordance with the assigned access parameter.

[0014] In another particular embodiment of the method, steps are further performed in which:

- determine the location of the person in the field of space and the location of at least one augmented reality device;

- convert data on the locations of the face and augmented reality device into coordinates in a 3D model of space;

- determine the route in the 3D model of space from the location of the augmented reality device to the mentioned person;

- form a pointer to the trajectory of movement along the route in the 3D model of space and display it on the display of the augmented reality device in augmented reality mode to navigate the employee with the augmented reality device to the area, space in which the mentioned person is located, along the route formed in the 3D space model.

[0015] In another particular embodiment of the method, steps are further performed in which a face indicator is formed in the 3D space model in accordance with its location in the 3D space model and the aforementioned pointer is displayed on the augmented reality device display in the augmented reality mode when the face is in the region employee visibility with augmented reality device, to indicate a person in the field of space.

[0016] In another preferred embodiment of the claimed solution, there is provided a facial recognition system in a video stream comprising at least one computing device and at least one memory containing machine-readable instructions that, when executed by at least one computing device, perform the above method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

[0018] In FIG. 1 shows an example implementation of a face recognition device.

[0019] FIG. Figure 2 shows an example implementation of a system for recognizing faces using augmented reality.

[0020] In FIG. 3 shows an example of a region of interest in an image.

[0021] In FIG. 4 is an example of a general view of a face recognition computing device.

DETAILED DESCRIPTION OF THE INVENTION

[0022] The concepts and terms necessary for understanding this technical solution will be described below.

[0023] In this technical solution, a system is meant, including a computer system, a computer (electronic computer), CNC (numerical program control), PLC (programmable logic controller), computerized control systems, and any other devices that can perform a given , a clearly defined sequence of operations (actions, instructions).

[0024] An instruction processing device is understood to mean an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs).

[0025] The command processing device reads and executes machine instructions (programs) from one or more data storage devices. Hard drives (HDD), flash memory, ROM (read-only memory), solid state drives (SSD), optical drives can act as storage devices.

[0026] A program is a sequence of instructions for execution by a computer control device or an instruction processing device.

[0027] In accordance with the circuit of FIG. 1, a face recognition system using augmented reality means comprises a video stream source 1, a face recognition device 10 and an augmented reality device 20.

[0028] The source of the video stream 1 may be an external surveillance camera (USB or IP) or an integrated camera in an augmented reality device. Various types of cameras can be used as surveillance cameras (sometimes video cameras), in particular PTZ cameras, IP cameras, stationary anti-vandal cameras, all-round cameras, etc., without limitation. The camera must be configured to transmit the video stream to the face recognition device 10, made on the basis of at least one computing device, for example, a server, which can be in the data cloud with a biometric identification platform 11. The source of the video stream 1, in particular a surveillance camera or AR the points and the face recognition device 10 can interact to transmit data over wired and / or wireless communications using Socket and / or WebSocket, and / or WCF, and / or REST, and / or whether HTTP and WebSocket, etc.

[0029] In a general view (see FIG. 4), the face recognition device (300) comprises one or more processors (301) combined by a common data bus, memory means such as RAM (302) and ROM (303), input interfaces I / O (304), input / output devices (305), and a device for network communication (306).

[0030] A processor (301) (or multiple processors, a multi-core processor, and the like) can be selected from a variety of devices that are currently widely used, for example, manufacturers such as Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, etc. Under a processor or one of the processors used in the system (300), it is also necessary to take into account a graphic processor, for example, an NVIDIA or Graphcore GPU, the type of which is also suitable for the full or partial execution of method (100), and can also be used for training and application of machine models training in various information systems.

[0031] RAM (302) is a random access memory and is intended to store machine-readable instructions executed by the processor (301) to perform the necessary operations for logical data processing. RAM (302), as a rule, contains executable instructions of the operating system and corresponding software components (applications, program modules, etc.). In this case, the available memory capacity of the graphics card or graphics processor may act as RAM (302).

[0032] The ROM (303) is one or more permanent storage devices, for example, a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0033] To organize the operation of system components (300) and organize the operation of external connected devices, various types of input / output interfaces (304) are used. The choice of appropriate interfaces depends on the particular computing device, which can be, but not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0034] To ensure user interaction with the computing system (300), various means (305) for inputting / outputting information are used, for example, a keyboard, a display (monitor), a touch screen, a touch pad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

[0035] The network interaction tool (306) provides data transmission via an internal or external computer network, for example, an Intranet, the Internet, a LAN, etc. As one or more means (306), but not limited to: an Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communications module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, etc.

[0036] Additionally, satellite navigation aids as part of system (300), for example, GPS, GLONASS, BeiDou, Galileo, can also be used.

[0037] The specific selection of device elements (300) for implementing various hardware and software solutions may vary while maintaining the required functionality.

[0038] The face recognition device 10, in a particular embodiment, also includes a biometric identification platform 11, a database (DB) of identifiable persons descriptors 12, a route building module 13, and a notification generation module 14. The biometric identification platform 11 determines the position and size of all persons present on the input video stream. Each fixed face is assigned an indicator of image quality, based on which the best images are automatically selected for further processing. The correspondence index for any pair of face images is determined by the Euclidean distance of the corresponding face descriptors. Low values of the Euclidean distance indicate a high similarity of two images of the same person. In this case, the application installed on the AR glasses is sent a response in JSON format with information about the recognized person in the following form:

{"name": "Ivanov OV," color ":" red "," article ": 191.5," info ":" Particularly dangerous "}

[0039] A compliance indicator refers to a one-to-many comparison, i.e. the image from the camera is compared with many images in the database 12. Each aligned face image is further processed to obtain a face descriptor using deep neural networks (Deep Neural Networks or DNN). Face descriptors are numerical vectors that summarize the characteristic features of a face. A key property of descriptors is their close similarity for images of the same person, and a strong difference for images of different people. Thus, face descriptors depend only on the individual characteristics of a particular person, changes in images due to changing shooting points, lighting, hairstyles and age are ignored. Each descriptor has many points, which can be a cloud of points. The distance between the corresponding points in different descriptors is measured, then they are summed to obtain a numerical face descriptor vector.

[0040] The route building module 13 and the notification generating module 14 may be implemented on the basis of at least one command processing device, such as a processor or microcontroller, executed in the firmware in such a way as to perform the functions assigned to these modules below. Also, the module 13 constructing the route can be implemented on the basis of software and hardware computing means of the augmented reality device 20 and can be placed directly in the architecture of the augmented reality device 20.

[0041] The web interface of the face recognition device 10 allows you to edit the description and add new faces to the database 12, as well as blacklist (stop list) marks indicating that the person is in the blacklist. The interaction of the client with the server side of the face recognition device 10 is carried out using the HTTP or HTTPS protocol.

[0042] The augmented reality device 20 is an information output means, which can be any known device configured to display information about physical and digital objects in real time, including: augmented reality glasses (or AR glasses) (for example , Hololens or Magic Leap), a mixed reality headset (MR) or a computing device (smartphone / tablet / phablet / laptop, etc.) with appropriate software that allows you to overlay images of digital objects on real-world images or virtual Second model of the real world. The window of the launched application displays the original face photo in the database, a face photo from the captured frame, and information about the identified person. The route to the zone in which the recognized face was recorded can also be displayed.

[0043] Based on the results of the study, potential vectors for the development of the prototype were formulated to eliminate the identified shortcomings and limitations, to improve the prototype, making it more functional and convenient to use.

[0044] The first vector involves overcoming the main limitation of the distance to the recognition object and the low speed of computing devices without revising the architecture of the solution. To increase the distance to the recognition object, you can use long-focus cameras with high resolution. In the case of using cameras with telephoto lenses, the system architecture is as follows. A camera with a telephoto lens transmits the image to a computing device (computer, server, etc.).

[0045] An application is installed on the computing device to locate faces in captured frames and redirect real-time data to a device with a biometric face recognition platform. AR glasses of augmented reality with the running application display information about recognized faces and plot a route to the recognized person.

[0046] To solve the performance problem, device OS optimization is required. It is also possible to increase productivity by adding / replacing the hardware components of the glasses.

[0047] In this technical solution, it is possible to implement several options for increasing productivity due to hardware:

- the use of a special purpose integrated circuit (for example, an ASIC special purpose integrated circuit), which is designed to solve a specific problem.

- Increasing the computing power of the augmented reality device by increasing the number of RAM (it is possible to use faster), using more productive CPUs or GPUs. The second vector allows, in addition to the main problem, also to optimize the UX and simplify the tasks of law enforcement officers when working in crowded places. The identification of intruders in mass crowds is very difficult, even if the intruder was noticed, tracking its movement among the mass of people is a difficult task.

[0048] This problem can be solved by building an integrated AR + CCTV system, including:

идентификацию лиц в местах скопления людей длиннофокусными камерами CCTV и системой видеоаналитики;

face identification in crowded places with long-focus CCTV cameras and video analytics system;

оповещение сотрудников в очках AR о выявлении нарушителя и указание направления к нему (навигация по GPS и/или маякам);

notification of employees in AR glasses about the identification of the intruder and an indication of the direction to him (navigation on GPS and / or beacons);

быструю идентификацию личности с использованием устройства дополненной реальности.

quick identification using an augmented reality device.

[0049] Closed Circuit Television (CCTV) stands for “Closed Circuit Television System,” and this term refers to a set of devices that provide video surveillance. A CCTV system can consist of many cameras directed to different areas of the controlled area space, while transmitting images to a server with the face stream application installed, where people are searched for in captured frames and transferred to a recognition server.

[0050] Each area covered by CCTV cameras is uniquely associated with the geographical coordinates of this area for the subsequent construction of the route.

[0051] In open areas, GPS navigation may be used. To create a navigation system on augmented reality glasses running on the Android OS, GooglePlayserviceslocationAPIs can be used, which allows you to access the GPS system built into the computing module of the glasses. Then lay a route from the user's location to the area in which the identified object is located.

[0052] Indoor navigation systems, for example, a method for navigating Bluetooth beacons, can be used to construct the route indoors. This method consists in placing special sensors on the perimeter of the room, which provide data on the user's location of the glasses relative to the zone in which the object is fixed. The location is calculated by comparing the measured in real time signal powers from the surrounding Bluetooth points with pre-measured values tied to a room map. More dense placement of sensors allows to increase the accuracy of the user's localization of glasses and the location of the object. An example of implementing navigation based on this technology is Apple's iBeacon system.

[0053] Also, indoor navigation by QR codes can be used to construct the indoor route. With this method of navigation, a 3D model of the room is built, in which a QR code is fixed in a certain place of this model and in the room. In order for the glasses user to be localized in the 3D model of the room, he needs to scan the QR code with the glasses camera in the room.

[0054] Next, the operation of the face recognition device 10 will be described with reference to FIG. 2 using AR glasses as an augmented reality device.

[0055] In accordance with the embedded software and hardware algorithm, at least one camera 100, which is the source of the video stream 1 and is part of the CCTV, captures a given area of space 110. The camera 100 can be located in any convenient for viewing area of space 110 for example, mounted on a ceiling, wall, etc.

[0056] The image of the space region from the camera 100 in the form of a video stream is supplied to the face recognition device 10, which analyzes the incoming video stream in real time to determine if at least one face 101 is present in the frames of the video stream. Together with the image from the camera 100, the device 10 also the identifier of the camera 100 can be received. Additionally, a video stream from the camera of AR glasses 20 and their identifier, for example, the MAC address, can also be received in the face recognition device 10.

[0057] When at least one face 101 is detected on the incoming video stream frames, the face recognition device 10 captures these frames of the video stream and sends them as a sequence of n frames to the biometric identification platform 11 to identify the face 101.

[0058] In the face recognition device 10, an area of interest in the incoming frames of the video stream can be set. The region of interest is specified in pixels as an array of the form [x, y, width, height], where (x, y) are the coordinates of the upper left point of the region of interest. The coordinate system in the image is set similarly to the figure in FIG. 3. If the width and height are equal to '0', the entire frame is considered the area of interest. Accordingly, in such an implementation of the claimed solution, the face recognition device 10 will capture only those frames of the video stream in which the image of the face 101 falls into a given region of interest. For example, the area of the turnstile placement can be selected as the area of interest. Thus, capture will be carried out only of those images of persons who are in the area of the turnstile.

[0059] Also, in addition, the face recognition device 10 may be configured to determine the presence of an image of a person in the frame, the image of the face 101 of which has been detected. To determine the presence of a person’s image in the frame, the device 10 after detecting the face image determines the presence in the frame of the image of the shoulders in the corresponding face image of the area or image of the human body. After the device 10 has determined the presence of a person in the frame whose face image 101 has been determined, the frames of the video stream are captured for transmission in the form of a sequence of n frames to the biometric identification platform 11 for identifying the person in the frame.

[0060] The biometric identification platform 11 determines the position and size of the face image 101 on each input captured frame in a sequence of n frames in which this face image 101 is recognized. If additional images of faces are defined on captured n frames, then the platform 11 can also determine the sizes and position of all face images in the manner described below. Based on the face position data, the platform 11 tracks the face image 101 on captured n frames.

[0061] The sizes of the face images 101 on the captured n frames, the biometric identification platform 11 compares with the threshold values for the size of the face images, and if the sizes of the face image 101 on the captured frame are larger than the predetermined threshold value, the said platform 11 proceeds to the step of determining face parameters based on the image of the face on the captured frame, in particular, the parameters: head rotation (in degrees) along each of the three rotation axes, tilt up / down relatively horizontally oh axis; tilt left / right relative to the vertical axis; rotation about a vertical axis. For images of faces whose size is less than a predetermined threshold value, the mentioned parameters are not determined.

[0062] The face image parameters determined by platform 11 of the face 101 on the captured frame are compared with predetermined threshold parameters of the face parameters and based on the deviation of the face image parameters determined in the previous step from the threshold settings of the parameters, the platform 11 assigns a face image quality indicator to the face image 101, wherein the lower the deviation, the higher the face image quality indicator. To increase the accuracy of identification of the face 101 for further analysis, the platform 11 selects the image of the face 101, in which the quality indicator is the maximum.

[0063] Accordingly, if additional face images are found in the video stream, besides the face image 101 that fall into the area of interest, they can also be processed in the manner described above to determine additional face images for which the quality indicator will be the maximum.

[0064] Further, based on the image of the face 101, in which the quality indicator is maximum, the biometric identification platform 11, using methods known in the art, constructs a point cloud of faces and calculates a vector characterizing the features of the face 101 image (hereinafter, the face 101 vector). The platform 11 compares the obtained vector with the pre-stored image vectors in the database 12. Each pre-stored image vector in the database 12 is assigned the corresponding identification information of the person based on the image of the person whose given image vector was calculated. Identification information may be, for example, passport data, photograph of a person, etc. Also, DB 12 may contain information, for example, a black list (stop list) label indicating that the person is on the black list (stop list), for example, due to the fact that he is an attacker or an intruder .

[0065] If the vector characterizing the features of the image of the face 101 coincides with the image vector stored in the database 12, then the biometric identification platform 11 determines the image of the face 101 as recognized / identified. Next, the platform 11 refers to the database 12 to retrieve the identity information of the person 101 in accordance with the image vector from the database 12 with which the face vector 101 matches, and check the presence of the blacklist mark to determine the access parameters of the person 101 in the area of space 110 captured by the camera, in accordance with a predefined classifier.

[0066] If the blacklist label is contained in the database 12, then the platform 11 assigns the access parameter to the face 101 indicating that the face 101 does not have permission to be in the area of space 110, in particular the blacklist classifier. If the blacklist label in the database 12 is absent, then the platform 11 assigns an access parameter to the face 101, indicating that the face 101 has permission to be in the area of space 110, in particular, the white list classifier. Thus, the face recognition device 10 assigns access parameters to the face 101, which determine the permission to be in the area of space 110, based on the image of the face 101 received from the camera 100.

[0067] The results of the comparison of vectors, in particular identification information and access parameters, can be output by the device 10 to image output means, for example, a face recognition device 10 together with a pointer to a face image 101 on real-time output frames of a video stream, for example, in JSON format with information about the recognized person.

[0068] If the vector characterizing the features of the image of the face 101 did not match any image vector stored in the database 12, then the biometric identification platform 11 determines the image of the face 101 as unrecognized / unidentified and assigns the access parameter 101 to the face indicating that the person 101 does not have permission to be in the area of space 110. The results of the comparison of vectors, in particular the information that the image of the face 101 is not recognized, and the access parameter can also be output to the image display device, for example, a user display of a face recognition device 10 or augmented reality device, in particular, on glasses of AR glasses 20, in JSON format with information about a recognized person.

[0069] Thus, since for the formation of a vector characterizing the features of the image of the face 101, only the image of the face 101 is used, in which the quality indicator is maximum, and not the entire frame of the video stream, the processing speed of the image of the face 101 increases, the time decreases and the identification accuracy increases persons and their classification for determining access parameters.

[0070] Next, the face recognition device 10 proceeds to the steps of determining the location of the face 101 with the assigned access parameter indicating that the face 101 does not have permission to be in the area of space 110 (i.e., the face image of which was not recognized or was determined, that the face in the recognized image is in the "black list"), and the construction of 101 routes to this person. To build a route, the face recognition device 10 can be additionally equipped with a module 13 for building a route to the face 101, which identifies the camera 100, by which the image of the face 101 was obtained, and retrieves the geographical coordinates of the area of space 110 located in the shooting area of the camera 100. The geographical coordinates of the area spaces 110 can be extracted by the module 13 from the database 12 based on the camera identifier, or requested by methods known from the prior art from the corresponding camera 100, from which frame with the face 101.

[0071] Also for constructing a route, the face recognition device 10 may be configured to determine the location of the face 101 whose face image has been recognized. For example, a command to build such a route can be sent from the user to the device 10, which, using the information input / output device, selects a person who was recognized in the image to build a route to it.

[0072] Further, the route building module 13 determines whether at least one employee in the AR glasses 20 is located in the area of the space 110 based on the location information of the AR glasses 20 obtained using a navigation system sensor, for example, a GPS transceiver, or a Bluetooth beacon integrated in the AR glasses 20. Reading the Bluetooth beacon can be carried out using a Bluetooth beacon reader 111 installed in the area of space 110 or in the space in which the employee in the AR glasses 20 is located.

[0073] To determine the location of the AR glasses 20, the route building module 13 can also, by means of reading the camera of AR glasses 20 located in different areas of the space, QR codes. The information of the QR code enters the route building module 13, which, based on the received information, determines the location of the AR glasses 20.

[0074] If at least one AR glasses 20 are in the region of space 110, then the route building module 13 sends a command to the notification generation module 14, according to which the module 14 generates a notification for the AR glasses 20 located in the region of the space 110 containing information that in this location area of the AR-glasses 20 there is a person with an assigned access parameter, indicating that the person does not have permission to be in the area of space 110 due to the fact that he is unrecognized or is in the “black list” ". Additionally, the notification may include an indication that the AR-glasses 20 will scan the area of space 110 to capture the image of the mentioned person 101. For example, the generated notification, in JSON format with information about the recognized person, is sent by the module 14 to the AR-glasses 20 to display a notification on glasses of AR glasses in augmented reality mode using methods known from the prior art.

[0075] If the route building module 13 determines that there is no employee in the AR glasses 20 in the area of space 110, then the module 13 determines at least one AR glasses located closest to the location data from all AR glasses to the area of space 110. Based on the location data of the AR glasses, module 13 generates a route from the area of space in which the AR glasses are located to the area of space 110 where the face 101 is unrecognized or blacklisted, and transfers it to forming module 14 notification, which generates a notification that in the space 110 is face 101. Formed notice and route are also sent to the module 14 on the AR-glasses to display them on the glass AR

- glasses (i.e. augmented reality device display) in augmented reality mode using methods known from the prior art.

[0076] Placing the route generating module 13 and the notification generating module 14 in the face recognition device 10 allows to reduce the computing load on the computing means of the AR glasses 20. However, in an alternative embodiment of the claimed solution to reduce the computing load on the face recognition device 10, the route forming module 13 and the notification generating unit 14 can be placed in the architecture of the AR glasses 20. Accordingly, the face recognition device 10: identifies a camera 100 by which ylo obtained face image 101; retrieves the geographical coordinates of the area of space 110 based on the identifier of the camera 100; determines at least one AR glasses located closest to the area of space 110; and sends the mentioned coordinates to the route generation module 13 located in the architecture of the AR glasses 20. Also, since only the coordinates and not the entire route are transmitted through the communication channel between the module 13 and the device 10, the transmission capacity of the communication channel between the AR glasses 20 increases and face recognition device 10.

[0077] Upon receiving the geographical coordinates of the area of space 110, the route generating unit 13 determines the location of the AR glasses 20 using a navigation system sensor, a Bluetooth beacon, or using a QR code read through the AR glasses and sends commands to the notification generating module 14 for output on glasses of AR-glasses located in the area of space 110, a notification containing information that in this area of location of AR-glasses 20 there is an unrecognized or “blacklisted” person 101, or output to glasses of AR-glasses in (i.e., the display of the augmented reality device) notifications that a face 101 is in the area of space 110 if the AR glasses 20 are not located in the area of space 110, and route information from the area of space in which the AR- data is located glasses, to the area of space 110, in which an unrecognized or blacklisted person 101 is located.

[0078] During the movement of the employee in the AR glasses along the route to the area of space 110, the route building module 13 can be configured to track the trajectory of the augmented reality device, in particular the movement of AR glasses in real time from the information received from them about the location and adjust the route in case of deviation of the trajectory of the augmented reality device from the previously formed route.

[0079] When this employee reaches the end point of the route — space area 110, module 14 sends a notification to AR glasses containing information that an unrecognized or blacklisted person 101 is in this AR glasses location, as well as if necessary, an indication of scanning using the AR-glasses camera in the area of space 110.

[0080] After the route building module 13 sent a notification to the AR glasses 20, indicating that the face 101 is in the location area of the AR glasses 20, the face recognition device 10 analyzes the incoming video stream from the AR- camera in real time points 20 for determining the presence in the frames of the video stream of at least one face 101 as previously described.

[0081] When at least one face is detected on the frames of the video stream from the camera of the AR glasses 20, the face recognition system 10 captures these frames of the video stream and sends them as a sequence of n frames to the biometric identification platform 11. The biometric identification platform 11 processes the received frames as described previously a method for calculating a vector characterizing the features of the face image 101.

[0082] When at least one face is detected on the frames of the video stream from the camera of the AR glasses 20, the face recognition system 10 captures these frames of the video stream and sends them as a sequence of n frames to the biometric identification platform 11. The biometric identification platform 11 processes the received frames as described previously, a method for calculating a vector characterizing the features of the face image 101. The platform 11 obtained at the previous stage compares with the previously obtained vector characterizing the prize Aki face image 101. When the coincidence of vectors of biometric identification platform 11 sends a corresponding command to the module 14 generating notifications to generate and output on the AR-20 points notification windows that face 101 is unrecognized or who are blacklisted entity. The notification is output by methods known from the prior art to a predetermined region of the glasses of AR glasses 20 (i.e., the display of the augmented reality device) in augmented reality mode along with information that the person in the field of view of the employee in AR glasses 20 is unrecognized person 101 or a person on the “black list”.

[0083] Further, the face recognition device 10 may be configured to determine the location of the face 101 in the area of the space 110. Accordingly, after the face recognition 10 determines the presence in the area of the space 110 of the face 101 that does not have permission to be in the area of the space 110 , the device 10, by means of the biometric identification platform 11, determines, by methods known from the prior art, the location of the face 101 in the region of space 110, for example, by determining the location images of the face 101 with respect to at least one object in the image whose location in the area of space 110 is previously known and set in the face recognition device 10. The obtained location data of the face 101 in the area of space 110 are converted to the coordinates of the face 101 in a 3D model of the space, the model of which is previously created and saved in database 12.

[0084] Also, the location of the face 101 in the area of space 110 can be carried out as follows. After the face recognition device 10 has determined the presence in the region of the space 110 of the face 101, which does not have permission to be in the region of the space 110, the route building module 13, based on the camera identifier, extracts from the database 12 information about the location of the camera and the area of the space 110 covered by it. Based on the information about the location of the camera, module 13 determines the coordinates of the camera in the 3D model of space and projects the area of space to be covered by the camera to a point on the 3D model. The coordinates of the obtained point on the 3D model are determined by module 13 as the coordinates of the face 101 in the 3D space model.

[0085] The obtained coordinates of the face 101 in the 3D space model are transmitted to the route building module 13, which also converts the location data of the AR glasses 20 into the coordinates of the glasses in the 3D space model to localize the AR glasses 20 in the 3D space model. Next, the route building module 13, based on the obtained coordinates of the face 101 and the coordinates of the glasses in the 3D space model, determines the route in the 3D space model from the location of the AR glasses 20 to the face 101 and generates a route path indicator in the 3D space model. The generated path indicator in the 3D space model is sent to the notification generation module 14 and displayed on the glasses of AR glasses 20 (i.e., the display of the augmented reality device) in augmented reality mode in accordance with the location of the AR glasses 20 for the employee’s navigation in AR-glasses 20 to the area of space 110 along the route formed in the 3D model of space.

[0086] Also, the notification generation module 14 may be configured to generate a 2D or 3D face indicator 101 in the 3D model of the space in accordance with its location in the 3D space model. The face indicator 101 is placed according to the previously determined coordinates of the face 101 in the 3D space model and is displayed by the module 14 on the glasses of AR glasses 20 in augmented reality mode when face 101 is in the employee’s field of view in AR glasses 20 to indicate face 101 in the area space 110.

[0087] Thus, due to the fact that the claimed system is made with the possibility of forming a route to the place of fixing by the cameras of the person 101, who does not have permission to be in the area of space 110, operational detection and identification of a person is ensured, and the speed of detention of violators is increased. Also, the presented system has a simple design, due to which the reliability of the system increases.

[0088] Modifications and improvements to the above described embodiments of the present technical solution will be apparent to those skilled in the art. The preceding description is provided as an example only and is not subject to any restrictions. Thus, the scope of the present technical solution is limited only by the scope of the attached claims.

Claims (37)

1. A method for recognizing faces in a video stream, performed by at least one computing device, comprising the steps of: - receive a video stream from at least one source of a video stream with an image of a region of space; - capture frames of the video stream upon detection of images of at least one face on them; - determine the position and size of the face image on each captured frame; - determine the quality indicators of the image of the face on the captured frames, on which the size of the face image is higher than the predetermined threshold values; - choose a face image with a maximum face image quality indicator; - calculate a vector characterizing the features of the image of the face, based on the selected image of the face; - comparing the calculated vector with the stored image vectors in the database (DB); - based on the results of comparing the calculated vector with the stored vectors, an access parameter is defined for the person that determines the permission to be in the space region; - outputting the access parameter to the image display device. 2. The method according to p. 1, characterized in that the image display device is an augmented reality device, wherein the access parameter is output in augmented reality mode in JSON format with information about a recognized or unrecognized person. 3. The method according to p. 1, characterized in that it further comprises the step of setting the area of interest for the incoming frames of the video stream, while capturing only those frames of the video stream in which the face image falls into the specified area of interest. 4. The method according to p. 1, characterized in that after capturing frames of the video stream upon detection of images of at least one face, they additionally perform the step of: - determine the presence of the image of a person in the frame, the image of the face of which was detected, and to determine the presence of the image of a person in the frame, determine the presence in the frame: image of the shoulders in the corresponding face image of the area or image of the human body. 5. The method according to claim 1, characterized in that the facial image quality indicators are determined based on at least one face parameter: - rotation of the head (in degrees) along each of the three axes of rotation, tilt up / down relative to the horizontal axis; - tilt left / right relative to the vertical axis; - rotation about a vertical axis. 6. The method according to p. 1, characterized in that it further comprises stages in which: - determine the location of the person with the assigned access parameter, indicating that the person does not have permission to be in the area of space; - determine at least one augmented reality device located closest to the mentioned area of space; - form a route from the region of space in which this device of augmented reality is located, to the region of space in which the said person is located; - display the route on the augmented reality device in augmented reality mode. 7. The method according to p. 6, characterized in that it further comprises stages in which: - track the movement of the augmented reality device in real time; - adjust the route in case of deviation of the trajectory of the augmented reality device from the previously formed route. 8. The method according to p. 6, characterized in that it further comprises stages in which: - when the augmented reality device reaches the area of space in which the said person is located, the incoming video stream from the camera of the augmented reality device is analyzed in real time to determine if there is at least one face in the video stream frames; - display in the specified area of the display of the augmented reality device in augmented reality mode information that the person in the field of vision of the employee with AR glasses does not have permission to be in the space in accordance with the assigned access parameter. 9. The method according to p. 1, characterized in that it further comprises stages in which: - determine the location of the person in the field of space and the location of at least one augmented reality device; - convert data on the locations of the face and augmented reality device into coordinates in a 3D model of space; - determine the route in the 3D model of space from the location of the augmented reality device to the mentioned person; - they form a pointer to the trajectory of movement along the route in the 3D model of space and display it on the display of the augmented reality device in augmented reality mode to navigate the employee with the augmented reality device to the area of space in which the mentioned person is located along the route formed in the 3D space model. 10. The method according to p. 9, characterized in that it further comprises stages in which: - form in the 3D model of space a face indicator in accordance with its location in the 3D model of space and display the aforementioned pointer on the augmented reality device display in augmented reality mode when the person is in the area of visibility of an employee with the augmented reality device to indicate a person in the space region . 11. A system for recognizing faces in a video stream comprising at least one computing device and at least one memory containing machine-readable instructions that, when executed by at least one computing device, perform the method according to any one of claims. 1-10.

Images