RU2595620C1 - Method for auto calibration of stereo cameras employed for automatic recognition of human faces - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: group of inventions relates to techniques for identification of images, and specifically to methods for personal identification employing video cameras. Disclosed is method for auto calibration of stereo cameras employed for automatic recognition of human faces. Method includes step of obtaining projections from stereo cameras three-dimensional image with face texture onto two-dimensional surface, then set of adjacent points is determined on obtained projections of facial images, corresponding to same areas of image from two or more stereo cameras. Some of found adjacent points therein are filtered to select most reliable points, that is followed by three-dimensional reconstruction of adjacent points, further, angles for turning and three-dimensional transfer for adjacent points in coordinate system, common for all stereo cameras. Followed by calculating mutual position of said stereo cameras relative to each other, and finally, calibrating stereo cameras is performed between each other based on mutual position of stereo cameras relative to each other.

EFFECT: technical result is higher accuracy of auto calibration of stereo cameras to each other.

5 cl, 8 dwg

Description

FIELD OF TECHNOLOGY

This invention relates to the field of pattern recognition, and in particular to methods for identifying a person’s identity using video cameras.

The invention also relates to the field of machine vision, and can be used in robotics.

BACKGROUND

Currently, to recognize a person’s face using two stereo cameras located at a certain distance from each other, it is important to calibrate the stereo cameras between themselves. Due to the large distance between the stereo cameras, the structural elements of the camera mount may be subject to vibration, as a result of which random changes are made to the calibration of the stereo cameras. Therefore, it is important to be able to determine the calibration between stereo cameras automatically, changing the current calibration value in real time, and not stopping the operation of the face recognition system.

There are many methods for calibrating cameras in a stereoscopic system, including using vanishing points, neural networks, perpendicular planes (Calibration of stereo cameras from two perpendicular planes), the Faugeras and Toscani method, a checkerboard calibration method, and a calibration method cameras using a zoom lens of Marina Kolesnik. All these methods have a common drawback - during the calibration process, the user is required to manually select the control points, lines or planes available on the images that are obtained using stereo pair cameras.

A solution is known (see non-patent document 1), in which registration of a set of images of a test object is performed, after which threshold filtering of the image is performed, the test object is recognized by size and shape, then the background is removed, the position of the four extreme points is determined, the first and second are calibrated cameras, and eventually calibrate the system as a whole.

The disadvantage of this solution is that to increase the accuracy of calibration, it is necessary to increase the number of images of test objects involved in the calibration of the system (see non-patent document 2). We can conclude that the calibration result is stabilized if there are more than twenty images, which increases the computational load and reduces the speed of the method.

The prior art method (see patent document 1), in which two or more stereo cameras are located at some distance between each other, each of the stereo cameras reconstructs its three-dimensional image of a part of the face, then the individual parts are combined into one common three-dimensional image, according to which recognition. To accurately combine parts of the image from different cameras, it is proposed to calibrate stereo cameras simultaneously by visible common image elements.

This patent does not specify a method for automatically calibrating stereo cameras, but merely states that calibration is performed on visible common areas of the image. In one embodiment, it is proposed to combine parts of a person’s face reconstructed with stereo cameras into a general three-dimensional reconstruction of a person’s face in an absolute distance scale, for which a set of two or more control points of the observation zone is used, the absolute distances between which are known in advance.

The disadvantages of this method of autocalibration are the absence on the scene of a set of two or more control points of the observation zone, the absolute distances between which are known in advance. It is also obvious that using all the visible image points is computationally expensive. The combination of all visible points in three-dimensional space relies only on geometric information about the object, and alignment errors are possible. There may be cases when one part of the three-dimensional points from one stereo camera is aligned to another part of the points visible in the second stereo camera in such a way that the combined three-dimensional points correspond to different parts of the person’s face, such a combination leads to distortion of the face shape and deterioration of recognition.

[Non-patent document 1] Authors: V.V. Korotaeva, A.V. Krasnyashchina, S.N. Yarysheva, H.V. Nguyen, name: METHOD OF AUTOMATIC CALIBRATION OF STEREOSCOPIC SYSTEM, Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2014, No. 4 (92). [Non-patent document 2] Krylov D.V. Investigation of the method of photogrammetric calibration of digital cameras using a spatial test object // News of universities. Geodesy and aerial photography. 2011. No1. S. 57-62.

[Patent document 1] RU 2488882 “Method for the identification of a person’s personality”, published July 27, 2013, patent holder of Vokord SoftLab LLC (Yai).

SUMMARY OF THE INVENTION

The present invention addresses the disadvantages inherent in existing solutions.

The technical result of this invention is to increase the accuracy of auto-calibration of stereo cameras with each other.

This technical result is achieved due to the method of auto-calibration of stereo cameras used for automatic recognition of a person’s face, in which projections of a three-dimensional image with a texture of a face onto a two-dimensional surface are obtained from stereo cameras, after which a set of conjugate points on the obtained projections of the face image corresponding to the same image parts on two or more stereo cameras, some of the found paired points being filtered to select the most reliable points, then perform three-dimensional reconstruction of the conjugate points, then determine the rotation angles and three-dimensional transfer for the conjugate points in the coordinate system common to all stereo cameras, then calculate the relative position of these stereo cameras relative to each other, and ultimately calibrate the stereo cameras between themselves according to the relative position of the stereo cameras relative to each other friend.

The projection surface on which the conjugate points are calculated is common to two or more stereo cameras.

The search for conjugate points can be carried out using the correlation function in a small specified window, by the optical flow method, or using special descriptors of points of a two-dimensional image.

Three-dimensional reconstruction of conjugate points is performed in the coordinate system of each of the stereo cameras.

The projection surface is a plane.

BRIEF DESCRIPTION OF THE DRAWINGS

Signs and advantages of the present invention will become apparent from the following detailed description of the invention, an example implementation and the accompanying drawings, in which:

FIG. 1 - Arrangement of two stereo cameras (1, 2 - stereo cameras, 3, 4, 5, 6 - lenses of the cameras included in stereo cameras, 7 - load-bearing columns on which stereo cameras are installed, 8 - bed for attaching load-bearing columns).

FIG. 2 - An example of 2D face images obtained by cameras 3, 4, 5, 6 in FIG. one.

FIG. 3 - The reconstructed field of 3D points of the face from two stereo cameras (a) for the left stereo camera, (b) for the right stereo camera.

FIG. 4 - Projections of a 3D image of a face onto a plane.

(a) for the left stereo camera, (b) for the right stereo camera.

FIG. 5 - The projection of all 3D sections of the face onto a plane corresponding to the initial calibration of the cameras, with the existing vibration of the supporting columns, with distortion of the proportions on the face (the nose has become narrower).

FIG. 6 - A reference image of the same person from the database with which the image is compared in FIG. 6.

FIG. 7 - Arrangement of conjugate points on the projections of 3D images of the face on a given plane, (a) for the left stereo camera, (b) for the right stereo camera.

FIG. 8 - The undistorted projection of all three-dimensional parts of the face on a given plane, after the auto-calibration procedure according to the proposed method.

DETAILED DESCRIPTION OF THE INVENTION

Below will be described the concepts and definitions necessary for the detailed disclosure of the invention.

A stereo camera is a type of video camera with two or more shooting lenses that create separate frames on film or matrix.

Camera calibration is the task of obtaining the internal and external parameters of the camera from the available photos or videos captured by it.

Autocalibration - obtaining calibration data directly from images, and the presence of special calibration objects in the scene is not required.

Conjugated points are points that correspond to the same parts of the face image on different stereo pairs, which are automatically determined after the projections are built.

This invention in its various embodiments can be implemented in the form of a method, including that implemented on a computer, in the form of a system or computer-readable medium containing instructions for performing the aforementioned method.

In this invention, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices that can perform a given, well-defined sequence of operations (actions, instructions).

By a command processing device is meant an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

The command processing device reads and executes machine instructions (programs) from one or more data storage devices. Data storage devices may include, but are not limited to, hard disks (HDDs), flash memory, ROM (read only memory), solid state drives (SSDs), and optical drives.

Program - a sequence of instructions intended for execution by a control device of a computer or a device for processing commands

According to the invention, a method for automatically calibrating stereo cameras used for automatic recognition of a person's face includes the following steps:

receive from stereo cameras projections of a three-dimensional image with a texture of the face on a two-dimensional surface;

Video cameras are arranged in pairs, while synchronizing with each other and calibrating among themselves, forming a stereo camera from two video cameras (Fig. 1). The stereo cameras are positioned at a predetermined distance from each other in an amount of two or more, with each stereo camera performing an independent three-dimensional reconstruction of the visible part of a person’s face, combining the reconstructed parts of a person’s face into a general three-dimensional reconstruction of a person’s face, while performing continuous or periodic calibration of stereo cameras between by myself. At the same time, it becomes possible to build at least two independent three-dimensional reconstructions of a person’s face.

Images of a person’s face are used explicitly with texture, formed on projections of three-dimensional images onto a certain surface. The specified surface can be selected independently for each stereo camera, or be common to two or more stereo cameras. The obtained images of the projection of the face onto the selected surfaces together with the texture are depicted in FIG. four.

determine a set of conjugate points on the obtained projections of the face image corresponding to the same image areas on two or more stereo cameras, some of the found conjugate points are filtered to select the most reliable points;

On the projection images, points are automatically determined that correspond to the same parts of the face image on different stereo pairs. The number of such points can be significantly less than the total number of all projection points in order to increase computational performance.

The search for conjugate points can be carried out in various known ways, for example, using the correlation function in a small specified window (usually a small window of 7-15 pixels is used), the optical flow method, or using special descriptors of two-dimensional image points (SIFT, SURF, etc.) but not limited to.

Some of the found conjugate points can be additionally filtered by certain signs (for example, at a specified point in the image low contrast, or its three-dimensional image is characterized by high noise in the vicinity), thereby, the most reliable conjugate points are selected.

In FIG. 7, the white and black dots show the conjugate points on two projections, and the white segments show the corresponding conjugate points from two stereo cameras.

perform three-dimensional reconstruction of the conjugate points;

Since all projection points are obtained as a result of projective conversion from a three-dimensional image of a face to a certain surface for a given stereo camera, then for each of these conjugate points, you can perform the inverse transformation and determine the three-dimensional coordinates of a point in the coordinate system of a given stereo camera.

Three-dimensional reconstruction is performed in the coordinate system of each of the stereo cameras.

Due to the violation of mutual calibration (for example, vibration of load-bearing elements of the stereo camera mount structure), the three-dimensional reconstruction of the conjugate points of one stereo camera may not spatially coincide with the three-dimensional reconstruction of another camera, if you select a common three-dimensional coordinate system for two stereo cameras.

determine the angles of rotation and three-dimensional transfer for the conjugate points in the coordinate system common to all stereo cameras;

To determine the discrepancy between the conjugate points of several stereo cameras, the selection of such an angle of rotation and movement in a common three-dimensional coordinate system that combine the three-dimensional conjugate points of one camera to the points of another camera.

It is known in advance which three-dimensional points should be combined, among the total number of all three-dimensional points in the reconstructed three-dimensional image of the face, since these points are obtained as conjugate.

calculate the relative position of these stereo cameras relative to each other, based on the position of the conjugate points in the coordinate system of each stereo camera, and the position of the conjugate points relative to each other in the common coordinate system, which was defined above.

calibrate the stereo cameras to each other according to the relative position of the stereo cameras relative to each other;

Calibration is performed based on the calculated relative position of the stereo cameras relative to each other in the previous step and the calibration parameters of the cameras inside the stereo camera. The latter are considered to be known in advance and can be determined in the manufacture of a stereo camera by an independent method, for example, using the template calibration algorithm Tsai (A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses - RY Tsai ДЕЕЕ J. Robotics Automat., Pages 323-344, Vol. RA-3, No. 41987)

The purpose of auto-calibration is to determine your own data, such as distortion, focal length, tilt angle, angle of rotation around the axis, camera height, sensor size, sensor resolution, which are determined automatically using known calibration methods [L. Shapiro, J. Stockman. Computer vision. Moscow: BINOM. Laboratory of Knowledge, 2006, pp. 567-578].

An advantage of the proposed method is the use of texture information on projections of facial images, and not just geometric information of a three-dimensional face shape.

It is proposed to use projections of the face image on the same given plane for two or more stereo cameras. This solution ensures the preservation of the geometric proportions of all elements of the face with the correct calibration. With a small calibration discrepancy between stereo cameras, the discrepancy between the projections is insignificant, and in most cases comes down to a shift of small portions of the image from one projection to another, as shown in FIG. 7, without significant deformations of the shape of small portions of the image. Therefore, in this method, the correlation function of points in a small predetermined rectangular window can be effectively used to search for conjugate points.

The proposed autocalibration method has been tested experimentally. The reconstruction was carried out using four time-synchronized cameras. The resolution of the cameras was: 1280 × 1024, frame rate: 12 frames / sec. The cameras were combined into two stereo modules (in two vertical pairs, time synchronized and calibrated with each other) located on the left and right side of the throughput system (see Fig. 1). The distance between the two cameras in the pair was 20 cm, between the stereo modules - about 1 m.

Autocalibration by the proposed method was carried out in conditions of minimum illumination (400-600 lux) at a distance of up to 2 m from the cameras. Testing was carried out on a database of faces (textures + three-dimensional models), consisting of approximately 1800 images of about 600 people.

As a result of this method, it was possible to implement auto-calibration "on the fly" for each passing person.

Using the CUDA computing platform, the processing time for performing automatic calibration by the developed method on a GeForce GTX 680 graphics card (1536 cores with a frequency of 1006 MHz) was 80 ms, i.e. corresponds to real-time operation at a frame rate of 12 per second.

This method made it possible to eliminate the discrepancy in the auto-calibration of stereo modules as a result of vibration of the supporting structural elements (fastening columns) in real time for each new incoming face image.

In FIG. 8 shows a reconstructed frontal image of a face after auto-calibration of stereo cameras in this way, which visually has a high degree of similarity with the reference in FIG. 6 ..

It will be apparent to those skilled in the art that specific embodiments of the method for auto-calibrating stereo cameras used to automatically recognize a person’s face have been described herein for purposes of illustration, various modifications are possible without departing from the scope and essence of the scope of the invention.

The proposed method for auto-calibration of stereo cameras used for automatic recognition of a person’s face is recommended for use at airports, train stations, and the underground, for use in identity identification systems, statistical analysis, and control of attempts to stow through passage of turnstiles, in law enforcement agencies to search for offenders in a stream of people, for identification of computer users, as well as for identification of persons in enterprises with access control.

Claims (5)

1. The method of auto-calibration of stereo cameras used for automatic recognition of a person’s face, including the following steps:
receive from stereo cameras projections of a three-dimensional image with a texture of the face on a two-dimensional surface;
determine a set of conjugate points on the obtained projections of the face image corresponding to the same image areas on two or more stereo cameras, some of the found conjugate points are filtered to select the most reliable points;
perform three-dimensional reconstruction of the conjugate points;
determine the angles of rotation and three-dimensional transfer for the conjugate points in the coordinate system common to all stereo cameras;
calculate the relative position of these stereo cameras relative to each other;
calibrate the stereo cameras to each other according to the relative position of the stereo cameras relative to each other; 2. The method according to claim 1, characterized in that said projection surface on which the conjugate points are calculated is common to two or more stereo cameras. 3. The method according to claim 1, characterized in that the search for conjugate points can be carried out using the correlation function in a small specified window, by the optical flow method, or using special descriptors of points of a two-dimensional image. 4. The method according to p. 1, characterized in that the three-dimensional reconstruction of the conjugate points is performed in the coordinate system of each of the stereo cameras. 5. The method according to p. 1, characterized in that the specified projection surface is a plane.

Images