BG110929A - A method and a device for passive recognition of moving objects against a mobile background - Google Patents

Abstract

The method and the device provide the recognition of movement and the identification of moving objects against a mobile background by using epipolar geometry and by incorporating a virtual observer to each discovered mobile object. The objects consist of at least nine points of nonvaried geometry and, with the help of their virtual observer, the following applicabilities are realized: a three-dimensional re-construction of a dynamic scene

Description

TECHNICAL FIELD

The invention relates to a method and device for passive recognition of moving objects on a moving background, which are used in the field of three-dimensional scanning, geodesy and mapping, passive navigation with terrestrial correlation, detection, classification by comparison with standards and object tracking.

DEFINITIONS OF THE TERMINOLOGY USED

For the sake of clarity, the following definitions of the terms used in the description of the invention are given below:

A scene is a collection of geometric solids called objects in three-dimensional space. For the observer, they exist as a perspective projection.

Perspective projection is the geometric location of the intersection points of rays intersecting a projective plane and passing through points from the scene, and the point of view of the observer called the pole.

An observer is a device that receives central projections on its projective plane with a focus coinciding with the pole and calibration parameters defining its position, direction and orientation, as well as the focal length between its focus and its projective plane.

Epipolus is the projection of the focus of an observer on the perspective projection of the same scene obtained by another observer.

Epipolar geometry is the set of relations between the projections of points on the stage, called corresponding, and the epipoles on at least two projections, which relations allow projective reconstruction of stationary objects from the stage.

Metric reconstruction of the scene is the obtaining of the three-dimensional coordinates of points from the scene and the observers with the available pair of projections, epipoles and a priori information, in this case the focal lengths of the observers.

A correlator in this case is a module for comparing characteristic points of projections, together with their surrounding area, in order to find the corresponding points.

ft ft ft ft ft · ft · ft ft ft ft ft ♦ ft ·· ft • · · ····· ft ft ····· ft ft ft • ft ·· ·· ··· ··· ···

BACKGROUND OF THE INVENTION

Numerous active and passive methods and devices for recognizing motion and identifying moving objects are known.

Known active methods and devices are based on the Doppler effect, which is a change in the wavelength of electromagnetic energy reflected by a moving object. A major disadvantage of these is the need for a source of such energy belonging to the devices. An example of this is radar onboard systems that are vulnerable to both interference caused by radio-electronic counteracting agents and as a unmasking factor that can be used to target infectious agents.

Known passive methods and devices, whose very important advantage is the lack of the need to emit any of their own electromagnetic energy, have as a major drawback the requirement for stationarity. They ensure that moving objects are detected and identified only from a stationary device to a scene.

The closest known in nature to a method of passive recognition of moving objects on a fixed background [1] and [2]; includes the following operations: obtaining sequentially over time digital images as arrays of pixels with attributes luminance and possibly color; pixel-by-pair comparison of consecutive images, which in the simplest case is performed as a difference in pixel brightness; summing the obtained absolute values of the differences for the whole pair of images; deciding whether or not there is movement as a result of comparing the amount received with some preset threshold value. In the case of a sufficiently large difference between the attributes of the moving objects relative to the scene, the moving or stationary background may be ignored and the movement of the recognized objects relative to the device-related coordinate system may be recorded and monitored.

These operations can be complicated to provide noise immunity and reliability, but what is characteristic of them is the use of only planar elements - from pixels to contours and areas for each of the images, without regard to their three-dimensional metric or projective compliance.

Methods are known for finding the structure of a scene on the basis of its perspective projections, obtained by a moving observer, and finding the ····· ··· <· · · · · · · · · · · · · · · Projective and metric matching based on epipolar geometry. Methods for finding the epipoles necessary for this purpose, using random sequence matching algorithms (RANSAC), are also known. These methods are currently applicable only for the reconstruction of stationary scenes from projections of two or more views.

Known passive methods and devices for recognizing moving objects on a stationary background are not applicable for motion recognition and identification of moving objects in the presence of movement of the observer device towards the scene, that is, on a moving background.

SUMMARY OF THE INVENTION

It is an object of the invention to provide a method and apparatus for passive recognition of moving objects on a moving background.

This task is solved by a method of passive recognition of moving objects on a moving background, which includes the following operations: a sequence of images is generated, representing perspective projections of the scene by an observer; each image is pre-processed including spectral averaging, digital filtration, contour and region separation, characterization; at least 9 pairs of corresponding points are determined by two-dimensional correlation of the characteristic points of each pair of consecutive images; a pair of epipoles is determined for each series of eight randomly selected corresponding points. The method according to the invention performs homography of all pairs of defined characteristic points by using three pairs of the series of eight characteristic points for which a pair of epipoles are found, and the pair of epipoluses themselves as baseline points for the homography; a second homography of the resulting pair of joint projections is performed with four output baselines lying on a pair of epipolar lines that align with the vertices of a horizontally oriented square, which results in the sending of the joint epipoles to infinity horizontally; looking for objects with one-dimensional correlation using the fact that the vertical coordinates of the transformed characteristic points have the same values if they belong to the same geometrically unchanged object from the scene; each of the pairs of characteristic points with the same vertical coordinates, together with the eight pairs used to find the epipoles, · · 4 · 4 · 4 · · · · · · · · · 4 4 · · · · • ••• 44 · · · · 4 ·· ·· · 4 · ······ characteristic points are assumed to belong to the same object in the scene; a virtual observer represented by its epipoles is added to each object found; remove the objects found from the set of characteristic points of the projections; the process of searching for new objects is repeated until all the objects are recognized or when there are points that cannot be defined as a geometrically unchanged object.

Finding only one object to which all the characteristic points in the scene belong is treated as a sign of lack of movement in the scene.

It is possible to use the method of passive recognition of moving objects on a moving background and to identify the individual objects found on the scene by comparing them with standards, which is performed for each standard separately. Instead of one of the two consecutive images, an image with a central projection of the standard is used as described above.

It is possible to use the method described above to perform a metric reconstruction for each of the objects found on the scene and to restore the mutual position of the objects and their virtual observers by using their epipoles and focal length of projections.

Using the method described, it is possible to determine motion parameters including the location, trajectory, kinematics, and dynamics of recognized objects and their virtual observers, each relative to the other, using three reciprocal positions obtained from the processed images on the scene and the time intervals between them.

This task is also solved with a device for passive recognition of moving objects on a moving background, implementing the method, which includes: a digital camera with a minimum exposure time and an interface, providing as direct access to the attributes of each pixel of the images it receives, representing the central planar projections of the scene and the focal length value used for each of them; an image pre-processing module designed to perform digital filtering, spectral smoothing, separation of areas and contours, and finding feature points in images acquired by the digital camera; memory, in which, in addition to the system part necessary to control the operation of the device, there are separate areas for storing a sequence of images, feature points and · · · · · · · · · · · · <· · · · · · · · · · · · · · · · · · · · · · · · · · · Objects; a control module that controls and synchronizes the overall operation of the device; a module for utilizing the information received, which in the simplest case provides an interface for accessing it. The device according to the invention also comprises: a two-dimensional correlator searching for corresponding pairs of points in the memory for a sequence of images; a module for the calculation of epipoles, processing a series of eight corresponding points; a one-dimensional correlator that performs a sequence of two homographs, verifying and establishing belonging to an object at corresponding points; self-study module for predefined standards; modules for metric reconstruction, object recognition and trajectory calculation used to recover the three-dimensional geometry of objects from the scene, the location of their virtual observers, their relative position and motion parameters - trajectories, linear and angular velocities and accelerations.

The advantage of the proposed method and device for passive recognition of moving objects on a moving background is the possibility of recognizing moving objects by a moving observer, which allows: recording motion in the scene; identification of recognized objects from the scene by comparison with standards; three-dimensional reconstruction of objects and positions of their virtual observer; finding the mutual location of objects and their virtual observers in three-dimensional space; tracking the parameters of the movement of objects against each other, objects and virtual observers relative to each other, and virtual observers against each other.

EXPLANATION OF THE FIGURES Attached

Figure 1 shows the acquisition of a sequence of scene projections and the use of epipolar geometry.

Figure 2 shows the sequence of image pre-processing operations.

Figure 3 shows the finding of corresponding points and the calculation of the epipoles of an object from the scene.

Figure 4 shows the pairing of images with two consecutive homographs and determining the affiliation of corresponding points to an object.

Figure 5 shows the basic operating cycle of the device.

· · · · · · · ·

4 4 4 · · 4 · 4 · · · · · ·

9 9 9 9 9 9 99

44 44 ··· ······

Figure 6 shows the results of a metric reconstruction of the scene and finding the motion parameters of an object

Figure 7 shows a comparison with a standard using only perspective projections, invariant to the relative positions of objects, standards and observers.

Figure 8 shows a block diagram of a device implementing the method of the invention.

EXAMPLES FOR THE IMPLEMENTATION OF THE INVENTION

The passive motion detection device on a moving background implementing the method includes: a digital camera 1 with a minimum exposure time and an interface providing as direct access to the attributes of each pixel of the images it receives, representing central planar projections of the scene, both the focal length value used for each of them; an image pre-processing module 2 designed to perform digital filtering, spectral alignment, separation of areas and contours, and finding feature points in the digital camera image 1; memory for storing a sequence of images 3; a two-dimensional correlator 4 searching for corresponding pairs of dots in the images stored in the memory for storing a sequence of images 3; memory for characteristic points 5; a module for the calculation of epipoles 6, processing series of eight corresponding points; a one-dimensional correlator 7 performing a sequence of two homographs, verifying and establishing affiliation to an object at corresponding points; object memory 8; a module for utilizing the information received 9, which in the simplest case provides an interface for accessing it; a control module 10 that controls and synchronizes the overall operation of the device; metric reconstruction modules 11; self-study module for predefined standards 12; object recognition module 13; module for calculating trajectories 14 used to recover the three-dimensional geometry of objects from the scene, the location of their virtual observers, their relative position and motion parameters - trajectories, linear and angular velocities and accelerations.

· · · · · · ·

* · • • 99 • 9 • · · • · • · · 9 • 9 9 9 · • • • • · ♦ ·· «» • · « ··· ···

SUMMARY OF THE INVENTION

The operation of the device and the nature of the method it uses can be illustrated by the arrangement shown in Figure 1. It contains an example scene and two consecutive positions 01 and 02 of an observer on whose projective planes two images VI and V2 are projected centrally. . The breakpoints E1 and E2 of line 01-02 connecting the two observer positions to the two projecting planes of the scene images are epipolyms for the two images VI and V2.

Each of the digital camera 1 sequence of images obtained by the observing digital camera, as described in cited literature 1, pp. 4749, 90, 95, is subjected to image pre-processing in module 2, including spectrum normalization, as specified in References 1, pp. 96-98, digital filtration described in the same literature on pp. 98-102, 108, and digital processing to detect their characteristic areas, contours, and, on the basis of them, to find probable characteristic points, such as described in the same literature of pp. 182, 190. Figure 2 shows the sequence of image pre-processing operations, where: 1h is an image obtained from a digital camera; I02 is an outline image with black lines; 1oz is an image with recognized feature points, indicated by X whose centers coincide with their geometric location. Each received and processed image is stored in the image sequence memory 3.

The characteristic points found for each pair of consecutive images are submitted for verification in the two-dimensional correlator 4. Its purpose is to detect possible corresponding points, each pair of which represents the projection of a point from the scene onto the projective planes of the consecutive positions of the observer. The two-dimensional correlator detects them by comparing the attributes of brightness and possibly color of the pixels surrounding each pair of neighborhoods, as described in Literature 1, p. 196. During initialization, characteristic points stored in the image sequence 3 memory are processed. The data from the already recognized points of the first of the new pair of images stored in the memory for the points 5 are then first processed in the work process, as well as information from the module for predefined standards 12, when it is not required identification of recognized objects.

The concept of separating elements from the scene into objects with invariable geometry based on their mutual motion and assigning them to their virtual • · · a · · · · 4 • · · e ···· • 44 «ф ··· · ·

4 4 4 4 4 44

4444444 44 • 4 44 44 <·· 444 444g observers are at the heart of the method and operation of its implementing device. Each such object consists generally of at least nine corresponding pairs of points, together with eight epipoly pairs calculated for eight of them, and the ninth pair is an indicator of the integrity of the object. The image pair A in Figure 3 illustrates the separation of the scene of objects. The object G1 is stationary, and the object G2 may also be stationary, or change its position to the position G3 shown in Ip. In the first case, the two fixed objects are perceived as one, and in the presence of displacement they are recognized as two different objects. In the same view A of Figure 3, the corresponding 2 points, numbered according to their image and order, are shown and found by the 2D correlator. Detection of objects in the scene requires verification of their projective correspondence using epipolar geometry. The corresponding points obtained form a tail for processing, and from these, on the basis of a random selection, a series of 8 points are formed, for which the probable epipoles for the pair of projections I] and 1 ₂ are calculated on the observed scene, as described in 2, page 240. , 243, The calculated locations of epipoles E1 and E2 for the corresponding corresponding points of view A in figure 3 are shown in view B of the same figure.

Object integrity checks are performed using the one-dimensional correlator 7. In order to make the fitting only horizontal, the T and T images together with the characteristic and corresponding points found are transformed with a sequence of two homographs. The base points P11-P21, P12P22 and P13-P23 for the first homograph are three of the series of eight corresponding points used to find the epipoles E1 and E2, as well as the epipoles E1 and E2, which are the fourth pair of base points, as shown in Figure S1 of Figure 4, where Pl, P2, P3 and E12 are denoted after matching, respectively.

The second homography is applied simultaneously to the joint images thus obtained In, using the starting points of the two epipolar line pairs of points Al, A2, A3 and A4, and the points of displacement using the vertices Bl, B2, B3 and B4 in a horizontally oriented square, which sends the joint epipoles to the infinity horizontally. This sequence of two homographs ensures the uniformity of the vertical coordinates of all the corresponding points found, whose geometry has not changed in the two images used. The illustrations S2 and S3 of Figure 4 show the application of the second homograph on joint projections in a fixed and moving object.

from the scene being watched. The criterion for the presence of motion is illustrated in C1 in the same figure, where R1 denotes no motion and coplanarity of the corresponding points with the plane P1-P2-P3, R2 denotes no motion due to the coincidence of the vertical coordinates of the corresponding points, a R3 is a result indicating movement.

After applying the two homographs, the one-dimensional correlator 7 seeks matching in the same way as the two-dimensional correlator 4, but only horizontally. First the relevant points from the scene, then all the characteristic points, are processed. As a result, all the corresponding points belonging to the virtual observer object determined by the calculated epipoles E1 and E2 are obtained.

If no match is found, it is decided that the eight-point series of corresponding points for which the epipoles are calculated belong to different objects, such as the point P110 with respect to the others in image Ιι in Figure A of Figure 3. Following is the processing of a new series of eight randomly selected matching points, finding the epipoles and matching until they are found, as shown in the example in Figure C of Figure 3. All matching matching points belonging to the object, along with its epipoles, are stored in memory 8 th of objects by removing the queue for processing.

The process of searching for objects continues cyclically until the possibilities for finding a new object are exhausted. Figure 5 shows the basic operating cycle of the device, comprising separating the recognized object G1 from the projections of the scene W1, recognizing the object G2 on W2, which has moved relative to G1 and has its own epipoles E21 and E22 defining its virtual observer. The end-of-cycle indication is shown in W3 projections where no new object can be recognized. The characteristic points remaining in the W3 projections are derived from the variable shading of contours of moving objects or as a sign of geometric deformation of elements of the scene. After exiting the cycle, the device has information about the presence or absence of movement. Indication of no movement is the belonging of all found relevant points from the scene to only one object.

Depending on the purpose of the device, the processing proceeds with the calculation of the three-dimensional geometry of the objects found from the scene and the positions of their virtual observers, as described in cited literature 2, page 277, which is performed by the metric reconstruction module 11. Thus obtained • · · · ·

information is used when the device operates as a three-dimensional scanner and as a means of remote sensing and passive navigation.

During operation, the device can self-train by complementing three-dimensional models of found objects with new attachments, remembering those that are not currently visible, and adding new objects to the module for predefined standards 11. Where object tracking is required, a time sequence of metrically reconstructed three-dimensional coordinates in their associated coordinate systems, the coordinates of their virtual observers, and their corresponding epipoles are used. The trajectory calculation module 14 finds the relative position, linear and angular velocities and accelerations that allow the calculation of the parameters of the mutual motion of the objects - trajectory, kinematics and dynamics of each other, each of them with respect to each of their virtual observers, as well as and the virtual observers themselves against each other. Object tracking finds application as a source of data for self-targeting or remote targeting, or for managing the desired distance from a pursuit. Figure 6 shows the results of metric reconstruction of the observed example scene by obtaining a three-dimensional model of the two objects G1 and G2 with the elimination of their mutual influence as a result of R1, finding the coordinates and orientation of the observer 01 in a coordinate-related system for passive navigation in the result R2 and finding the trajectory T1 and the motion parameters of the objects, both relative to each other and to their virtual observers, used to track and vector an object or observer in result R3.

In parallel, if object recognition is required compared to a reference, it is performed in the same way as in the object finder cycle, but here a correspondence is sought between perspective projections of predefined standards and objects found. This is necessary when the device is used for passive navigation with terrain correlation, passive target recognition and passive recognition of its / foreign / neutral. Figure 7, Figure 1b shows a comparison with reference Ml using only perspective projections G1-G4 of objects from an observed scene, which is invariant to the relative location of the objects, the standard and the observer. The comparison process is illustrated in 1c for each of the objects G1-G4 with the standard M1, where differences in DI, D2, D3 and D4 are found.

The geometry of the objects relative to the standard. The result is illustrated in 1 cm, where the metric reconstruction of the objects and their differences are shown.

The completion of the processing of the image pair VI and V2 re-initializes a further external cycle controlled by the control module 10. After the processing is completed, the next image pair is formed, the second image V2 being first and the newly received image processing module 2 stored in the sequence memory image 3 becomes second. The processing of the new pair of images is done as described above. The results obtained from each circuit of the external loop, performed as the basic mode of operation of the device, are fed to the module for disposal of the information received 9. In the simplest case, this module 9 contains a memory for results and an interface for accessing them from external the device components of systems of which it is a part.

The main advantage of being able to detect and identify moving objects on a moving background is to ignore the observer's own movement to the surrounding scene. The observed scene is decomposed into objects on the basis of their mutual motion, invariant to the positions of the observer, which is achieved using epipolar geometry. The corresponding points found by the 2D correlator 4 are used by the epipoly 6 calculation module, which are then identified as belonging or not to a moving but geometrically unchanged object from the scene by means of a projective compliance check performed in the one-dimensional correlator 7. In the same way, but using predefined projections of a standard, instead of one of the compared images on the scene, identification of the objects found is made, which is invariant to their geometric location and scale, and is based only on projective conformity. Metric reconstruction of objects, together with their attached virtual observers, allows the restoration of their three-dimensional coordinates, the position and parameters of their mutual movement, as well as self-learning of the device.

At present, the method and the device have been implemented software as a technological demonstrator, serving to verify and analyze the key concepts and solutions embedded in them, and to prove their workability. All illustrations in Figures 1 to 7 are automatically prepared and are the result of the operation of the process demonstrator of the method and apparatus.

·· 99 ·

REFERENCES:

1. Computer Vision Algorithms and Applications by Richard Szeliski, 2011, Springer,

ISBN 978-1-84882-934-3

2. Multiple View Geometry in Computer Vision by Richard Hartley and Andrew

Zissermanq, 2003, Cambridge University Press, ICBN 0521 54051

Claims (5)

1. A method of passive recognition of moving objects on a moving background, which includes the following operations: • A sequence of images is generated, representing perspective projections of the scene by an observer. • Each image is pre-processed, including: - averaging of the spectra - digital filtration; - separation of contours and regions; - identification of characteristic points; • at least 9 pairs of corresponding points are determined by two-dimensional correlation of the characteristic points of each pair of consecutive images; • a pair of epipoles is determined for each series of eight randomly selected points; characterized in that: • homographs of all pairs of defined characteristic points are made using three pairs of the series of eight characteristic points for which a pair of epipoles are found, and the pair of epipoluses themselves, as base points for the homography; • a second homography of the resulting pair of joint projections is performed with four output baselines lying on a pair of epipolar lines that align with the vertices of a horizontally oriented square, which results in the sending of the joint epipoles to infinity horizontally; • objects with one-dimensional correlation are searched using the fact that the vertical coordinates of the transformed characteristic points have the same values if they belong to the same geometrically unchanged object from the scene; • each of the pairs of characteristic points with the same vertical coordinates, together with the eight pairs of characteristic points used to find the epipoles, are assumed to belong to the same object in the scene; • a virtual observer represented by its epipoles is added to each object found; • remove the objects found from the set of characteristic projection points; • Repeat the search with the search for new objects until all objects are recognized or when there are points that cannot be defined as geometrically unchanged objects. © • finding only one object to which all the characteristic points in the scene belong is treated as a sign of lack of movement in the scene. A method for passively recognizing moving objects on a moving background according to claim 1, characterized in that the individual objects found on the scene are identified by comparing them with standards, which is performed for each standard individually, instead of one of the two sequential images a central projection image of the standard is used. 3. A method for passively recognizing moving objects on a moving background according to claims 1 and 2, characterized in that a metric reconstruction is performed for each of the objects found on the scene and the mutual location of the objects and their virtual observers is restored by using their epipoles and focal length of projections. 4. A method for passively recognizing moving objects on a movable background according to claims 1, 2 and 3, characterized in that motion parameters are determined including the location, trajectory, kinematics and dynamics of the recognized objects and their virtual observers each to the others by using their three reciprocal placements from the processed images on stage and the intervals between them. 5. A device for the passive recognition of moving objects on a movable background, implementing the method according to claims 1, 2, 3 and 4, which includes: • a digital camera (1) with a minimum exposure time and an interface providing both direct access to the attributes of each pixel of images obtained from the center plane projections of the scene and of the image used value of focal length for each of them; • an image pre-processing module (2) designed to perform digital filtering, spectral alignment, separation of areas and contours, and to find characteristic points in the digital camera (1) images obtained; Memory in which, in addition to the system part necessary for managing the operation of the device, there are separate areas for storing a sequence of images (3), feature points (5) and objects (8); A control module (10) that controls and synchronizes the overall operation of the device; A module for utilizing the received information (9), which in the simplest case provides an interface for accessing it; characterized in that it also contains: • a two-dimensional correlator (4) searching for corresponding pairs of points in the memory for the sequence of images (3); A module for the calculation of epipoles (6), processing series of eight corresponding points; • a one-dimensional correlator (7) performing a sequence of two homographs, verifying and establishing affiliation to an object at corresponding points; • self-study module for predefined standards (12); • modules for metric reconstruction (11), for object recognition (13) and for the calculation of trajectories (14) used to recover the three-dimensional geometry of objects from the scene, the location of their virtual observers, their relative position and motion parameters - trajectories, linear and angular velocities and accelerations.