RU2552123C2 - Method of selecting objects on remote background - Google Patents

Abstract

FIELD: physics, optics.

SUBSTANCE: method relates to optical stereoscopic methods of locating an object in surrounding space. The method includes receiving and recording reference and compared images with two identical optical systems; generating differential images by subtracting the compared image from the reference image and the reference image from the compared image; nulling negative values in the differential images; and determining the distance to the object based on the shift between non-zero fragments of the differential images; wherein the distance between the recording points of each pair of the reference and compared images is successively reduced when objects approach the optical system.

EFFECT: matching the base distance of recording stereo pair frames while optical systems move in space.

4 cl, 8 dwg

Description

Technical field

The invention relates to methods for photographing and filming. More specifically, to optical stereoscopic methods for determining the location of an object in the surrounding space, including determining the distances to objects of interest, as well as their angular coordinates. The invention is applicable, for example, in automatic systems for determining the location of objects in space, in particular in transport, as a means of preventing collisions.

State of the art

Known stereoscopic method for measuring distances and ship range finder-direction finder described in the patent of the Russian Federation No. 2468336, publ. 11/27/2012, G01C 3/02, G01S 17/06, author Guzevich S.N. In the method, the observation axes are pointed at the object, the object displays on the measurement planes orthogonal to the optical axes from the centers of two identical optical devices spaced on a known base. Conducted on the measurement planes through the projection points of the optical axes of the measuring coordinate axes parallel to the base. The positions of the boundary points of the object mappings from the centers of the projections of the optical axes are measured and the distances to the object are calculated using the size of the base as an integral part of the reference parameter. The size of the base and the distance from the centers of the optical devices to the measurement planes are controlled, and the calculations are performed using the product of the base length and the distance from the centers of the optical devices to the measurement planes as a reference parameter.

The disadvantage of this method is that the distance between the cameras (basis) is limited by the dimensions of the base, for example, the ship on which the cameras are installed. If the base is large, then adjustment of the basis is possible. If the base is small, then it is impossible to implement the necessary basis with a mechanical device for moving the cameras. In addition, the method is difficult to implement. An additional mechanical system for changing the distance between the cameras is required. The method and device have insufficient performance in the case of detection (selection) of rapidly moving objects, as well as in the case of use on fast moving objects (ships).

As a prototype, a method for selecting objects on a remote background was selected in RF patent No. 2363018, class. G01S 17/06, publ. 07.27.2009, authors Podgornov V.A., Podgornov S.V., Scherbina A.N. The method consists in receiving and forming the reference and compared images by two identical optical systems installed at a small distance between themselves relative to the distant background. Lenses have parallel main optical axes. Both images are recorded simultaneously. They search for fragments of the images of objects in the images and put them in correspondence with the magnitude of the parallactic shift. The distances to the detected objects in space are calculated by the formula R = L · A / Δ _i , where L is the distance between the centers of the lenses of the optical systems, A is the distance from the photomatrix to the main optical axis of the lens of the optical system, Δ _i is the value of the parallactic shift of the ith image of the object. This method allows with high accuracy to detect (select) single and multiple objects with a predetermined brightness in conditions of high contrast backgrounds.

The disadvantage of this method is the limited range of selection of objects in range, since it is impossible to increase the base distance between optical systems in the process of their operation. This requires the introduction of an additional mechanism for moving optical systems. In addition, the coordinates of highly distant objects of observation are not sufficiently determined.

The objective of the invention is to increase the selection range in the surrounding space of objects during their observation from a moving base (vehicle), without using special additional mechanisms for moving optical systems, while maintaining the accuracy of selection of moving objects.

The technical result consists in increasing the base distance due to the desynchronization of the moment of registration of the frames of the stereo pair in the process of moving optical systems in space. Moreover, in the case of observing moving objects approaching optical systems, the desynchronization time gradually decreases.

Disclosure of invention

To achieve a technical result, a method for selecting objects on a remote background is proposed. The method consists in receiving and forming the reference and compared images with two identical optical systems based on multi-element photodetectors, with parallel main optical axes of their lenses. Both images are recorded, two differential images are formed. The first is by subtracting the compared image from the reference, the second by subtracting the reference from the compared. Zero negative values in difference images. The magnitude of the shift between nonzero fragments of the first difference image and the corresponding nonzero fragments of the second difference image to their maximum coincidence is searched in the direction of the background parallax shift. Similarly, the shift values between non-zero fragments of the second differential image and the corresponding fragments of the first differential image are found. The distances to the detected objects are calculated by the formula R = a × f / Δ _i , where a is the distance between the main optical axes of the lenses, f is the focal length of the lens of the optical system, Δ _i is the shift between the corresponding nonzero fragments of the ith object. Receive and form images in the process of translational movement of optical systems. The motion vector forms an angle with the direction of the main optical axes of the lenses, and each pair of reference and compared images is sequentially recorded at two spaced points of space, the distance between which is selected from the condition for providing a shift between the corresponding nonzero fragments of the difference images by no less than the size of the photosensitive element photodetector, while the distance between the registration points of each pair of reference and compared images in series menshayut when approaching objects to the optical system.

The combination of these essential features allows you to increase the base distance due to the desynchronization of the registration of stereo frames in the process of moving optical systems. This allows you to increase the range of selection of objects in the surrounding space. Moreover, in the case of observation of moving objects approaching optical systems, the desynchronization time gradually decreases as they approach. This allows you to minimize the negative impact of the movement of the observed objects in the time interval between the moments of registration of the frames of the stereo pair, which increases the accuracy of determining the distance to objects (increases the accuracy of selection).

Before receiving the differential images, the compared image is parallactically shifted relative to the reference image to the maximum coincidence of the remote background. These actions are performed in order to eliminate the error when setting the parallelism of the main optical axes of identical optical systems.

The reference and reference image are aligned in brightness until differential images are formed. These actions are performed in order to eliminate the instrumental error of the photosensitivity of photodetectors of optical systems.

In difference images, the values are reset to a certain level corresponding to the sensitivity spread of the photodetectors of optical systems. These actions are performed in order to eliminate noise caused, for example, by optical aberration of the lenses of optical systems.

In our available sources of information, no technical solutions have been found that together contain features similar to the distinguishing features of the proposed method. Therefore, the invention meets the criterion of inventive step.

The presence of new significant features together with the well-known and common with the prototype allowed us to create a new technical solution - a method of selecting an object on a remote background. The method solves the complex task of increasing the range of determining the location of objects in the surrounding space, including moving objects in the process of observing them from a moving base.

Brief description of figures and drawings

The proposed method for determining the location of an object in the surrounding space is illustrated by the drawings:

figure 1 is an optical scheme for registering images of the claimed method;

figure 2 - schematic diagram of the device of optical systems.

figure 3 (a, b) is a schematic representation of a registered reference A1 and the compared A2 image;

figure 4 (a, b) is a diagram of the displacement (cropping) of the image matrices A1 and A2 relative to each other, in the presence of a deviation from parallelism of the main optical axes of the lenses;

Fig. 5 (a, b) is the first and second difference images of Raznost1 and Raznost2, respectively;

6 (a, b) is a diagram for determining the parallactic displacement of objects;

Fig. 7 (a, b) is the result of determining the parallactic displacement of objects, (a is lighter than the surrounding background, b is darker than the background), where the images of the objects are shown each in its own color corresponding to the displacement (color decoding of the correspondence is shown in the middle);

Fig. 8 is a variant of presenting the result of selecting objects against a remote background, the images of objects are highlighted in color, the color corresponding to the distance to the object (color decoding is shown on the right).

Embodiments of the invention

The method is implemented as follows.

Consider the scheme of photographing. To register optical images, two identical optical systems 1 and 2 (Fig. 1) are used, mounted on a movable base 3 (airplane). As shown in FIG. 2, each optical system 1 and 2 consists of a multi-element photodetector 4 and an associated lens 5. The optical systems 1 and 2 are oriented relative to each other so that the main optical axes (dash-dot lines) of their lenses 5 are parallel to each other, (figure 1, 2). Optical systems 1 and 2 are electrically interfaced with a control unit 6, which provides power and control of the entire device as a whole.

The control unit 6 includes power sources, as well as an electrical circuit for controlling the operation of the device, for example, a programmable logic integrated circuit (FPGA).

As multi-element photodetectors 4, for example, complementary metal oxide semiconductor (CMOS) arrays are used. The number of photosensitive elements in the CMOS matrix can be different. In this embodiment of the method, a square matrix of format n by n pixels with side length L is used (FIG. 2).

As lens 5, almost any type of optical lens can be used, providing an overview of the surrounding space at a given solid angle ϑ, determined based on the shooting conditions, and satisfying the requirements for obtaining images with sufficient resolution (Figs. 1, 2). In this embodiment of the method, a single-lens lens with a focal length f is used (FIG. 2).

The resolution of optical systems 1 and 2 is considered sufficient if the images of the observed objects 7 and 8 in the recorded images are formed by at least 1-2 pixels. Exposure during the registration of frames by optical systems is considered optimal if the images of the observed objects 7 and 8 are captured on images without significant blurring.

The base 3, on which the optical systems 1 and 2 are installed, in the case under consideration moves progressively with a constant speed V _c (Fig. 1). The angle between the vector Vc and the main optical axes of the lenses 5 is 90 degrees. The observed objects 7 and 8 move rectilinearly with constant speeds V _rev1 , V _rev2 , respectively (Fig. 1).

At time t _{1, the} optical system 1 receives, using its own lens 5, an optical image of the surrounding space and forms it in the plane of the multi-element photodetector 4, which performs its registration. The registered image is taken as reference A1 (figa).

After a predetermined time interval Δt, at time t ₂ , the optical system 2 receives, using its own lens 5, an optical image of the surrounding space and forms it in the plane of the multi-element photodetector 4, which performs its registration. The registered image is taken as the compared A2 (figb).

The time interval Δt between the time t _{1 of the} registration of the reference image A1 and the time t _{2 of the} registration of the compared image A2 is selected from the following conditions:

where R _max is the distance to the distant background (selected based on the parameters of the observed scene), figure 1; L is the side of the square of the photosensitive area of the CMOS sensor array 4; f is the focal length of the lens 5; V _c - the speed of movement of the base 3 (the speed of the aircraft in flight); n is the number of pixels located along the side of the square of the photosensitive area of the CMOS photodetector array 4.

The above-described scheme of photographing allows you to sequentially register the reference and compared images at two spaced points of space (O ₁ and O ₂ ), (in FIG. 1). The distance O ₁ O ₂ is actually a stereoscopic basis of a . The basis a is selected from the condition for providing a parallactic shift on the recorded stereo pair of images of objects 7 and 8, which are closer than the remote background, by no less than the size of the photosensitive element (pixel) of the photodetector 4. This photographing scheme allows you to organize almost any necessary stereoscopic basis a based on a given value of R _max necessary to increase the selection range of objects 7 and 8 in the surrounding space. This is the technical result of the invention. Stereoscopic basis a = Δt × V _c .

On the reference image (Fig.3A) there are images 9 and 10 corresponding to objects 7 and 8 (Fig.1), observed against a remote background 11, (Fig.3a). On the compared image (fig.3b) there are images 12 and 13, corresponding to objects 7 and 8, which moved during the time interval Δt to position 14 and 15 (Fig.1), observed against a remote background 11 (Fig.3b).

Remote background 11 in the reference (figa) and compared (fig.3b) images located at a distance greater than the value of R _max shifted by less than one pixel. The shift of the distant background 11 in the recorded images is indistinguishably small. As a result of this, the magnitude of the parallactic shift of the background 11 can be neglected and the background 11 can be considered immobile.

The images 12 and 13 in the compared image (Fig.3b) are shifted to the right (the direction of the parallactic shift) relative to the images 9 and 10 in the reference image (Fig.3a). The values of the values of the shifts Δ ₁ , Δ ₂ in the direction of the parallactic displacement of the background 11 depend on the degree of remoteness of objects 7 and 8 from the optical systems 1 and 2, namely, on the distances R _{ob1_t1} , R _{ob1_t2} , and R _{ob2_t1} , R _{ob2_t2} (Fig. 1) . The larger the shifts Δ ₁ , Δ ₂ , the closer the objects 7 and 8 are to the optical systems 1 and 2, and vice versa.

The distances R _{ob1_t1} , and R _{ob2_t1} to objects 7 and 8 are determined by the relative positions of their images 9 and 12 on the reference A1, and 10 and 13 on the compared A2 images, based on the values of the shifts Δ ₁ , Δ ₂ , respectively. Moreover, the smaller the displacement of objects 7 and 8 over the time interval Δt, the more accurately determine the distance R _{ob1_t1} , and R _{ob2_t1} . To minimize the error in measuring the distances to moving objects 7 and 8 approaching the optical systems 1 and 2, it is required to reduce the distances between the recording points O1 and O2 of each subsequent pair of reference A1 and the compared A2 images (i.e., reducing the stereoscopic basis a ). To this end, it is possible to shorten the time interval Δt to the minimum necessary, for example, by using the distance R _max defined in the calculations, which is defined as the distance to the farthest object 7 observed in the previous scene (on the previous pair of reference A1 and compared A2 images). This confirms the achievement of the technical result.

The process of identifying images 9, 10 and 12, 13 of objects 7 and 8 on registered images to determine the parallactic shifts Δ ₁ , Δ ₂ is as follows.

The reference and compared images are represented in the form of matrices A1 and A2 with a size of I × J, respectively (Fig. 4). On the registered images (figa, b) on a remote high-contrast background there are images of objects 9 and 12 lighter than the background surrounding them, and images of objects 10 and 13 are darker than the background.

. Размер получившихся матриц принимают равным I×J и сохраняют их обозначения (A1 и A2).If the main optical axes of the recording systems 1 and 2 have a slight deviation of parallelism between them, the matrices are “cut off” in the direction opposite to the parallactic displacement of the background 11. Fig. 4 shows a diagram of the displacement (cropping) of the image matrices A1 and A2 relative to each other in the presence of deviations from parallelism of the main optical axes of the recording systems 1 and 2, where dx and dy are the horizontal and vertical projections of the parallactic displacement of the distant background 11. The magnitude of the parallactic displacement of the distant background is 11 p is assumed constant for optical systems and is calculated by the formula

. The size of the resulting matrices is taken equal to I × J and their designations are retained (A1 and A2).

In the case of a brightness deviation in the recorded images A1 and A2, caused, for example, by errors in the production of photodetectors 4, the reference A1 and the compared A2 images are aligned in brightness to their maximum coincidence.

The following sequence of actions carried out in the control unit 6 suppresses a remote, high-contrast background and at the same time, objects with previously unknown brightness are highlighted on it (light objects against a dark background or dark objects against a light background):

1. Form two differential images.

The first difference image corresponds to a matrix equal to: Raznoct1 = A1-A2 (figa). The second difference image corresponds to an equal matrix: Raznoct2 = A2-A1 (Fig.5b).

2. In the resulting difference images, zero values are zeroed. If noise is present on the difference images due, for example, to the optical aberrations of the lenses 5, then the zeroing of the values in the difference matrices is carried out starting from a certain positive value, i.e.:

Raznoct1 (i, j) = 0 if Raznoct1 (i, j) <b

Raznoct2 (i, j) = 0 if Raznoct2 (i, j) <b

The value (b) is constant and is determined in advance based on the difference of the distant background in the reference and compared image.

On figa and 5b presents the first and second difference images. On the obtained first difference image (Fig. 5a), fragment 16 corresponding to the brighter object in the reference image A1, as well as fragment 17 corresponding to the darker object in the compared image A2 are highlighted (not zeroed), and all other values are reset. In the obtained second difference image (Fig. 5b), fragment 18 corresponding to the darker object in the reference image A1, as well as fragment 19 corresponding to the brighter object in the compared image A2 are highlighted (not zeroed), and all other values are reset.

To calculate the distances R _{ob1_t1} , R _{ob2_t1} to the detected objects 7 and 8, the parallactic displacements Δ ₁ , Δ _{2 of} fragments corresponding to objects 7 and 8 in the difference images Raznoct1 and Raznoct2 are determined. The determination of bias values is divided into two stages:

1. The first step (determining the displacement of the object lighter than the background) is to determine the corresponding parallactic displacement Δ _i (i = 1, 2, for the case under consideration), as an argument of the function F ₁ (Δ _i ), at which it is minimal. The function F ₁ (Δ _i ) is the module of the arithmetic mean value of the difference of the nonzero fragment of the first difference image and the corresponding section of the second difference image. In this case, a fragment of the first difference image is superimposed on the second difference image with a coordinate offset (from 0 to dX), in the direction of the background parallax, a nonzero fragment of the first difference image and the corresponding section of the second difference image:

where Raznoct1 (X, Y) is a nonzero fragment of the first difference image corresponding to the object; Raznoct1 (X + Δ _i , Y) is the portion of the second difference image in size and geometry corresponding to Raznoct1 (X, Y), and shifted by Δ _i in the direction of the background parallax; N is the fragment size in pixels; dX is the maximum parallactic displacement determined by the detection range of objects and their sizes.

Moreover, the values of the function F ₁ (Δ _i ) for the fragments corresponding to darker objects are constant, due to the absence on the second difference image of the corresponding fragments shifted in the direction of the parallactic shift (Fig. 6a). As a result, Δ _{i is} taken equal to zero, i.e. the fragment is equated to the area of the removed background (figa).

2. The second stage (determining the displacement of the object darker than the background) is to determine Δ _i (i = 1, 2 for the case under consideration) as an argument to the function F ₂ (Δ _i ) at which it is minimal. The function F ₂ (Δ _i ) is the module of the arithmetic mean value of the difference of the nonzero fragment of the second difference image and the corresponding section of the first difference image. In this case, a fragment of the second difference image is superimposed on the first difference image with a coordinate offset (from 0 to dX), in the direction of the background parallax, a nonzero fragment of the second difference image and the corresponding section of the first difference image:

where Raznoct2 (X, Y) is a nonzero fragment of the second difference image corresponding to the object; Raznoct1 (X + Δ _i , Y) is the portion of the first difference image in size and geometry corresponding to Raznoct1 (X, Y) and shifted by Δ _i in the direction of background parallactic displacement.

Moreover, the values of the function F ₂ (Δ _i ) for fragments corresponding to lighter objects are constant, due to the absence on the first difference image of the corresponding fragments shifted in the direction of the parallactic shift (Fig.6b). As a result, Δ _{i is} taken equal to zero, i.e. the fragment is equated to the remote background area (Fig.7b).

The distance to the detected objects 7 and 8 is calculated by the formula R _{vol.i_t1} = a · f / Δ _i , where a is the distance between the main optical axes of the lenses 5, f is the focal length of the lens 5, Δ _i is the amount of shift between the corresponding nonzero fragments i -th object (the magnitude of the parallactic shift of the i-th object).

As a result, at the signal output of the control unit 6, information is obtained about the detected objects 7 and 8 located at different distances from the optical systems 1 and 2 in the form of the coordinates of the objects in the reference image and the distances to the objects set by them (Fig. 8).

At the same time, the degree of remoteness of objects 7 and 8 from video systems 1 and 2 is shown to the operator in the form of a color gamut (or shades of gray), which corresponds to a range of distances (Fig. 8).

The spatial angular coordinates of objects 7 and 8 are determined according to the coordinates of the images of objects 16 and 17 in the reference image A1.

Industrial applicability

The claimed method of selecting objects on a remote background can be used in automatic systems for determining the location of objects in space. In particular, in air transport to prevent collisions of aircraft in the air. For example, in civil aviation on airliners, helicopters.

The technical implementation of the method of selecting objects on a remote background in a particular device is quite feasible. When creating optical systems and a control unit, optical-electronic devices mastered by the industry, structural units, and elements can be used. The algorithm of the specified device is also technically feasible. In a device that implements a method of selecting objects on a remote background, commercially available video cameras can be used as optical systems, which capture images of the surrounding space at a frequency of 30 frames per second. Two identical video cameras are connected in parallel to the control unit, for example, FPGA. The time of desynchronization between the moments of registration of frames by video cameras is set by FPGA, which in turn receives registered images and carries out their pipelining with the output of the calculation results in the form of images (Fig. 8) to the operator (pilot) monitor.

When using video cameras with a resolution of 1920 × 1080 pixels (n = 1920) and a photosensitive area of photodetectors 14.2 × 8 mm (L = 14.2) equipped with lenses with a focal length (f) of 4 mm (viewing angle ϑ is 122 × 90 degrees ), the maximum range (R _max ) of object selection is 1,500 m, when the cameras are out of sync in one frame (Δt = 1/30 s) and the speed (V _c ) of the base movement (aircraft flight speed) is 300 km / h.

Claims (4)

1. A method of selecting objects on a remote background, which consists in receiving and forming the reference and compared images with two identical optical systems based on multi-element photodetectors, with parallel main optical axes of their lenses, recording both images, forming two difference images, the first by subtracting the compared image from reference, the second - by subtracting the reference from the comparison, zeroing the negative values in the difference images, searching in the direction of parallactic the background shift of the magnitude of the shift between non-zero fragments of the first differential image and their corresponding non-zero fragments of the second differential image to their maximum match, similar to finding the magnitude of the shift between non-zero fragments of the second differential image and the corresponding fragments of the first differential image, calculating the distance to the detected objects using the formula R = a · f / Δ _i , where a is the distance between the main optical axes of the lenses, f is the focal length of the optical lens system, Δ _i is the shift between the corresponding non-zero fragments of the i-th object, characterized in that they receive and form images in the process of translational movement of the optical systems, the motion vector forms an angle with the direction of the main optical axes of the lenses of the optical systems, with each pair of reference and the compared image is sequentially recorded at two spaced points of space, the distance between which is selected from the condition for ensuring a shift between the corresponding nonzero fragments of difference images by an amount no less than the size of the photosensitive element of the photodetector, while the distance between the registration points of each pair of reference and compared images is successively reduced as objects approach optical systems. 2. The method of selecting objects on a remote background according to claim 1, characterized in that before receiving the difference images, the compared image is parallactically shifted relative to the reference image to the maximum coincidence of the remote background. 3. The method of selecting objects on a remote background according to claim 1, characterized in that the reference and compared image prior to the formation of difference images are aligned in brightness. 4. The method of selecting objects on a remote background according to claim 1, characterized in that in the difference images, the values are zeroed from a certain level corresponding to the sensitivity spread of the photodetectors of optical systems.

Images