RU2746614C1 - Method for suppressing backlight when forming images of road environment in front of vehicle and device for implementing method - Google Patents

Abstract

FIELD: image formation.

SUBSTANCE: group of inventions relates to devices for suppressing negative illumination of oncoming radiation when forming images of the terrain in front of the vehicle. The backlight suppression device contains a computing module and two associated source modules and a receiver module. The source modules are used as the main high and / or low beam headlamps for a vehicle. The computing module is designed to collect data from modules, process data and form an image and is configured to generate pseudo-random masks for source modules, calculate the spatial distribution of light intensity at the output from sources, process data received from a receiver module, calculate a cross-correlation function between a signal, received from the receiver, and the calculated light intensities at the output from the sources, the formation of two 2D images of the terrain using the principle of phantom images and the formation of a 3D image using the principle of stereoscopic images.

EFFECT: increased reliability of vehicle control in conditions of strong light interference.

15 cl, 3 dwg

Description

The invention relates to computer vision devices for vehicles, namely, to devices for suppressing negative illumination of oncoming radiation when forming images of the terrain in front of the vehicle, and is aimed at improving the quality of the generated images in conditions of oncoming illumination.

The invention is applicable, in particular, in imaging systems for unmanned vehicles.

Currently, the main imaging method is video capture with a camera. The image obtained with the help of the camera can subsequently be displayed on the driver's screen or used by on-board systems when driving in an automatic (unmanned) mode. The problem of the negative influence of light exposure in such devices is solved by different methods.

In US patent 20180012374 "Image capturing device and method, program, and record medium to perform exposure control based on the brightness in an attention area corresponding to a detected object" this area. Based on the brightness value, the device changes the exposure time on the camera.

In the utility model RU 169 980 U1 "Matrix optical radiation attenuator", parasitic illumination is suppressed by introducing a matrix optical shutter into the optical system, made in the form of a digital micromirror device. The micro-mirror device makes it possible to exclude the background illumination of the light-sensitive surface due to the local attenuation of the flux of optical radiation from high-intensity point light sources, for example, from the headlights of an oncoming car, street lights, etc.

There are also known methods of constructing images without using video cameras. Devices using cameraless imaging techniques include optical lidars and recently introduced phantom imaging systems. A method for constructing an image of the surrounding space using the method of phantom images is known in the literature by Yang, Zhaohua, et al. "3-D Computational Ghost Imaging With Extended Depth of Field for Measurement." IEEE Transactions on Instrumentation and Measurement 68.12 (2019): 4906-4912. The paper describes a method for obtaining a three-dimensional image using two phantom images. The device has two micromirror matrices separated by some distance. There are also two optical systems and a backlight system. With their help, the depicted object is illuminated with random patterns (so-called speckle pictures), and from two different angles. The light scattered by the object is registered by a non-spatial resolution bucket-detector. Using the methods of phantom optics, the image of an object in two different angles is restored, and its three-dimensional image is constructed using the principles of binocular vision. It is noted that, in addition to constructing a three-dimensional image, the use of the phantom image method allows increasing the depth of field.

The proposed invention of backlight suppression in the formation of images of the road environment in front of a vehicle is also based on the phantom imaging method and differs from [Yang, Zhaohua, et al] in that it allows the backlight to be suppressed. In addition, a camera is used instead of a non-space-resolving detector. In conditions of strong light interference, some of the camera pixels are in saturation mode, and their signal is not taken into account when constructing a phantom image. Only the signal that lies in the dynamic range of the camera and gives information about the image is taken into account. Using the camera allows you to solve one of the important problems that occur when calculating phantom images - the need to collect large statistics for image formation. Since the correlation function is calculated pointwise, unlike, for example, from [Yang, Zhaohua, et al], the number of realizations over which averaging is carried out decreases by at least 2 orders of magnitude.

The technical problem of the claimed invention is to improve the reliability of control of vehicles, especially unmanned vehicles, in conditions of strong light interference.

The technical result consists in increasing the accuracy of the received information about the surrounding traffic situation with a greater value of the permissible external illumination, and is achieved by suppressing the influence of external illumination on the quality of the obtained 2D and 3D images using the phantom image technique.

The technical result is achieved due to the fact that the method for suppressing illumination during the formation of images of the road environment in front of the vehicle contains the stages at which: light is formed with a pseudo-random intensity distribution, while in each source the light from the LED falls on the corresponding matrix of micro-mirrors, for each a set of pseudo-random masks is generated, the masks are sent to the control units of the micro-mirror matrices, in accordance with each mask of the micro-mirror matrix, the radiation of the photodiode is modulated, so that the specified radiation is directed to the area in front of the vehicle, using a high-speed video camera is recorded light from the illuminated area, build 2D and 3D images of the terrain in front of the vehicle by the method of phantom images.

и

, где

- корреляционная функция между сигналом в пикселе видеокамеры с радиус-вектором r и сигналом в соответствующим пикселе в матрице микро-зеркал первого источника, а

аналогичная функция, но для матрицы микро-зеркал второго источника.

и

являются 2D изображениями области перед транспортным средством и представляют из себя стереопару, а 3D изображение строится на основе полученной стереопары. An additional feature is that a 3D image of the terrain in front of the vehicle is built using an algorithm that includes stages at which two cross correlation functions are calculated for each pixel of the video camera

and

where

is the correlation function between the signal in the pixel of the video camera with the radius vector r and the signal in the corresponding pixel in the matrix of micro-mirrors of the first source, and

a similar function, but for the second source micro-mirror array.

and

are 2D images of the area in front of the vehicle and represent a stereopair, and the 3D image is built on the basis of the resulting stereopair.

The technical result is also achieved in a device for suppressing backlight when forming images of a road environment in front of a vehicle, containing a computing module and associated at least two source modules and at least one receiver module, characterized in that the source modules are used as main headlights of a high and / or low beam for a vehicle and are made with the possibility of illumination both in the usual and in the infrared range, and the computing module is designed to collect data from the remaining modules, process data and form an image and is configured to generate pseudo-random masks for the modules sources, calculating the spatial distribution of the light intensity at the output from the sources, processing the data received from the receiver module, calculating the cross correlation function between the signal received from the receiver and the calculated light intensities at the output from the sources, formation of 2 2D images of terrain by using the principle of phantom images, forming a 3D image by using the principle of stereoscopic images.

An additional feature is that the source modules contain a matrix of micromirrors and an LED.

An additional feature is that the source modules contain a two-dimensional array of LEDs, the radiation of which is modulated in time and space.

An additional feature is that the source modules contain a liquid crystal spatial light modulator

An additional feature is that the source modules contain a 1D array of LEDs and a spatial beam scanner

An additional feature is that the source modules contain a fast beam scanner.

An additional feature is that the receiver module contains a high-speed video camera.

An additional feature is that the video camera registers radiation from each radiation source included in the device, as well as radiation from external sources.

An additional feature is that the receiver module contains a photodetector operating in a linear mode, and the image is built from the calculation of correlations between the total signal from this photodiode and the signal in each pixel of the micro-mirror arrays.

An additional feature is that the power supply of the used photodetector is modulated synchronously with the light source.

An additional feature is that only one radiation source is used for illumination, and the radiation is recorded by two video cameras remote from each other.

An additional feature is that two remote photodetectors are used.

An additional feature is that the masks are pre-generated.

An additional feature is that it uses one source module, one photodiode and builds a 2D image of the area in front of the vehicle.

An additional feature is that one light source and one camera are used as a stereopair, and the displacement map is built based on the offset of the correlation function of the source and the matrix receiver.

The claimed invention is illustrated using the drawings:

FIG. 1- schematic diagram of the device;

FIG. 2- schematic diagram of the source;

Fig. 3 is a schematic diagram of a receiver;

The main technological advantages are as follows:

1. The device simultaneously performs three functions: illumination of the road environment, imaging and suppression of backlight;

2. The device is capable of suppressing the illumination from another vehicle as the headlights of which exactly the same device is used;

3. Ability to completely suppress backlight;

4. The high frequency of modulation of radiation (more than 10 kHz) is indistinguishable by the human eye and video cameras used as sensors on unmanned vehicles, which will allow the device to be used as the main headlights of a vehicle.

5. High frequency modulation of radiation allows to achieve high speed of data collection from the sensor (from 50 Hz).

6. Since the way of the sensor operation is not critical to the spectral band of radiation, the sensor can be made both with an emitter in the visible range (in the form factor of a headlamp or side light), and with an emitter in the infrared range.

7. Two devices, being used simultaneously (for example, on two cars), do not affect the operation of each other, in particular, the ability to suppress the negative influence of stray light

An example of using the invention, implemented according to claim 1 in the claims as headlights of a car.

One of the possible implementations is described below, based on the following scheme (Fig. 1):

1. Source modules (4) are installed instead of car headlights.

2. Each source includes:

- Light source Philips X-tremeUltinon LED gen2;

- Micro-mirror modulator DLPLCR70EVM with switching frequency up to 30 kHz;

- Micro-mirror resonator controller DLPLCRC410EVM;

3. The receiver module (8) is installed on the roof of the vehicle.

4. The receiver includes a Phantom C110 speed camera

5. The computing module (1) is built into the dashboard of the car and is a computer.

6. The source modules (4) operate in a continuous mode, in which they receive pseudo-random masks (2) and (3) generated by the computational module (1).

7. The switching frequency between masks is 25 kHz

8. Light (5) with a pseudo-random intensity distribution hits an oncoming vehicle (6).

9. The receiver module (8) makes video recording at a speed of 25 kHz.

10. The receiver module registers the light (5) generated in (4) and scattered by (6), as well as the light (9), which is illuminated by a similar invention (10), but installed on the car (6).

11. The computing module builds a 3D image by the method of phantom stereoscopic images. In the resulting image, light from an invention mounted on an oncoming vehicle is suppressed.

12. In this case, a sequence of 3D images is formed, i.e. video with a frame rate of 25 Hz.

Suppression of backlight in the formation of images of the road environment in front of the vehicle in the present invention is implemented by using the principle of phantom images. Phantom images are formed using the following modules that are part of the invention:

• Source module - a light source with a pseudo-random spatial intensity distribution;

• Receiver module - radiation detector;

• Computing module - a computing module designed to collect data from other modules, process data and generate an image.

The device includes a computing module (1) that generates pseudo-random masks (2) and (3) and sends them to two independent source modules (4). The source modules (4) generate light with a pseudo-random intensity distribution in accordance with the obtained masks (2) and (3). Switching between masks occurs with a frequency of up to 30 KHz. Two light beams (5) from the source modules (4) illuminate the space in front of the vehicle on which they are installed. Objects (6) located in the region of interest in space scatter light (5). The receiver module (8) registers part of the radiation scattered by objects (7). In addition, light (9) from external sources (10), for example, from the headlights of oncoming vehicles, enters the source module (8). The receiver module (8) sends signals containing information about the detected radiation to the computing module (1). From the received signals from (4) and (8), the computing module (1) forms two 2D images using the phantom image method, which is based on calculating the correlation function of the intensities of the illuminating and detected radiation. Each beam (5) at the exit from the source modules (4) has unique correlation properties, which are determined by masks (2) and (3). These properties allow separating two 2D images from the total signal from the receiver module. In addition, since light (5) has unique correlation properties, light (9) will be suppressed when forming images. The images are a stereopair, from which a 3D image of the terrain can be obtained by the method of stereoscopic images. Background suppression in stereopair leads to background suppression in 3D image.

A powerful "white" LED with a radiation power of 30 to 50 W and a uniform intensity distribution along the beam profile is used as an initial radiation source. The creation of light with a pseudo-random intensity distribution is performed using an array of micro-mirrors (MSM). The MSM has a cell structure, in which each cell is an independent mirror. In the present invention, two positions are available for each mirror: "on" and "off". When set to "on", the mirror reflects light into the output aperture of the source. In the "off" position, the mirror reflects the light into the blanking screen.

One of the possible implementations of the source module is shown in Fig. 2.

In this case, the LED (11), emitting light (12) in the visible range (450-650 nm), irradiates the MMZ (13). MMZ (13) has a control unit (14), which receives a mask (15) from the computing module (1). According to the mask obtained, block (14) exposes all micro-mirrors to (13). In this case, the switching frequency between masks does not exceed 30 kHz. The reflected modulated light (16) enters the output aperture of the source (17). Light (18) from a source that does not participate in the formation of the image is extinguished on the screen (19).

At the output of such a source, the intensity distribution will have pseudo-thermal statistics, which makes it possible to implement the principle of phantom images.

To form phantom images, you need to use one more module - the receiver module. One of the possible implementations of the receiver module is shown in Fig. 3. The input of the receiver module receives light (20), consisting of light generated in the source modules and scattered on objects in front of the vehicle, as well as light from external sources, such as headlights of oncoming cars.

The optical system (21) forms an image of the area in front of the vehicle and projects it onto the photosensitive matrix (22) of the video camera. The camcorder operates in fast shooting mode with a shooting frequency of up to 30 kHz. Each frame (23) is sent to the computational module.

The phantom image is extracted from the correlation function of the intensities of the light irradiating the object and detected by the receiver module. For this it is necessary to use one more module - the computational module.

The computing module is a multifunctional computer, whose tasks include:

- generation of pseudo-random masks for source modules;

- calculation of the spatial distribution of light intensity at the exit from the sources;

- processing of data received from the receiver module;

- calculation of the cross-correlation function between the signal received from the receiver and the calculated intensities of light at the output from the sources;

- formation of 2 2D images of the terrain by using the principle of phantom images with eliminated the influence of backlighting;

- formation of 3D images by using the principle of stereoscopic images.

The combination of these modules makes it possible to realize the purpose of the invention.

Claims (15)

1. A method for suppressing illumination in the formation of images of a road environment in front of a vehicle contains the stages at which: light with a pseudo-random intensity distribution is generated, while in each source the light from the LED falls on the corresponding matrix of micromirrors, for each matrix of micromirrors a set of pseudo-random masks, masks are sent to the control units of the micromirror matrices, in accordance with each mask of the micromirror matrix, the radiation of the photodiode is modulated so that the specified radiation is directed to the area in front of the vehicle, with the help of a high-speed video camera, the light from the illuminated area is recorded, 2D and 3D images of the terrain in front of the transport by the method of stereoscopic phantom images. 2. A device for suppressing illumination when forming images of a road environment in front of a vehicle, comprising a computing module and associated with at least two source modules and at least one receiver module, characterized in that the source modules are used in the form of main headlights of a distant and / or low beam for a vehicle and are made with the possibility of illumination both in the usual and in the infrared range, and the computing module is designed to collect data from the remaining modules, process data and form an image and is configured to generate pseudo-random masks for source modules, calculate the spatial distribution light intensity at the output from the sources, processing the data received from the receiver module, calculating the cross correlation function between the signal received from the receiver and the calculated light intensities at the output from the sources, forming two 2D images of the terrain by using using the principle of phantom images, forming a 3D image by using the principle of stereoscopic images. 3. The device according to claim 2, characterized in that the source modules contain a matrix of micromirrors and a light-emitting diode, and the receiver module contains a high-speed video camera, while a 3D image of the terrain in front of the vehicle is built using an algorithm that includes stages at which for each pixel of the video camera two cross correlation functions are calculated

and

where

is the correlation function between the signal in the pixel of the video camera with the radius vector r and the signal in the corresponding pixel in the matrix of micromirrors of the first source, and

a similar function, but for the matrix of micromirrors of the second source,

and

are 2D images of the area in front of the vehicle and represent a stereopair, and the 3D image is built on the basis of the resulting stereopair. 4. The device according to claim. 3, characterized in that the source modules contain a two-dimensional array of LEDs, the radiation of which is modulated in time and space. 5. The device according to claim 3, wherein the source modules comprise a liquid crystal spatial light modulator. 6. The device according to claim. 3, characterized in that the source modules contain a one-dimensional array of LEDs and a spatial beam scanner. 7. The device according to claim 3, wherein the source modules contain a fast beam scanner. 8. The device according to claim 3, characterized in that the high-speed video camera is made with a number of pixels equal to the number of elements in the micromirror matrix. 9. The device according to claim 3, characterized in that the receiver module contains a photodetector operating in a linear mode, and the image is built from the calculation of correlations between the total signal from this photodiode and the signal in each pixel of the micromirror matrices. 10. The device according to claim 9, characterized in that the power supply of the used photodetector is modulated synchronously with the light source. 11. A device according to claim 2, characterized in that only one radiation source is used for illumination, and the radiation is recorded by several receiver modules located at a distance from each other. 12. The device according to claim 1, wherein the masks are generated in advance. 13. The device according to claim 2, characterized in that one source module, one photodiode is used, and a 2D image of the area in front of the vehicle is built. 14. The device according to claim 2, characterized in that one light source is used as a stereopair and one camera and a displacement map is built on the basis of the displacement of the collection function of the source and the matrix receiver. 15. A device according to claim 3, characterized in that the number of pixels in the micromirror matrix and the video camera does not match.

Images