RU2490819C1 - Method of obtaining stereoscopic television images with automatic real-time measurement of object space - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: image is captured by synchronised video cameras; video signals of adjacent rows of all cameras are stored and compared; signals from instantaneous images of the object point in permissible ranges of linear parallaxes are found and temporal parallaxes between signals of parallax cameras with a signal of a reference camera in a single time reference system are measured; parallax signals with the video signal of the row of the reference full-resolution video camera are formed and synchronised; the obtained signal stream is transmitted to the receiving side and stored; a video signal of the second frame of a stereoscopic pair is restored by shifting elements of signals from the full-resolution camera by adjacent temporal parallaxes and the image is displayed on a stereoscopic monitor; and spatial coordinates of object points on the transmitting or receiving side are measured based on the functional relationship between range and the base length of stereoscopic capture, the focal length of objects and values of temporal parallaxes; the obtained information is transmitted to an analytical unit in which the series of stereoscopic frames is analysed and the trajectory functions of object points of interest and objects are calculated with calculation of first and second derivatives thereof.

EFFECT: obtaining and automatic measurement of parameters of stereoscopic television images.

1 dwg

Description

The present invention relates to television technology, in particular to stereo television and navigation systems, which can be used in the direct spectral visibility zone in automatic location and navigation systems on any mobile and stationary platforms, with real-time measurement of the surrounding space, in automatic pattern recognition systems , for solving special applied problems in 3D broadcast television systems.

All existing types of locations are based on backlighting by electromagnetic or optical pulses of the subject space and processing and analysis of the received reflected signal. Each of them has its own specific advantages, but none of them is intended for visualization of the subject space, which is necessary for remote control of autonomous vehicles for their confident orientation in the subject space and performing accurate dynamic control in the developing environment with reliable recognition of surrounding objects.

The use of 3D TV technologies for location, navigation and broadcast television allows:

- measure the surrounding space;

- immerse the operator of the remote vehicle in virtual 3D reality providing the full control effect of this vehicle:

- significantly increase the consumer properties of broadcast television.

However, the widespread use of these technologies is constrained by the use of complex video processing algorithms that require large computational resources, significantly increasing the cost of the necessary technical means.

The present invention relates to stereo television, and in particular to stereo systems with a parallax method for determining range.

An analogue was chosen as a method of automated measurement of the coordinates of the external environment to build its three-dimensional model in a stereo television system of technical vision [1]. The method is intended for use in a man-machine complex containing a technical vision system consisting of a television stereo unit paired with a personal computer, a display for image visualization, and a cursor movement control device. The measured point of the observed scene is indicated using the cursor on the display screen, and its three spatial coordinates are calculated automatically using a PC program and visualization on the display screen. The disadvantages of this method are: doubling the video stream for transmitting frames of a stereo pair, the ability to measure the coordinates of subject points of only static scenes, because operator participation is required to select the set of measured points of the object required to calculate its volumetric geometric model.

As a prototype adopted the closest solution to the method using the so-called 2D + Z format "Digital 3D TV" [2].

The technical implementation of 3D translation in this way is as follows:

1. From a stereoscopic camera, video is transmitted via two HD SDI ports to a video encoding server. For video capture, Elecard HD Access 2 boards are used, specially modified to ensure the synchronization of the reception of two channels of stereoscopic video.

2. Triaxes Vision software synchronizes two video channels. Then, for each frame of the video, a “depth map” is calculated and a 2D + Z stream is generated. To calculate the map, an algorithm is used to analyze the differences between the two video channels.

3. 2D + Z video is compressed by the Elecard codec and packaged into a standard MPEG transport stream (TS). The received TS is then broadcast over the local IP network. In principle, such a TS can already be encapsulated and broadcast via DVB-T.

4. Triaxes3D Player set-top boxes (STBs) are used as receivers (built on the basis of the commercially available Elecard with special software. STB receives by IP, decompresses the stream and displays it on Philips WOWvx displays.

Any ordinary (2D) image is matched with information about the distance of each pixel from the observer (Z-coordinate). This representation of the image is called the “2D + Z format”, and the Z coordinate plane is called the “depth map”. The map is presented in the form of a monochrome image in which gradations of gray indicate the distance of the object points from the observer. At the same time, an algorithm for analyzing the differences between the two video recording channels is used to calculate the map. The received video streams of the full resolution channel and the synchronized video stream of a monochrome image of a map of depth Z through two communication channels are transmitted to the receiving side.

To reconstruct a three-dimensional image, it is necessary to calculate a series of frames. The stereoscopic image is restored by interpolating the original image taking into account the depth map. The resulting series of frames simulates 3D video and then is demonstrated using, for example, a raster stereo display.

The main disadvantages of this method include the need to use complex algorithms that require analysis of personnel images and large computing resources to obtain and reproduce stereoscopic images; the impossibility of automatic measurement of subject space in real time; problematic application of the method in navigation systems and vehicle control systems including remote and automatic control when working in different environments.

The measurement of the local range of an object point in stereo shooting consists in finding and determining the distance between its corresponding (homologous, comparator) images in frames of a stereo pair. This is a well-known parallax method that has long been used in astronomy and artillery for measuring ranges to stars and to targets.

To determine the image of the entire subject space, it is necessary to calculate all or most of the image points in the frames of the stereo pair. In known methods of reconstructing an object from stereopair images, each measured point is determined with the participation of an operator, which requires a lot of time and can be used only for static images, for example, in cartography when processing stereo photographs made by aerial photography on a stereo comparator.

To automate this process, many algorithms have been proposed [3]. However, this task is very complex and, apparently, is still far from a solution. An analysis of a stereopair assumes the presence in the computer memory of very extensive knowledge about the world, without which decoding of a stereopair is generally unlikely [4].

The aim of the proposed method is to create a television stereo system based on existing television technology that provides automatic measurement of subject space in real time with a rapidly changing environment, with the minimum necessary and sufficient amount of information transmitted in the communication channel using simple processing algorithms that require minimal computational resources.

This goal is achieved in that the acquisition and automatic measurement of stereoscopic television images is obtained by shooting at least two synchronized video cameras: parallax - monochrome resolution and reference - full resolution, store and compare the video signals of the paired lines of all cameras, find signals from linear parallax ranges from points and measure the time parallaxes between the signals of parallax cameras with the signal of the reference camera in a single time system from accounts, form and synchronize the parallax signals with the video signal of the line of the reference full-resolution video camera, transmit it to the receiving side and store the received signal stream, restore the video signal of the second stereo pair by shifting the elements of the full-resolution video signals by conjugate time parallaxes and reproduce the image on the stereo monitor, measure the spatial coordinates of the subject points on the transmitting or receiving side according to the functional dependence of the range on the basis value tereophotography, focal length of lenses and time parallax values, transmit the obtained information to an analytical unit in which a series of stereo frames are analyzed and the functions of the trajectories of the object points and objects of interest are calculated with the calculation of their first and second derivatives necessary for recognizing the dynamic image of the surrounding object space and navigation in it with the goal of solving specific tasks.

In fig. indicated by:

1 - objects and points A, B, C belonging to them in the subject space;

2 - lens of the left video camera of full resolution;

3 - lens of the right video camera of monochrome resolution;

4, 5 - photocathodes or CCDs of the matrix, respectively, of the left and right cameras;

6 - video pulses from images of subject points A, B, C;

7 - parallax pulse;

8 - linear size of the active part of the video cathode or CCD of the video matrix;

9 - video signals of the left channel and the restored right video channel;

10 - stereo monitor;

b - base stereo;

F 'is the focal length of the lenses;

Ol; Op - the central points of the lenses of video cameras; A'l; A'p; V'l; V'p; S'l; С'п - images of subject points in the left and right video cameras, respectively;

to - time to start scanning the string;

tA'l; tA'p; tB'l; ... tC'p - time distances from the beginning of the line to video pulses from images of subject points A, B, C;

Δt; ΔtB; ΔtC - time parallaxes, respectively, for video signals from images of subject points A, B and C;

Δt is the parallax signal.

As you know, the principles of television transmission of images are based on the construction of optical-electronic scanning of images in the video camera of each point in the subject space and converting it into an electronic construction video signal from which image frames are formed. The whole process of converting an optical image into an electronic video signal and back to the image on the receiving side is carried out in a single time system by the line and frame scan unit of the transmitting video camera.

For clarity of the time parallaxes in the figure, the ray segments from the object points A, B, and C are parallel transferred from the point Op to the point Ol and are indicated by dashed lines. It is seen that the time parallaxes for images of subject points A, B, and C are equal to the difference in the corresponding time intervals between the signals from their images in the left and right cameras. So, for example, for the image of point A:

ΔtA = tA'p-tA'l;

where tA'p; tA'l - time intervals from the reference point to to the video pulses from the images of point A in the right and left video cameras, respectively, determined by the scan speed.

Here, any constant in time and common parameter for the whole system can be taken as the reference point for time intervals to. In particular, it can be a line clock. Similarly, time parallaxes are determined for images B and C and all other points of objects in the subject space.

To analyze the line-by-line video signals of video cameras, they are stored, the positions of the video pulses in the cameras are determined from the images of each subject point, and time parallaxes (intervals) between them are measured. Thus, a line-by-line parallax signal 7 is formed, which is synchronized with the signal of the full-resolution video camera 6. Then both signals, through the communication channel, are transmitted to the receiving side.

On the receiving side, the transmitted video signals of the full resolution camera and the parallax signal synchronized with it are stored line by line. A temporary shift of the elements of the full video signal 6 by the corresponding values of the parallax signal 7 form the complete video signal of the second frame of the stereo pair 9. The resulting two video signals are reproduced on the stereo monitor 10 by any known method.

For the technical implementation of the method on the transmitting side, it is necessary to add at least one monochrome video camera horizontally mounted at the distance of the stereo base to the full resolution video camera. Provide control of the scans of both cameras with one block of line and frame scans. Additionally, include in the electronic circuit a unit that provides RAM and analysis of the video signals of both cameras to determine the comparator points, measure and form a parallax signal, synchronize it with a full resolution video signal and transmit them to the communication channel.

The receiving side can be performed in any known manner. For example, the method of raster separation of images of stereo pair frames which ensures the creation of compatible tools with existing receiving television equipment [5].

Measurement and analysis of the subject space using known parallax values can be performed by existing algorithms and technical means based on the functional dependence of the distance to the subject point Z = f (F; b; ΔL), where ΔL is the linear parallax for the measured point, equal to the time parallax product Δt and sweep speed [3].

It can be seen from the optical diagram in the figure that the subject space is divided by the optical axes of the cameras (indicated by dashed lines) into left and right side vision zones and the direct vision zone limited by the size b, the base of stereo recording. Comfortable vision in a person is in the direct vision zone, and the size of the subject area that he is able to perceive in full is determined by the size of the eye yellow spot. Therefore, a comfortable viewing of three-dimensional images will be provided for scenes located in the direct vision zone and in the limited area where the viewer is located relative to the stereo display.

In vision devices, these zones are significantly expanded due to the large linear dimensions of the photocathodes or video matrices of the transmitting cameras compared to the yellow spot of the retina and the great capabilities of the analytical and computational performance of modern technical means. These features can significantly expand the comfort viewing area comparable to the area with the multi-angle method of transmitting stereoscopic images.

The reliability of recognition of video pulses from comparator images of object points in different cameras is based on the assumption that each object point is displayed by the original video pulse and that their position depends on what field of view it is in. So, for all subject points in the direct field of view, their images and corresponding video pulses are located on opposite sides of the optical axes of the lenses. For the subject points of the right and left side vision, they are located on one side of them and, respectively, to the left and right. It should be noted that human binocular vision has the same properties. These criteria determine the boundaries of the acceptable ranges for finding linear and corresponding time parallaxes and are the initial data for the algorithm of the analytical unit.

LITERATURE

1. RU 2065133 C1. A method of automated measurement of the coordinates of points of the external environment to build its three-dimensional model in a stereo television system of technical vision.

2. Digital 3D-TV (Variants of constructing a three-dimensional visualization system).

3. A.A. Vedenov. Mathematics of stereo images. M., 1991.

4. Encyclopedia of physics and technology. "Stereoscopic image."

5. RU 2192104 C2, 10.27.2002. "A method of obtaining stereoscopic television images."

Claims (1)

A method of obtaining and automatically measuring stereoscopic television images, including stereoscopic shooting of at least two synchronized video cameras, ranging, broadcasting video signals to the receiving part, restoring the video signals of the stereo pair frames and reproducing them on the stereo monitor, characterized in that the receiving and automatic measurement of stereoscopic television images is performed as at least two synchronized cameras: parallax - monochrome p resolution and reference - full resolution, store and compare the video signals of the conjugate lines of all cameras, find signals from instantaneous images of the object point in the acceptable ranges of linear parallaxes and measure the time parallaxes between the signals of parallax cameras with the signal of the reference camera in a single time frame, form and synchronize parallax signals with the video signal of the line of the reference video camera of full resolution, transmit to the receiving side and store the received signal stream, restoring t the video signal of the second frame of the stereo pair by shifting the elements of the full resolution signals to the paired time parallax and reproduce the image on the stereo monitor, measure the spatial coordinates of the object points on the transmitting or receiving side by the functional dependence of the distance on the magnitude of the basis of stereo shooting, the focal length of the lenses and the magnitude of the time parallaxes, transmit the received information into the analytical unit, in which a series of stereo frames is analyzed and the functions of the trajectories are calculated general object points and objects with the calculation of their first and second derivatives, necessary for recognition of the dynamic image of the surrounding object space and navigation in it in order to solve specific problems.