RU2313189C2 - Method for spatial and temporal processing of images on basis of matrices of photo-sensitive devices with charge-coupled devices and device for realization of said method - Google Patents

Abstract

FIELD: television, possible use for engineering applied systems.

SUBSTANCE: in accordance to the invention, spatial and temporal processing of images on basis of matrices of photo-sensitive charge-coupled devices is realized in form of folding of image projected onto matrix of photo-sensitive charge-coupled devices with required impulse characteristic, which defines spatial and temporal filtration being realized. Time of accumulation of charges in each point of space is changed proportionally to value of counts of impulse characteristic of spatial and temporal filter being realized.

EFFECT: realization of non-distorting spatial and temporal processing of image, ensuring selection of immoveable and moving objects on non-stationary background.

2 cl, 3 dwg

Description

The invention relates to television and can be used to create application systems.

A known method of highlighting moving objects in an image (Press F. P. Photosensitive devices with charge coupling. - M .: Radio and communications, 1991. - 264 p.), Based on the use of photosensitive devices with charge communication (FPSS) in the time delay mode with accumulation (WZN), in which discrete accumulation of charges photogenerated by the projected image of charges in potential wells of the FPSS line is carried out, and electrical transfer is carried out between individual accumulations with a control clock frequency Pulsing impulses along the FPSS line. In the case of movement of charges synchronously with the movement of the object (the speed and direction of movement of the charges in the PPS must coincide with the same parameters of the movement of the object), the signal / background ratio increases due to coherent accumulation.

Closest to the claimed method is a method of spatio-temporal image processing based on arrays of photosensitive devices with charge coupling and a device for its implementation, taken as a prototype (Patent 2216117 of the Russian Federation, MKI ⁵ H04N 5/14. - No. 20011113366/09; application. 14.05 .2001; publ. 10.11.2003, Bull. No. 31), based on the fact that spatio-temporal processing is performed by discrete accumulation of charges photogenerated under the influence of the projected image in potential FPSS wells, and the image moves in space In particular, the FPSS matrices, and the accumulation time at each point in space are changed proportionally to the value of the samples of space; they are changed proportionally to the values of the impulse characteristics of the implemented space-time filter, depending on the type of filtering and observation conditions, for example, the illumination of the scene. A method for realizing spatio-temporal image processing based on FPGA matrices is that spatio-temporal processing is carried out in the form of a convolution of the image projected onto the FPGA with the required impulse response that defines the spatio-temporal filtering being implemented, as follows: in the mode of discrete accumulation, several cycles are performed accumulation over time corresponding to the value of the impulse response samples, followed by a spatial shift after each Ikla then organized storage mode accumulated charge before the next stage of the discrete accumulation. The number of cycles of accumulation - spatial shift in the mode of discrete accumulation corresponds to the dimension of the aperture of the impulse response in space, and the number of stages of discrete accumulation - storage - to the dimension of the aperture in time. The spatial shift between adjacent accumulation points corresponds to the sampling interval of the impulse response in space, the interval between adjacent stages of discrete accumulation corresponds to the time sampling interval. The storage of accumulated charge packets until the next stage of discrete accumulation begins is organized by shielding the photosensitive surface of the FPGA matrix for storage time using a shutter based on the Kerr and Pokels electro-optical effects, which allow switching the luminous flux with a frequency of up to 1013 Hz / I. Vereshchagin / et al. Introduction to optoelectronics: A manual for technical colleges / I.K. Vereshchagin, L.A. Kosyachenko, S.M. Konin. - M.: Higher School, 1991. - 191 p. /. Mutual movement of the FPSS matrix and the optical image is realized by applying the electro-optical Kerr effect: continuous beam deflection with high accuracy of movement can be obtained using a prism of electro-optical material with metal electrodes deposited on its end faces, to which the control voltage is applied / I. Vereshchagin . et al. Introduction to optoelectronics: A manual for technical colleges / I.K. Vereshchagin, L.A. Kosyachenko, S.M. Konin. - M.: Higher School, 1991. - 191 p. /.

The determination of the impulse response samples is performed on the basis of well-developed methods for the synthesis of discrete filters.

The structural diagram of a device for implementing the known method of spatio-temporal image processing is shown in figure 1. The device consists of an optical system (OS) 1, a control device (UU) 2, a subtracting device (VU) 3, blocks FPZS-1 4 and FPZS-2 5, as well as additionally introduced obturator 6, a device for mutual displacement of the OS and FPZS blocks ( UVS) 7. Optical communication is shown by a dashed line. A scene image formed by the OS is projected onto the shutter, and a control signal from the output of the control unit is input. From the shutter, the image signal is projected onto the UVS, to the input of which the control signal from the output of the SU arrives. From the UVS output, the image signal is projected onto the accumulation sections FPZS-1 and FPZS-2. Phase control pulses FN _{1, 2, 3} , FP _{1, 2, 3} , FR _{1, 2, 3} from the output of the control unit arrive at the inputs of the accumulation sections, memory, and output registers FPSS-1 and FPSS-2. From the output registers ФПЗС-1 and ФПЗС-2, the image signals are fed to the inputs of the control unit forming the output signal.

The implementation of spatio-temporal processing based on FPSS blocks, a shutter, an air-blast unit under the control of a control unit is carried out as follows. The FPSS performs discrete accumulation of charges generated by the projected OS image of the scene and stores the accumulated charge packets between the individual steps of the discrete storage when screening the FPSS accumulation section from the light flux using an obturator based on an electro-optical modulator. Block ФПЗС-1 implements the convolution of the image with positive coefficients of the impulse response of the filter, ФПЗС-2 - with negative ones. The need to use two separate FPSS matrices is due to the fact that the values of the impulse response characteristics of the filter can be both positive and negative, and it is impossible to perform convolution with samples of different signs on the same matrix due to the non-negativity of the light characteristics and physical limitations of the FPS (charges can only be accumulated one character). The subtracting device combines the signals from the outputs of both FPSS matrices and generates the output signal of the processing device. The UVS in accordance with the control signals of the SU carries out mutual spatial displacement of the optical stream and the FPS. The synchronization of the operation of the processing device during image filtering is performed by the control unit by changing the order of arrival and the duration of the phase pulses of the control of the accumulation section, the memory section, the output register of the FPSS blocks, the control signals for the shutter, and the control signals for the UVS.

The disadvantage of this method is the introduction of distortion in the implementation of spatio-temporal processing.

According to the description of the invention, "... in the discrete accumulation mode, several accumulation cycles are performed during the time corresponding to the value of the impulse response samples ...". Thus, the time interval between the centers of the accumulation cycles will be unstable due to the different values of the impulse response samples. Therefore, "... the determination of the magnitude of the impulse response samples, based on well-developed methods for the synthesis of discrete filters" will lead to distortions in the implementation of spatio-temporal processing (ADT) due to the fact that these methods imply the constancy of time and space intervals between points signal samples / Goldenberg L.M. et al. Digital signal processing: Handbook / L.M. Goldenberg, B.D. Matyushkin, M.N. Polyak. - M.: Radio and Communications, 1985. - 312 p. /. The reference point is the center of the accumulation cycle, which includes discrete accumulation and a fast spatial shift.

To eliminate variability of the time interval T between the _{frame of reference} signal samples offered between a discrete accumulation of duration T _i and shift storage space to introduce _cartilage step duration T _i (see FIG. 2). Then the time interval between the samples of the signal will be determined by the time interval between the centers of the stages of accumulation.

The implementation of spatio-temporal processing while ensuring the constancy of the time interval between the centers of the accumulation stages will eliminate air defense distortions and also allow adaptively changing its characteristics due to the difference in the mutual spatial shift of the projected image and the FPS during the cycle time for objects with different speeds.

Thus, if we take that the time interval between the centers of the stages of accumulation is constant and equal to ΔT, then the duration of each ith accumulation cycle T _ci should be equal to

where T _{i + 1} is the accumulation time in the next (i + 1) cycle,

T _i - accumulation time in the current i-th cycle.

Если взять, что время накопления в (i+1)- и i-м циклах равно максимальному значению Т_макс, то отсюда следует, что значение ΔТ должно быть ≥ Т_макс.Therefore, each cycle should consist of an accumulation stage of duration T _i and a storage stage of duration

If we take that the accumulation time in the (i + 1) - and i-th cycles is equal to the maximum value of T _max , then it follows that the value of ΔT should be ≥ T _max .

A device that implements the known method also has the disadvantage that it is impossible to ensure operation with negative coefficients. To implement positive and negative coefficients, it is necessary to transfer one of the FPSS to storage mode when accumulated in another FPSS. The implementation of this condition using a common shutter is impossible. Therefore, a beam splitting system and separate obturators with a shift system for each FPSS are necessary.

The technical result of the invention is the implementation of non-distorting spatio-temporal image processing, providing the selection of stationary and moving objects on an unsteady background.

The essence of the invention lies in the fact that to eliminate distortions of spatio-temporal image processing in a known manner based on FPGA matrices, which consists in using discrete accumulation of charges photogenerated in potential wells of the FPGA matrix under the influence of the projected image, in combination with the mutual spatial displacement of the image and the FPGA matrix moreover, the parameters of movement are determined by the frequency characteristics of the implemented space-time filter, and the accumulation time for the rows at each point in space are changed proportionally to the value of the impulse response samples of the implemented space-time filter; it is necessary to enter the storage stage between accumulations and spatial displacement for the time necessary to ensure the constancy of the time interval between the centers of accumulations.

A method for realizing spatio-temporal image processing based on FPGA matrices is that spatio-temporal processing is carried out in the form of convolution of an image projected on the FPGA with the required impulse response that defines the implemented spatio-temporal filtering, as follows: a projected image that is moved continuously in space, agreed with the object being followed, it is processed in several stages, consisting of cycles, including discrete accumulation, storage, space shift and pause until the next stage of discrete accumulation begins. The number of cycles of discrete accumulation - storage - spatial shift corresponds to the dimension of the aperture of the impulse response in space, and the number of stages corresponds to the dimension of the aperture in time. Discrete accumulation during the cycle occurs during the time corresponding to the value of the impulse response samples. The storage stage during the cycle ensures the constancy of the time interval between the centers of the stages of the past and subsequent discrete accumulation. The spatial shift between adjacent cycles corresponds to the sampling interval of the impulse response in space, the time interval between adjacent stages of discrete accumulation corresponds to the time sampling interval. Storage of accumulated charge packets during storage stages, as well as pauses between stages, is organized by shielding the photosensitive surface of the FPGA matrix using a shutter based on the Kerr and Pokkels electro-optical effects, which allow switching the luminous flux with a frequency of up to 1013 Hz / I. Vereshchagin / et al. Introduction to optoelectronics: A manual for technical colleges / I.K. Vereshchagin, L.A. Kosyachenko, S.M. Konin. - M.: Higher School, 1991. - 191 p. /. Mutual movement of the FPSS matrix and the optical image is realized by applying the electro-optical Kerr effect: continuous beam deflection with high accuracy of movement can be obtained using a prism of electro-optical material with metal electrodes deposited on its end faces, to which the control voltage is applied / I. Vereshchagin . et al. Introduction to optoelectronics: A manual for technical colleges / I.K. Vereshchagin, L.A. Kosyachenko, S.M. Konin. - M.: Higher School, 1991. - 191 p. /.

As a result of the elimination of air defense distortions, the proposed method for the implementation of spatio-temporal processing will have wider functionality, since it becomes possible to carry out processing that takes into account the characteristics of the parameters of a moving object and interferes with the observation of the background image, as well as adaptive spatial processing of objects moving at different speeds. It is possible, in addition to highlighting moving objects like the WZN mode, to implement other types of filtering: low-frequency spatio-temporal and notch filtering of spatio-temporal frequencies, similar to the Wiener one and taking into account the characteristics of the background that interferes with observation. The determination of the impulse response samples is carried out on the basis of well-developed methods for the synthesis of discrete filters.

Figure 3 shows a structural diagram of a device for implementing the proposed method of spatio-temporal image processing. The device consists of an optical system (OS) 1, a control device (UU) 2, a subtracting device (VU) 3, blocks FPZS-1 4 and FPZS-2 5, obturator 6, a device for mutual displacement of the OS and blocks FPZS-1 (UVS- 1) 7, as well as additionally introduced a beam splitting system 8, a shutter 9, a device for mutual displacement of the OS and the blocks ФПЗС-2 (УВС-2) 10. Optical communication is shown by a dashed line. On the obturators 6, 9 through the beam-splitting system, the scene image formed by the OS is projected, and control signals from the output of the control unit are sent to their electrical inputs. From the obturator 6, the image signal is projected to the UVS-1, and from the obturator 9 to the UVS-2, the electrical inputs of which receive control signals from the output of the UU. From the UVS-1 output, the image signal is projected onto the FPZS-1 accumulation section, and from the UVS-2 output - onto the FPZS-2 accumulation section. Phase control pulses FN _{1, 2, 3} , FP _{1, 2, 3} , FR _{1, 2, 3} from the output of the control unit arrive at the inputs of the accumulation sections, memory, and output registers FPSS-1 and FPSS-2. From the output registers ФПЗС-1 and ФПЗС-2, the image signals are fed to the inputs of the control unit forming the output signal.

The implementation of spatio-temporal processing based on FPSS blocks, a beam splitting system, shutters, OS mutual bias devices and FPSS blocks under control of a control unit is carried out as follows. Block ФПЗС-1 together with shutter 6 and УВС-1 implements image convolution with positive coefficients of impulse response of the filter, ФПЗС-2 together with shutter 9 and УВС-2 - with negative ones. Accordingly, when convolving an image with positive coefficients of the impulse response of the filter, the FPZS-2 unit stores the accumulated charge packets when shielding the FPZS-2 accumulation section from the light flux using an obturator based on an electro-optical modulator, and when implementing convolution with negative coefficients, FPZS-1. The need to use two separate FPSS matrices is due to the fact that the values of the impulse response characteristics of the filter can be both positive and negative, and it is impossible to perform convolution with samples of different signs on the same matrix due to the non-negativity of the light characteristics and physical limitations of the FPS (charges can only be accumulated one character). The FPSS performs discrete accumulation of charges photogenerated by the projected OS image of the scene, and stores the accumulated charge packets during pauses between accumulations. The subtracting device combines the signals from the outputs of both FPSS matrices and generates the output signal of the processing device. The UVS in accordance with the control signals of the SU carries out mutual spatial displacement of the optical flux and FPS: continuous coordinated with the accompanied object and spasmodic during each cycle. The processing device synchronizes the operation of the control unit by changing the order of arrival and the duration of the phase pulses of the control of the accumulation section, the memory section, the output register of the FPSS blocks, the control signals of the shutters, the control signals of the UVS-1 and UVS-2.

The application of the proposed method will allow to realize a wide range of tasks of spatio-temporal image processing simultaneously with their formation in real time on the basis of a device having the minimum mass-dimensional and energy indicators characteristic of the FPS.

Claims (2)

1. A method of spatio-temporal image processing based on arrays of charge-sensitive photosensitive devices, carried out in the form of a convolution of a projection onto a matrix of photosensitive devices with charge-coupled images with the desired impulse response, which determines the real-time space-time filtering, which consists in the use of discrete accumulation of charges, photo-generated in potential pits of a matrix of photosensitive devices under the influence of a projected image, in combination and with mutual spatial displacement of the image and the matrix of charge-sensitive photosensitive devices, the displacement parameters being determined by the frequency characteristics of the implemented space-time filter, and the accumulation time of charges at each point in space is proportional to the value of the impulse response samples of the implemented space-time filter, characterized in that in the accumulation cycle between the stage of accumulation of charges of duration T _i and spatial displacement enter the stage of storing charges for a time T _{xp i} is equal to

, where T _{i + 1} is the accumulation time in the next (i + 1) accumulation cycle; T _i - accumulation time in the current i-th accumulation cycle; ΔТ is the time interval between the centers of the stages of accumulation, taken equal in different cycles of accumulation, ensuring the constancy of the time interval T _count between the samples of the signal. 2. A spatio-temporal image processing device comprising an optical system, a shutter, a subtractor, two blocks of charge-sensitive photosensitive devices (FPZS-1 and FPZS-2) and a mutual displacement device of the optical system and blocks of charge-sensitive photosensitive devices, the optical the system is optically connected to the obturator, the obturator is optically connected to the device of mutual displacement of the optical system and the blocks of photosensitive devices with charge coupling, the device is biased I of the optical system and blocks of charge-sensitive photosensitive devices are optically connected to the accumulation sections of the FPZS-1 and FPZS-2 blocks, and the subtracting device combines the signals from the outputs of both FPZS matrices and generates the output signal of the spatio-temporal image processing device, characterized in that a beam splitting system, separate obturator and a device for mutual displacement of the optical system and blocks of photosensitive devices with charge coupling for the FPZS-2 block are additionally introduced moreover, the optical system is optically connected to the beam splitting system, the beam splitting system is optically connected to the shutters, each obturator is optically connected to its mutual displacement device of the optical system and the charge-coupled photosensitive devices, each device is mutually displaced to the optical system and the charge-coupled photosensitive devices, optically connected to the storage section of its FPZS, and a control device configured to synchronize the operation of the properties by changing the order of arrival and duration of phase control pulses of the accumulation section, memory section, output register of blocks ФПЗС-1 and ФПЗС-2, control signals of shutters, control signals of devices of mutual displacement of the optical system and blocks of photosensitive devices with charge coupling, at the inputs sections of accumulation, memory and output registers of blocks ФПЗС-1 and ФПЗС-2 receive phase control pulses from the first output of the control device, the input of the first shutter is connected to the second output control device, the input of the mutual displacement device of the optical system and the blocks of photosensitive devices with charge coupling of the FPZS-1 unit is connected to the third output of the control device, the input of the second shutter is connected to the fourth output of the control device, and the input of the device of the mutual displacement of the optical system and blocks of photosensitive devices with a charge the communication unit FPS-2 is connected with the fifth output of the control device.

Images