RU2542876C2 - Apparatus for selecting highly detailed objects on scene image - Google Patents

Abstract

FIELD: physics, video.

SUBSTANCE: invention relates to apparatus for selecting highly detailed objects. The apparatus comprises an input realisation storage unit, a control unit, a Canny edge detector unit, a unit for determining a common detailing coefficient, a sliding window selection unit, a unit for determining the detailing coefficient in the sliding window, a comparison unit, an output realisation storage unit and a clock pulse generator.

EFFECT: selecting highly detailed objects on images and video sequences on an arbitrary low-detailed background.

1 dwg

Description

The invention relates to the field of computer technology and can be used in digital television and photo systems, global positioning and surveillance systems.

The main problem to be solved is the selection of a highly detailed object against an arbitrary low-detailed background.

When automating information, radio-electronic and other systems working with images, as well as for more accurate analysis and decision making, an important aspect is the reception of a high-quality two-dimensional signal, however, if it is impossible to obtain, it becomes necessary to improve it, that is, to carry out preliminary processing.

Pre-processing includes: separating the object from the background, increasing the clarity of the contours, filtering with various parameters. However, the limited amount of computational resources used leads to an increase in the signal processing time or to the use of less computationally intensive methods, but in most cases not providing sufficient image filtering quality.

The solution of the problem of separating the object from the background in the image makes it possible to highlight areas with high detail of objects and areas - the background, which do not carry useful information. Therefore, the task of finding and highlighting an object or areas of interest in the scene image is a complex complex task, the solution of which will allow:

1. To process the selected areas by methods requiring high computational costs, but being accurate and high quality.

2. Localize processing areas and increase the performance of computing systems.

A known method and system for extracting data about the image of the foreground object based on data on color and depth [Patent No. 2426172, IPC G06K 9/34]. The invention relates to the field of image recognition and segmentation, in particular to a method for extracting a target from a background image using a mask. The technical result is the creation of an improved method for extracting data about the image of an object using data about the depth of the image. The specified technical result is achieved by creating a scalar image based on the separation of the object and the background. The difference is determined by the difference in illumination. In the area where the illumination difference is lower than a predetermined threshold value, a mask is initialized based on the color difference from the results obtained from the previous video frame. Where the scalar image of the difference is less than a predetermined threshold, the mask is filled with units where the corresponding pixel belongs to the object, and zeros otherwise. Then, the scalar image of the difference is clustered based on several clusters and a mask is created for each pixel position of the video frame. Using the centers of gravity of the clusters of the scalar difference and depth data for the current pixel position, compensate for changes in the background of the scene over time by updating the background image based on the use of the created mask and the difference image. The resulting separated object is stored for further processing.

The signs of the method and system is similar, matching the features of the proposed technical solution, the following: the selection of the target object from the background image.

The disadvantages of the known method and system are:

Mandatory use of cameras is required.

The study of the video stream requires large computational costs.

There is a method of isolating an object in an image based on the solution of Poisson matting for images [Patent USA No. 7636128]. The method is based on solving a system of Poisson equations with boundary conditions for an image segmented into three areas: foreground, background, unknown region separating the foreground and background. To separate objects, they solve the Poisson equation of the form:

$$\alpha *=\arg \underset{\alpha }{\min }{\displaystyle \iint {}_{z\in \Omega }\Vert \nabla {\alpha }_z-\frac{\mathrm{one}}{F_z-B_z}}\nabla I_z\Vert {}^{2}dz$$

with the Dirichlet boundary condition:

$$\alpha {\left|{}_{\partial \Omega }=\overline{\alpha }\right|}_{\partial \Omega },\overline{\alpha }|{}_{\partial \Omega }=\{\begin{array}{l}\mathrm{one},p\in {\Omega }_F\\ 0,p\in {\Omega }_B\end{array}.$$

The found solution of the system of Poisson equations allows separation of objects, i.e. alpha channel of the image, for refinement of which local filters are used, which allow you to manually correct the final result by solving local Poisson equations.

The signs of the method is an analogue that coincides with the features of the proposed technical solution, the following: the selection of the object from the background image.

The disadvantages of this method are: the use of local filters, manually correcting the final result by solving local Poisson equations, which does not allow to obtain an effective automated processing system.

A device for isolating the contours of objects in an image is known [Patent No. 2362210, IPC G06K 9/36, G06K 9/62, A61B 5/04]. The invention relates to the field of pattern recognition and can be used in vision systems for solving problems of image preprocessing.

The image processing algorithm includes the following operations: filtering, spatial differentiation, skeletonization and threshold processing.

The image is captured by the image sensor, converting it to a television signal. When a frame clock appears, a frame start signal is generated, according to which the DSP goes into the waiting state for confirmation of data arriving at the end of the horizontal clock, according to which the DSP sequentially reads the image line from the information output of the ADC to RAM. Thus, the entire frame is introduced. Image processing is carried out in parallel with the input of the next frame.

When filtering, the DSP transfers five adjacent lines of the image to the buffer memory of the filtering unit. Five adjacent lines of the image are written to the buffer memory of the filtering unit. Based on the five lines of the image located in the buffer memory of the filtering unit, the filtering unit generates the resulting line of the filtered image according to the formula:

$$f^{\prime }\left(z,y\right)=f\left(x,y\right)\cdot {\overline{T}}_D={\displaystyle \sum _{i=\mathrm{one}}^{N_T}{\displaystyle \sum _{j=\mathrm{one}}^{N_T}f\left(x+j-\frac{N_T}{2}-\mathrm{one},y+i-\frac{N_T}{2}-\mathrm{one}\right)\cdot {\overline{t}}_{ij},{\overline{T}}_D={\Vert {\overline{t}}_{ij}\Vert }_{i={\overline{\mathrm{one},N_T,}}_{j=\overline{\mathrm{one},N_T}}},}}$$

- маска свертки.where: N _T is the size of the mask, $${\overline{T}}_D$$

- convolution mask.

$${\overline{T}}_D=\left(\begin{array}{ccccc}\mathrm{one}& 5& 7& 5& \mathrm{one}\\ 5& \mathrm{twenty}& 33& \mathrm{twenty}& 5\\ 7& 33& 55& 33& 7\\ 5& \mathrm{twenty}& 33& \mathrm{twenty}& 5\\ \mathrm{one}& 5& 7& 5& \mathrm{one}\end{array}\right)$$

During spatial differentiation, the DSP transmits three adjacent lines of the image to the buffer memory of the spatial differentiation unit. Based on the three lines of the image located in the buffer memory of the spatial differentiation unit, which forms the resulting line of the processed image according to the formula

$$Gr=〈\sqrt{d_x^2+d_y^2},arctg\left(\frac{d_y}{d_x}\right)〉$$

where d _x (x, y), d _y (x, y) are the results of convolution of the image I ′ with horizontal Н _х and vertical Н _at masks, respectively

$$d_x\left(x,y\right)=f^{\prime }\left(x,y\right)\cdot H_x={\displaystyle \sum _{i=\mathrm{one}}^3{\displaystyle \sum _{j=\mathrm{one}}^3{f}^{\prime }}\left(x+j-2,y+k-2\right)\cdot h_{jk}^{(x)},i=\overline{\mathrm{one},2,}}$$

$$d_y\left(x,y\right)=f^{\prime }\left(x,y\right)\cdot H_y={\displaystyle \sum _{i=\mathrm{one}}^3{\displaystyle \sum _{j=\mathrm{one}}^3{f}^{\prime }}\left(x+j-2,y+k-2\right)\cdot h_{jk}^{(y)},i=\overline{\mathrm{one},2,}}$$

$$H_x=\Vert \begin{array}{ccc}{h}_{12}^{(x)}& h_{12}^{(x)}& h_{13}^{(x)}\\ h_{21}^{(x)}& h_{22}^{(x)}& h_{23}^{(x)}\\ h_{31}^{(x)}& h_{32}^{(x)}& h_{33}^{(x)}\end{array}\Vert =\Vert \begin{array}{ccc}\mathrm{one}& 0& -\mathrm{one}\\ 2& 0& -2\\ \mathrm{one}& 0& -\mathrm{one}\end{array}\Vert ,$$

$$H_y=\Vert \begin{array}{ccc}{h}_{12}^{(y)}& h_{12}^{(y)}& h_{13}^{(y)}\\ h_{21}^{(y)}& h_{22}^{(y)}& h_{23}^{(y)}\\ h_{31}^{(y)}& h_{32}^{(y)}& h_{33}^{(y)}\end{array}\Vert =\Vert \begin{array}{ccc}\mathrm{one}& 0& -\mathrm{one}\\ 2& 0& -2\\ \mathrm{one}& 0& -\mathrm{one}\end{array}\Vert ,$$

Thus, the calculations performed by the spatial differentiation unit are reduced to addition operations performed with the help of adders and logic elements, which makes it possible to implement this unit on the FPGA. Skeletonization and binarization operations are completely performed by the processor.

The signs of the device is an analogue, coinciding with the signs of the proposed technical solution, the following: the selection of the contours of the object in the image.

The disadvantages of the known device are: Circuit search is performed by analyzing data obtained from the analysis of the video stream, and requires large computational costs.

Closest to the invention is a method for automatically detecting faces in an electronic digital image [Patent EP 1777643 A4]. The invention relates to methods for pattern recognition by highlighting details or image characteristics and can be used to automatically detect faces on an electronic digital image obtained from a digital camera or camera in real, specially uncontrolled shooting conditions. Automatic detection of the face in the image is carried out in two stages: at the first stage, preliminary filtering procedures are performed, which consist of improving the image by means of averaging filters, filters to identify the edges of the image, the procedure for subsequent filtering of contours - supposedly belonging to the face elements: oval, eye, mouth, etc. .d. At the next stage, face elements are searched for, followed by identification by checking the statistical relationships between them, characterized in that they preliminarily narrow the face search area by determining the head location using preliminary filtering of the upper fragments of the head contour.

Filtering the upper fragments of the head contour is carried out by searching for the most likely candidates for other fragments of the head contour: the sides, eyes, shoulders, checking the statistical relationships between them and then searching in the areas of the most probable location using the head contour model templates. Determining the position of the eyes in the image in the vicinity of the found position of the head, preliminary filtering of face elements is carried out using filters of horizontal and vertical edges.

The signs of the prototype method, which coincide with the features of the claimed technical solution, are as follows: processing a two-dimensional digital signal, selecting objects with a complex outline (eyes, mouth, hair) on an arbitrary, little detailed background (facial skin) and their preprocessing.

The disadvantages of the known method and device that implements it are the requirement for the correct formation of an array of templates required for subsequent search, high computational costs for processing.

The proposed device for selecting the contours of objects on a textured background when processing digital images allows you to select a highly detailed object on an arbitrary low-detailed background. The device implements the following algorithm:

- The first step is a search on the source image / borders using a Canny detector (Gonzalez R., Woods R., Eddins S. Digital image processing in MATLAB. - M .: Technosphere, 2006. - 616 s). This method of detecting boundaries is based on sequentially performing the image smoothing operation in order to increase the signal-to-noise ratio by finding the image gradient to highlight areas with high spatial resolution, suppress all pixels that are not at the maximum (non-maximum suppression) and reduce the gradient array by using hysteresis in order to track the remaining pixels that were not suppressed. The filtering result is the determination of the spatial function of the image of the object using the optimal Canny operator - Gaussian

f (x) = - x ^* exp (-x ² / k ₂ s ² ),

where x is a variable; s is the standard deviation of the Gauss operator; * - “optimal” linear operator for convolution with the image; k ₂ = 2.

- At the second step, using the obtained boundaries, the total factor of detail is calculated, which is determined by the formula (1):

$$\frac{{\displaystyle \sum I\left(x,y\right)}}{i\cdot j}=P_{\text{common}},\text{(one)}$$

where I (x, y) is the pixel value with x and y coordinates; i is the number of rows; j is the number of columns; P is the coefficient of detail.

In the third step, similar to expression (1), the coefficient of detail in each sliding window is calculated:

$$\frac{{\displaystyle \sum I\left(x,y\right)}}{0.1\cdot i\cdot j}=P_{\text{window}},\text{(2)}$$

where 0.1 is the averaging coefficient associated with the automatic selection of the window size equal to 10% of the total image.

- In the fourth step, the detail coefficients P _common and P of the _{window are} compared, after which a decision is made on the detail in this window, and the window is shifted.

The device for selecting highly detailed objects in the scene image (Fig. 1) contains an input implementation 1 storage unit, the input of which is the information input of the device, the output is connected to the input of the control unit 2 and the first input of the Canny boundary detector 3, the output of which is connected to the first input of the selection unit a sliding window 5 and the first input of the unit for determining the overall coefficient of detail 4, the output of which is connected to the first input of the comparison unit 7, the output of which is connected to the first input of the storage unit alization 8, the output of which is the information output of the device; the output of the selection block of the sliding window 5 is connected to the first input of the block determining the coefficient of detail in the sliding window 6, the output of which is connected to the second input of the comparison unit 7; the output of the control unit 2 is connected to the second input of the boundary detection unit Canni 3, to the second input of the block for determining the overall detail coefficient 4, to the second input of the block for selecting the sliding window 5, to the second input of the block for determining the coefficient of detail in the sliding window 6 and to the second input of the storage unit output implementation 8; the synchronization of the device is provided by the clock generator 9.

The device for selecting highly detailed objects in the image of the scene works as follows: the input image, on which some object is presented against an arbitrary background, is input to the storage unit of the input implementation 1. Using the control unit 2, the image sizes, the averaging coefficient associated with the automatic selection of the window size, and the operator coefficient are the Gaussian of the Canny detector. From the output of the storage block of the input implementation 1, the image goes to the first input of the Canny border detector 3. The coefficient of the operator - Gaussian comes from the output of control unit 2 to the second input of the Canny border detector 3. In the Canny border detector 3, the borders are selected on the original image . The resulting image with the selected boundaries from the output of the Canny 3 boundary detector block is supplied to the first input of the block for determining the overall coefficient of detail 4, to the second input of which the output of the control unit 2 is connected, which supplies the sizes of the image. In the block for determining the overall coefficient of detail 4, the calculation of P _total occurs according to the formula (1). The resulting coefficient P _total is supplied from the output of the block for determining the overall coefficient of detail 4 to the input of the comparison unit 7. At the same time, an image with selected borders from the output of the block of the detector of boundaries Canny 3 is fed to the second input of the block the selection of the sliding window 5 is connected to the output of the control unit 2, which supplies the values of the image sizes i, j, as well as the averaging coefficient associated with the automatic selection of the window size. The sliding window selection unit 5 selects an image fragment with the dimensions of this sliding window. The resulting image fragment from the output of the selection block of the sliding window 5 is fed to the first input of the block for determining the coefficient of detail in the sliding window 6, the second input of which is connected to the control unit 2, which supplies the image sizes i, j, as well as the averaging coefficient associated with automatic selection window size. In the block detail coefficient determination block in the sliding window 6, the P _window is calculated according to expression (2). The obtained coefficient of detail P of the _window from the output of the block for determining the coefficient of detail in the sliding window 6 is fed to the second input of the comparison unit 7, where P _common and P _{windows are} compared and a decision is made on the granularity of this sliding window. The received data from the output of the comparison unit 7 is fed to the input of the storage unit of the output implementation 8, where it is recorded in the specified coordinates i, j, which are fed to the second input of the storage unit of the output implementation 8 from the output of the control unit 2. The output of the storage unit of the output implementation 8 is the information output of the device . The synchronization of the device is provided by the clock generator 9.

EFFECT: highlighting on images and video sequences of highly detailed objects against an arbitrary low-detailed background.

Claims (1)

A device for extracting highly detailed objects in a scene image containing an input implementation storage unit, the input of which is the information input of the device and the output implementation storage unit, the output of which is the information output of the device, characterized in that the output of the input implementation storage unit is connected to the input of the control unit and the first input the Canny boundary detector block, the output of which is connected to the first input of the sliding window selection block and the first input of the common coefficient determination block cient detail, the output of which is connected to the first input of the comparator, whose output is connected to a first input of output realization storage unit; the output of the sliding window selection unit is connected to the first input of the detail coefficient determination unit in the sliding window, the output of which is connected to the second input of the comparison unit; the output of the control unit is connected to the second input of the Canny boundary detector, to the second input of the block for determining the overall detail coefficient, to the second input of the block for selecting the sliding window, to the second input of the block for determining the coefficient of detail in the sliding window and to the second input of the storage block of the output implementation; synchronization of the device is provided by the clock generator.