JPH08297744A - Moving picture area dividing device - Google Patents

Abstract

PURPOSE: To automatically and accurately divide a picture into a background and an object by providing a movement vector detection part and using the movement vector information of the object in the picture to extract the features. CONSTITUTION: A motion vector detection part 105 calculates a movement vector from the original picture and outputs it to a feature extraction part 102. A fourth binary picture generation part in the feature extraction part 102 detects positions, where the motion vector is changed from 0 to not 0 in the horizontal direction or is changed from not 0 to 0, based on the output from the motion vector detection part 105. A low pass filter is applied in the perpendicular direction to extract the edges, thus generating a binary picture. That is, the motion vector detection part 105 is provided to extract the features of motion vector information of the object in the picture, and the output of the motion vector detection part 105 is inputted to the feature extraction part 102, and the position of the outline of the object is determined by the binary picture generation part to perform accurate area division.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for automatically segmenting an image, which is used in a device for digital image processing, and more particularly to a morphological segmentation technique which is one of the algorithms for realizing this device. It relates to a device to which is applied.

[0002]

2. Description of the Related Art Conventionally, various techniques for automatically segmenting an image have been studied, but one of the techniques of this type, morphological segmentation technique, is the document "Morphlogical multiscale segmentation for image".
coding '' (Signal Processing, Vol. 38, pp. 359-386, 199
As described in 4), it is a hierarchical top-down algorithm, and unlike other image segmentation techniques, it uses features such as size, shape, contrast, and continuity of the object. In the first hierarchy, we do the most holistic view of the image,
It is assumed that the entire image is one area, and is divided into layers by going through the layers. As shown in FIG. 9, the division device using this algorithm is a preprocessing unit 901, a feature extraction unit 902, a boundary line determination unit 903, and a modeling unit 9.
Each of the four different regions 04 is defined as one layer, and the layers are layered.

The preprocessing unit 901 functions to simplify a complicated original image and remove information that is not needed in the hierarchy. A morphological filter based on a reduction filter and an enlargement filter is used for this processing. The morphological filter is an operation on a two-dimensional square region, and its size is (2 ^k +1) * (2 ^k +1) and k =
1, 2, 3, 4, 5, and 6. At the first level of hierarchy, k is large and the size of the filter is large. Reduction and enlargement filters are described in the document "Morphological Multiscale".
Image Segmentation '' (Proc. Of Visual Communicatio
ns and Image Processing '92, pp.620-631, Boston, 199
It is shown in 2) and given by the following equation.

[0004]

[Equation 1]

[0005]

[Equation 2]

Here, x is a vector representing the position of the target pixel, f (x) means the pixel value of the target pixel, and M _k
Indicates an operation area of size (2 ^k +1) * (2 ^k +1).
The reduction filter sets the minimum value in the operation area as the value of the target pixel. That is, it functions to erode bright areas. Also,
The enlargement filter sets the maximum value in the operation area as the value of the target pixel. That is, it functions to enlarge the bright area. A combination of these is called an opening filter or a closing filter and is defined by the following equation.

[0007]

(Equation 3)

[0008]

[Equation 4]

The opening filter has a function of erasing a bright area smaller than the operation area, and the closing filter has a function of erasing a small dark area. That is, it has a function of preferentially selecting a larger area than the operation area.
Since these filters do not preserve the edges of the remaining areas, it is necessary to repair the edges after filtering. This process is expressed by the following equation.

[0010]

(Equation 5)

[0011]

(Equation 6)

Where k is the number of iterations and PR _k
NR _k is a filter for repairing edges damaged by the opening filter, and NR _k is a filter for repairing edges damaged by the closing filter.
Theoretically, this filtering should be repeated until there are no changes, but in general, a few iterations will give sufficient results, so a few iterations will be necessary for the actual processing. Done in. The processing performed by the pretreatment device is shown in FIG.

The feature extraction unit 902 extracts the rough position of the boundary line of the region by using the features of the image created by the pre-processing unit 901 and performs the process of expressing it as a binary image. The image output from the preprocessing unit 901 is an image having a constant gradation value and a clear contour. FIG. 11 shows the internal structure of the feature extraction unit 902.

In the first binary image creating unit 1101, the inside of the region divided up to the previous layer is set to zero and the boundary of the region is set to non-zero based on the output result of the boundary line determining unit 903 of the previous layer. Create a binary image.

In the second binary image creating unit 1102, the image output from the pre-processing unit 901 has a region having a constant gradation value. Create a binary image using the threshold method. The gradient threshold method is a method of calculating a standard deviation with respect to an image obtained by the morphological gradient method and creating a binary image using the standard deviation as a threshold. The morphological gradient method is defined by the following equation.

[0016]

(Equation 7)

The result of binarizing g (x) is as shown in FIG. Among these, a region showing a low gradation value is a flat region. A binary image is created using the standard deviation in this image as a threshold. However, the morphological gradient method has a drawback that it cannot be detected when the contour change is smooth. In order to make up for this drawback, the third 2
A value image creation unit 1103 is configured. The third binary image creation unit 1103 performs feature extraction on the region where the contrast is not clear by the pre-processing unit 901 using the following equation for extracting the contrast.

[0018]

(Equation 8)

The size of the operation area in this equation is smaller than the size of the filter used in the preprocessing section 901, so that the preprocessing section 9 has already been removed.
A filter larger than the size of the filter used in 01 is used. Therefore, the operation is performed with the size of the filter used in the preprocessing unit 901 of the previous layer.

Finally, for the result h (x) of the contrast extraction, a binary image is created with a threshold value as in the gradient method. Finally, the binary image fusion unit 1104 fuses the results obtained from the above three binary images into one by performing an OR operation.

The boundary determining unit 903 has a feature extracting unit 90.
Based on the output from 2, the processing for accurately obtaining the position of the contour of the output area is performed. This process is based on the document "Morphologikal
Segmentation '' (Journal of Visual Communication and
Image Representation, Vol.1, pp.21-46, 1990). Pixel information at the same position in the original image is given to pixels to which non-zero is assigned in the output image from the feature extraction unit 902. That is, only the undetermined area around the extraction area produces an image having the information of the original image. A conceptual diagram is shown in FIG. In this image, a terrain surface is assumed such that a pixel with a high gradation value is located at a high position on the ground surface. By pouring water at a constant speed, only the boundaries of the feature-extracted regions will appear. Based on such a concept, we will clarify the boundaries of the feature-extracted regions. The Watershed algorithm is described in the document Color Image.
Segmentation '' (Proceedings 4th International Con
conference on Image Processing and its Applications, p
p.303-307, Maastricht, 1992) and is easily implemented by using a queue.

The modeling unit 904 performs a process of approximating the gradation value for each divided area with a polynomial, calculating the difference between the original image and the image, and outputting the difference to the next hierarchical level. The modeling unit 904 has a model polynomial, and is approximated by this. The degree of the polynomial is low in a region where the change is small, and a high-order polynomial is used in a place where the change is large.

[0023]

In the above-mentioned prior art, since the feature extraction method is insufficient, there are cases in which false contours occur in the image, and as shown in FIG. 14, the contours cannot be extracted and disqualification occurs. . In the case of disqualification, the boundary determination unit 9
According to 03, the extraction area is regarded as the same area as the background and cannot be extracted. The present invention has been made in view of the above, and an object thereof is to automatically and accurately divide a background and an object.

[0024]

In order to achieve the above object, a motion vector detecting unit 105 is provided for the purpose of using motion vector information of an object in an image for feature extraction. The moving image region dividing device is configured so that the output of 105 is input to the feature extraction unit 102. In the feature extraction unit, the motion vector detection unit 1
A fourth binary image creation unit 204 that receives the output from 05 is provided and input to the binary image fusion unit 205.

The fourth binary image forming section 204 comprises a horizontal change point detecting section 301, a vertical low pass filter (hereinafter abbreviated as LPF) section 302, and an edge extracting section 303.

The fourth binary image creating section 204 further includes
Horizontal change point detection unit 401 and vertical LPF unit 402
A vertical change point detection unit 403, a horizontal LPF unit 404, a logical sum calculation unit 405, and an edge extraction unit 406.
It is also possible to configure with.

The fourth binary image creating section 204 further includes
A horizontal direction change point detection unit 501, a vertical direction change point detection unit 502, an OR operation unit 503, an LPF unit 504,
It can also be configured with the edge extraction unit 505.

[0028]

The motion vector detecting section 105 calculates a motion vector from the original image. The fourth binary image creation unit 204 inside the feature extraction unit creates a binary image based on the output from the motion vector detection unit 105. The output from the fourth binary image creation unit 204 is the output of the first binary image creation unit 201 that creates a binary image from the results up to the previous layer,
Second second for creating a binary image from the output of the preprocessing unit 101
The output from the value image creation unit 202 and the third binary image creation unit 203 and the binary image fusion unit 205 form one binary image.

The horizontal change point detection unit 301 inside the binary image generation unit 204 detects the position where the motion vector changes from zero to non-zero and the position where the motion vector changes from non-zero to zero in the horizontal direction. The vertical LPF unit 302 applies a low-pass filter in the vertical direction, and the edge extraction unit 303 extracts an edge.

A binary image creating section 20 having another structure
4, the horizontal change point detection unit 401 detects the position where the motion vector changes from zero to non-zero in the horizontal direction and the position where the motion vector changes from non-zero to zero, and the vertical direction L
The PF unit 402 applies a low pass filter in the vertical direction, and the vertical direction change point detection unit 403 detects a position where the motion vector changes from zero to non-zero and a position where the motion vector changes from non-zero to zero in the vertical direction, The horizontal LPF unit 404 applies a low pass filter in the horizontal direction. OR operation unit 40
5, in the vertical direction LPF unit 402 and the horizontal direction L
The output of the PF unit 404 is logically ORed, and the edge extraction unit 406 extracts the edge.

A binary image creating section 20 having another structure
The operation inside 4 is that the horizontal direction change point detection unit 501 detects the position where the motion vector changes from zero to non-zero in the horizontal direction and the position where the motion vector changes from non-zero to zero, and the vertical direction change point detection unit 502 detects the position. The position where the motion vector changes from zero to non-zero in the vertical direction and the position where the motion vector changes from non-zero to zero are detected. The logical sum calculation unit 503 performs logical sum calculation of the outputs of the horizontal direction change point detection unit 501 and the vertical direction change point detection unit 502, the LPF unit 504 applies a two-dimensional low-pass filter, and the edge extraction unit 505 Extract edges.

[0032]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows the structure of one layer in a moving image area dividing device according to the present invention. The preprocessing unit 101, the boundary line determining unit 103, and the modeling unit 104 are the same as those in the conventional technique, and thus the description thereof is omitted. Here, what is different from the conventional technique is that there is a motion vector detection unit 105 and that the feature extraction unit 102 receives an input from the motion vector detection unit 105 and performs feature extraction using a motion vector.

The motion vector detection unit 105 calculates the motion vector of each pixel by the per-recursive motion estimation. As the motion vector of the pixel of interest, the average value of the motion vector of the pixel on the left side of the pixel of interest and the motion vector of the pixel on the upper side is given as an initial value, and the motion vector of the pixel of interest and the surrounding pixels of the pixel of interest in the previous frame It is replaced by the vector with the smallest difference. The structure of the feature extraction unit is shown in FIG. From the first binary image creation unit 201 of the feature extraction unit to the third
The description of the binary image creating unit 203 of No. 1 is omitted here, and thus the fourth binary image creating unit 204 will be described.

As shown in FIG. 1, the fourth binary image creating unit 204 includes a horizontal change point extracting unit 301 and a vertical direction LP.
The F unit 302 and the edge extraction unit 303 are included. In the horizontal direction change point extraction unit 301, the motion vector detection result is scanned in the horizontal direction, the magnitude of the motion vector MV of the pixel on the left side of the target pixel is non-zero (| MV | ≠ 0), and | MV is selected at the target pixel. The positions of the pixels where | = 0 and the pixel on the left side of the pixel of interest | MV | = 0 and the pixels of | MV | ≠ 0 on the pixel of interest are extracted, and non-zero pixels are extracted between those pixels. And the other pixels are assigned zero to create a binary image. FIG. 6 shows an example of the distribution of motion vector magnitudes. Further, FIG. 7 shows an example of a corresponding binary image created by the horizontal change point extraction unit.

The binary image created in this way is shown in FIG.
It may include disqualifications as shown in. The vertical LPF unit 303 performs one-dimensional low-pass filter processing in the vertical direction in order to remove such a region.

Finally, the morphological gradient method is applied to this image to output a binary image as shown in FIG.

<Other Embodiments> (1) The entire structure and the structure of the feature extraction unit are the same as those in the above embodiment, and the structure of the fourth binary image forming unit 204 is shown in FIG. The fourth binary image creation unit 204 includes a horizontal direction change point extraction unit 401, a vertical direction LPF unit 402, a vertical direction change point detection unit 403, a horizontal direction LPF unit 404, a logical sum operation unit 405, and an edge extraction unit 406. Composed. The horizontal direction change point extraction unit 401 and the vertical direction LPF unit 402 are the same as those in the above embodiment. In the vertical direction change point extraction unit 403, the magnitude of the motion vector MV of the pixel above the target pixel is non-zero (| MV | ≠ 0), and | MV | = 0 at the target pixel.
| MV | = 0 for the pixels that have become
The position of the pixel for which | MV | ≠ 0 is extracted in the pixel of interest,
A non-zero pixel is assigned to pixels between those pixels, and a binary image in which the others are zero is created.

The binary image thus created may include a vertical disqualification portion as in the case of the output from the horizontal change point extraction unit 402. In order to remove such a region, the horizontal LPF unit 404 performs one-dimensional low-pass filter processing in the horizontal direction.

The logical sum calculation unit 405 performs a logical sum calculation on these two types of images to create one binary image.

Finally, with respect to the output of the logical sum operation unit 405,
In the edge extraction unit 406, the operation of the morphological gradient method is performed and a binary image is output. As a result, it is possible to extract the features more accurately than when performing only the detection in the horizontal direction.

(2) The entire structure and the structure of the feature extraction unit are the same as those in the above embodiment, and the structure of the fourth binary image creation unit 204 is shown in FIG. The fourth binary image creation unit 204 includes a horizontal change point extraction unit 501 and a vertical change point detection unit 50.
2, an LPF unit 503, and an edge extraction unit 504. The horizontal direction change point extraction unit 501 and the vertical direction change point detection unit 502 are the same as those in the embodiment (1).

The logical sum calculation unit 503 performs a logical sum calculation on the two output images from the horizontal direction change point detection unit 501 and the vertical direction change point detection unit 502 to create one binary image. . In the LPF unit 504, the binary image output from the logical sum operation unit may include disqualified portions in both the horizontal and vertical directions, similar to the output from the horizontal direction change point extraction unit 502. In order to remove such a region, the LPF unit 504 performs a two-dimensional low pass filter process.

Finally, the edge extraction unit 505 performs the operation of the morphological gradient method and outputs a binary image.
As a result, as in the other embodiment (1), it is possible to extract the features more accurately than when only the horizontal detection is performed.

[0044]

In the moving image area dividing device according to the first aspect of the present invention, the motion vector detecting unit 105 is provided in order to use the motion vector information of the object in the image for feature extraction. Output the feature extraction unit 10
By inputting to 2, accurate area division can be performed.

In the moving picture area dividing device according to the second aspect, the feature extracting section is provided with a fourth binary image creating section 204 which receives the output from the motion vector detecting section 105, and is input to the binary image fusing section 205. By doing so, the first binary image creating unit, the second binary image creating unit, and the third binary image creating unit
When used in combination with the output from the binary image creating unit, the more accurate feature extraction and the accurate area division are possible.

In the moving image area dividing device according to the third aspect, the fourth binary image creating unit 204 is composed of the horizontal change point detecting unit 301, the vertical LPF unit 302, and the edge extracting unit 303. With this, it is possible to create the required binary image information in the boundary line determining unit 103 from the motion vector information.

In the moving image area dividing device according to the fourth aspect, the fourth binary image creating unit 204 includes a horizontal direction change point detection unit 401, a vertical direction LPF unit 402, and a vertical direction change point detection unit 403. The horizontal direction LPF unit 404, the logical sum operation unit 405, and the edge extraction unit 406 make it possible for the boundary line determination unit 103 to create the required binary image information from the motion vector information.

In the moving image area dividing device according to the fifth aspect, the fourth binary image forming unit 204 includes a horizontal change point detection unit 501, a vertical change point detection unit 502, and a logical sum operation unit 503. , LPF section 504 and edge extraction section 50
With the configuration of No. 5, it is possible to create the required binary image information in the boundary determining unit 103 from the motion vector information.

[Brief description of drawings]

FIG. 1 is a device configuration diagram showing one layer of an embodiment of the present invention.

FIG. 2 is a diagram showing an internal configuration of a feature extraction unit in the device of one embodiment of the present invention.

FIG. 3 is an example diagram of an internal configuration of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 4 is an example diagram of an internal configuration of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 5 is an example diagram of an internal configuration of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 6 is a diagram regarding processing of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 7 is a diagram regarding processing of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 8 is a diagram regarding processing of a fourth binary image creating unit in the apparatus according to the embodiment of the present invention.

FIG. 9 is a configuration diagram of a conventional device.

FIG. 10 is a pre-processing unit 202 in the apparatus of FIG. 9 of the conventional example.
4 is a flowchart of processing performed in FIG.

FIG. 11 is a device configuration diagram in a feature extraction unit in a device of a conventional example.

FIG. 12 is a diagram showing an example of an output result of a second binary image creating unit in the apparatus of FIG. 11 of the conventional example.

FIG. 13 is a diagram regarding processing of a boundary line determining unit in the apparatus of FIG. 9 of the conventional example.

14 is a diagram showing an example of an output from a feature extraction unit in the device of FIG. 9 of the conventional example.

[Description of Reference Signs] 101 pre-processing unit 102 feature extraction unit 103 boundary determination unit 104 modeling unit 105 motion vector detection unit 201 first binary image creation unit 202 second binary image creation unit 203 third binary Image creation unit 204 Fourth binary image creation unit 205 Binary image fusion unit 301 Horizontal direction change point detection unit 302 Vertical direction LPF unit 303 Edge extraction unit 401 Horizontal direction change point detection unit 402 Vertical direction LPF unit 403 Vertical direction change Point detection unit 404 Horizontal direction LPF unit 405 Logical sum calculation unit 406 Edge extraction unit 501 Horizontal direction change point detection unit 502 Vertical direction change point detection unit 503 Logical sum calculation unit 504 LPF unit 505 Edge extraction unit 901 Pre-processing unit 902 Feature extraction Part 903 boundary determination part 904 modeling part 1101 first binary image creation part 1102 second 2 Image creating unit 1103 third binary image generating unit 1104 binary image fusion unit

Claims (5)

[Claims] 1. An apparatus for hierarchically extracting features from a moving image and automatically performing area division, and a pre-processing unit for simplifying difference information from the previous layer, and detecting a motion vector of each pixel from an original image. A motion vector detection unit, a feature extraction unit that extracts a rough position of the boundary line of the region based on the results up to the previous layer and the outputs of the preprocessing unit and the motion vector detection unit, the original image and the features A boundary line determining unit that determines the boundary line based on the output of the extracting unit, and a modeling unit that polynomially approximates the pixel value of the original image based on the output of the original image and the boundary line determining unit A moving image area dividing device. 2. A first binary image creation unit that creates a binary image from the results up to the previous layer in the feature extraction unit,
A binary image is created based on outputs from the second binary image creating unit, a third binary image creating unit that creates a binary image from the output of the preprocessing unit, and the motion vector detecting unit. The moving image region dividing device according to claim 1, further comprising a fourth binary image creating unit and a binary image fusion unit that fuses the four binary images. 3. A horizontal direction change point detection unit for detecting a position where a motion vector changes from zero to non-zero in the horizontal direction and a position where the motion vector changes from non-zero to zero in the fourth binary image creation unit and a vertical direction. 3. The moving image area dividing device according to claim 2, further comprising a vertical LPF section that applies a low-pass filter in a direction, and an edge extracting section. 4. A horizontal change point detection unit for detecting a position where a motion vector changes from zero to non-zero in the horizontal direction and a position where the motion vector changes from non-zero to zero in the fourth binary image creation unit and a vertical direction. Vertical LPF section that applies a low-pass filter in the direction, and the vertical change point detection section that detects the position where the motion vector changes from zero to non-zero in the vertical direction and the position where the motion vector changes from zero to zero in the horizontal direction. A horizontal direction LPF section for applying a low-pass filter, a logical sum operation section for performing a logical sum operation of outputs of the vertical direction LPF section and the horizontal direction LPF section, and an edge extraction section. Item 2. The moving image area dividing device according to item 2. 5. A horizontal change point detection unit for detecting a position where a motion vector changes from zero to non-zero in the horizontal direction and a position where the motion vector changes from non-zero to zero in the fourth binary image creation unit, A vertical direction change point detection unit for detecting a position where a motion vector changes from zero to non-zero in the vertical direction and a position where the motion vector changes from non-zero to zero; and the horizontal direction change point detection unit and the vertical direction change point detection unit. 3. The moving image area dividing device according to claim 2, further comprising: a logical sum operation unit that performs an output logical sum operation, an LPF unit that applies a two-dimensional low-pass filter, and an edge extraction unit.