JP2507182B2 - Video region segmentation device - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to a method for realizing a highly efficient compression of moving image data in a TV, an HTV, a videophone, etc., and a component object for video composition in the creation of a TV program etc. The present invention relates to a moving image region dividing device applicable to a clipping method, a retrieval key image extracting method for constructing a moving image database, and the like.

2. Description of the Related Art Conventionally, in order to perform area division in a moving image, in order to associate the (divided) areas obtained for each frame between frames, a plurality of temporally consecutive frames are used. A method is known in which the similarity is calculated based on the size of the area, the distribution of shading, the center of gravity, the position of the feature point in the area, and the like, and the areas are associated with each other between consecutive frames using the similarity as a criterion. There is. For example, "Automatic Analysis o
f Moving images ”[Automatic Analysis of Moving Images] (Masahiko Yachida et al .: EYE TRANSACTIONS ON PATTERN ANALINS AND MACHINE INTELLIGENCE 3rd
Volume January 1, 1981; IEEE TRANSACTIONS ON PATTERN ANAL
YSIS AND MACHINE INTELLIGENCE, VOL.PAMI−3, NO.1, JA
NUARY, 1981) describes a method for searching and tracking a moving object in a stationary background using this method. The feature of this method is that each area in a frame is described by a parameter expressing the characteristics of that area, and the correspondence of the areas between the frames is obtained based on the distance in the parameter space. The conceptual configuration of the conventional device will be described below with reference to FIG. 1101 is a moving image memory that stores moving image data input according to the flow of time, 1102 is a frame region dividing device that divides the regions of the frames in the moving image memory, and 1103 is a frame region dividing result that stores the result. A storage device, 1104 is an expression device that expresses each area divided for each frame with parameters, and 1105 is an inter-frame search that searches for the correspondence between areas based on the distance in the parameter space expressed by the expression device 1104. It is an area correspondence search device.

However, if the result of the area division for each frame is not perfect just by finding out the rough correspondence of the areas between the frames with the above configuration, that is, there is a defect in the area in a certain frame. On the other hand, if it includes other areas, on the other hand, it cannot be corrected, and there is a problem in that the accuracy of the overall area division is limited.

In view of the above problems, the present invention provides a moving image area dividing device capable of correcting the area division for each frame and significantly improving the accuracy of the area division as a whole.

Means for Solving the Problems In general, in moving image data, two consecutive
The position and shape of the region of the object existing inside the frame do not change significantly between the frames. Therefore, in the present invention,
In order to solve the above problem, a calculation means for finding correspondence between boundaries of regions divided for each frame between consecutive frames, and loss of a region or erosion to another region based on the associated region boundaries The calculation means for correcting the above is provided.

The present invention has the above-described configuration, and in the area division of a moving image, by searching the area division result for each frame for a correspondence of the area boundary between frames, the area division result for each frame can be applied to the area boundary between frames. It is possible to correct the area division for each frame by searching the correspondence of, and it is possible to significantly improve the accuracy of the area division as a whole. Therefore, the present invention can be applied to, for example, highly efficient moving image compression in a TV telephone or a video conference, and construction of a moving image component database for image synthesis in a broadcasting system.

Embodiment An embodiment of the present invention will be described below with reference to FIG. FIG. 1 is a block conceptual diagram of a moving picture area dividing device in an embodiment of the present invention. In FIG. 1, the moving image data input according to the flow of time is stored in the storage means 10
Memorized in 1. Reference numeral 102 denotes an operation unit that divides one frame of the moving image data stored in the storage unit 101 into regions, and 103 denotes, for each frame, a result of operating the operation unit 102 on each frame of the moving image data 101. It is a storage means for storing. The result of the area division for each frame stored in the storage unit 103 is generally incomplete, but since the position and shape of the corresponding area do not change much between temporally consecutive frames, the divided area is divided. The boundaries of the frames are roughly associated with each other with respect to the area division result stored in the storage unit 103. The result of the association is stored in the storage means 105, and from this and the result of the first area division stored in the storage means 102, the calculation means 106 corrects the area division for each frame, and the result is stored in the storage means 107. Remember. Reference numeral 108 denotes an operation unit that labels all pixels in the corresponding regions of all frames from the correction region division result for each frame stored in the storage unit 107.

A more specific embodiment of the present invention will be described below with reference to FIG. 201 inputs moving image data along the flow of time. Storage device consisting of multiple frame memories, 20
2 is a dedicated processor that divides the image of each frame memory of the storage device 201 into areas, 203 is a memory that saves the results of area division for each frame, and 204 is a dedicated search for the correspondence of area boundaries between frames from the division results A processor, 205 is a memory for storing the correspondence result, and 206 is a processor for correcting the area division from the correspondence result, and corrects (changes) the contents of the memory 203. A processor 207 assigns a common label to pixels belonging to the same area based on the area boundary correspondence information of the memory 205 and the area division result for each frame of the memory 203.

As a method of dividing an area for each frame, there is a method of performing edge detection as a method of using only the data of the frame, but details are Rosen field; Digital picture
Processing Second Edition Volume 2 pp. 84-112 [ROSENFELD: Digital Picture Processing seco
nd edition Vol.2 pp84-112 (1982)], etc. In addition, the method of calculating the velocity boundary in a frame using the data of the frame of the neighborhood time and performing the region segmentation based on this is the Ando: Differential product-sum operator (automatic measurement Journal of Control Society Vol.25, No4, 496/503 (198
9).

Hereinafter, a method of searching for correspondences between area boundaries between frames and correcting the result data of the area division initially stored in the memory 203 will be described in detail with reference to FIGS. 3 to 10. It is assumed that the frame memory of the storage device 201 stores an image as shown in FIG. Reference numeral 301 in FIG. 3 is a memory for storing the result of area division of the memory 203 in FIG. 2, and stores data as shown in FIG. Fifth
The regions shown by the same pattern in the figure correspond to one divided region. 302 to 303 are dedicated processors 204 in FIG.
It is a dedicated processor that searches the correspondence of the area boundary between the frames. The data of 301 is a result of the area division for each frame, that is, at a certain time, and each area is a projection on the two-dimensional plane of the object existing in the target world of the image at that time. Now, as shown in FIG. 6, a three-dimensional coordinate system is set by taking the XY coordinate axes on one frame and the Z axis in the direction orthogonal to the frame. Here, when a series of frames is arranged in the Z-axis direction in time order and the portion corresponding to the area A in the frame at time t is traced, this forms one solid in the three-dimensional space. The hatched portion in FIG. 7 shows a cross section of the solid plane Y = C (C is a constant). The black circles (•) in FIG. 7 represent grid points on the boundary of the region. In FIG. 3, 302 is such an XYZ3.
A device for generating a cross section of a solid in a dimensional space by a plane parallel to the XZ plane or the YZ plane, and 303 is a processor for searching the boundary of this cross section. If this cross-sectional view is generated in the range of 0 ≦ Y ≦ << height of frame >>, and the area boundary is searched in each cross-sectional view, this corresponds to searching the surface of the solid body described above. This means that the correspondence between the area boundaries was searched between. FIG. 8 shows the result of searching the boundary of the sectional view of FIG. Normally, it can be assumed that the motion of the object in the target world of the image is sufficiently small with respect to the sampling interval of the frame, so that the region boundary in the cross sectional view as shown in FIG. 7 should be a smooth curve.
This is not smooth as shown in FIG. 8 because the accuracy of the area division for each initial frame is not sufficient, and this is corrected to have a smooth area boundary as shown in FIG. Generating the cross section is the main role of the 206 processor in FIG. First, 206 generates a smooth approximated curve as shown in FIG. 9 from the area boundary curve as shown in FIG. 8, and then uses this as a boundary curve to generate a corrected divided area on the cross section. The same correction division area is generated for all Y cross sections of 0 ≦ Y ≦ << frame height >>, and the content of the memory 203 in FIG. 2 is changed (corrected) based on this. Finally, what is shown in FIG. 10 is obtained.

EFFECTS OF THE INVENTION As described above, according to the present invention, in the area division of a moving image,
It is possible to correct the area division for each frame by searching the correspondence of the area boundaries between the frames from the area division result for each frame, and it is possible to greatly improve the accuracy of the area division as a whole. Therefore, the present invention is a TV
It can be applied to high-efficiency moving image compression in telephones and video conferences, and construction of moving image parts database for video composition in broadcasting system, and its effect is great.

[Brief description of drawings]

FIG. 1 is a block diagram showing a conceptual configuration of a moving image area dividing device according to the present invention, FIG. 2 is an overall block diagram of a moving image area dividing device according to an embodiment of the present invention, and FIG. 3 is FIG. A detailed block diagram of the region division result storage memory and the boundary correspondence search processor in FIGS. 4 to 10 is for explaining FIG. 3 and a method for correcting the region division in the embodiment in detail. FIG. 11 is a block diagram of a conventional moving image segmentation device, which shows sample data used in the above, intermediate results, and the like. 101 ... Moving image data storage means, 102 ... Area division operation means, 103 ... Area division result storage means, 104 ... Interframe area boundary correspondence search means, 105 ... Boundary correspondence storage means, 106 ... Calculation means for performing area division correction, 107 ... Area storage correction result storage means, 108 ... Labeling operation means.

Claims (1)

(57) [Claims] 1. A first storage means for storing moving image data input along a flow of time, and a section for dividing one frame of the moving image data stored in the first storing means into regions. No. 1 calculation means, second storage means for storing the results of area division respectively obtained from a plurality of temporally consecutive frames by the calculation means, and frames stored in the second storage means. Second operation means for searching correspondence of area boundaries between a plurality of consecutive frames based on the result data of each area division, and storage of correspondence of area boundaries searched by the second operation means The result of the area division for each frame stored in the second storage means is corrected based on the correspondence between the third storage means and the area boundary between the frames stored in the third storage means. A third calculation means for Fourth storage means for storing the result of the area division for each frame corrected by the calculation means, and the first data based on the result data of the correction area division for each frame stored in the fourth storage means. Moving image area dividing device, which includes a fourth calculating means for labeling all pixels belonging to the same area in all frames of moving image data stored in the storage means.