JPH02278479A - Labelling device for moving image - Google Patents

Abstract

PURPOSE:To apply separate labels to respective objects even when the objects are mutually overlapped in a moving image by using the calculation technique of optical flow, which shows the motion of each picture element for each frame, for labelling processing. CONSTITUTION:A labelling device 1, optical flow calculator 2, histogram calculator 3, peak calculator 4, center of gravity calculator 5, processed result storage device 6, inter-frame correspondence device 7 and table updating device 8 are provided. Based on a label picture, for which the moving image of a certain frame is labelled, and the optical flow of the frame, the two-dimensional histogram of the optical flow is calculated for each frame and compared with the processed result of the preceding frame concerning the peak of the histogram and quantity expressing the average position of each label. Thus, even when the objects are mutually overlapped in a certain frame, the original labels can be applied to the respective objects.

Description

[Detailed Description of the Invention] [Table of Contents] Overview Industrial Application Fields Conventional Technology and Problems to be Solved by the Invention Means for Solving the Problems Actions Examples Effects of the Invention [Summary] Labels for moving images Regarding the image processing device that performs labeling, even when labeled objects overlap in a moving image, it is possible to assign separate labels to them and to correspond between the objects between frames. ,
The purpose of the present invention is to configure an image processing device that can perform the processing more reliably, and includes a device that assigns a temporary label to each frame of a moving image, and a device that calculates an optical flow for each frame of the moving image. A device that calculates an optical flow histogram for each label assigned to each frame of the moving image, a device that calculates peak candidates of the calculated histogram, and a device that calculates the average position of each label. and a device for storing each of the processing results for the previous frame, and a device for storing the processing results for the previous frame, and the average position and peak candidate of the current frame. The system is configured to include a device that associates labels between frames and the current frame, and a device that calculates a correct Rahel image from the above-mentioned association results.

[Industrial application field]

The present invention relates to an image processing device that labels moving images.

In image processing, moving images are similar to TV screens.
It refers to a large number of images that change little by little over time, and each of these images is called a frame.

Therefore, as something specific to the moving image processing, there is a process of tracking which object in a certain frame corresponds to which object in another frame between frames.

This process can be performed, for example, in factory automation (F
It is essential for the robot's eyes in A) or for autonomous vehicles that recognize the outside world and run unmanned.

However, in the process of labeling objects in each frame of the moving image, in the conventional technology, if objects overlap, one label is assigned to them, and the frames are There is a problem in that the correspondence between objects is not perfect, and there is a need for an apparatus that can correctly label objects in each frame of a moving image.

[Prior art and problems to be solved by the invention] Fig. 3 is a diagram explaining the conventional labeling process for moving images, in which (a) shows an example of a human-powered image, and (b) shows a binarized image. Processing results are shown, (c) shows an example of the results of conventional labeling processing, FIG. 4 is a diagram illustrating calculation of optical flow, and (a) shows an example of an input image, (b
) shows an example of the optical flow calculation result.

First, conventional labeling processing will be explained with reference to FIG. Regarding the concept of the labeling process, for example, [“
Latest trends in image processing algorithms”.

Oplus E, separate issue, published by Shintechi Co., Ltd. Communications, provides details, but one example is shown below.

Input images from video cameras, TV cameras, etc. are
It is a grayscale image with a certain density.

Figure (a) shows an image of several frames of this image, and at this stage, the background and object are not completely separated.

This grayscale image is first subjected to binarization processing. The binarization process is a process in which the values of pixels with a certain density or higher are set to 1', and the values of pixels with a density lower than that are set to 0. The background and object can be completely separated.

The disulfurized image is subjected to a labeling process in which the connected pixels with the density value "1" are grouped together and given one label. For example, labeling is performed by scanning the image through a window necessary for labeling, and propagating the initially assigned label in the scanning direction. The label uses numbers.

The labeling process is performed for each frame, so
(c) As shown in the figure, the same label is not necessarily given to the same object.

Further, for objects that overlap in a certain frame, only one label is given in that frame even if different labels have been given in previous frames.

Next, optical flow calculation processing related to the present invention will be explained with reference to FIG.

Research on object recognition using computers has been actively conducted for a long time, but the current level is still not sufficient.

However, in the object recognition process, the technology for calculating the apparent velocity field (referred to as optical flow) from several 'a-light images located at the beginning is at a satisfactory level.

For details on calculating the optical flow, see, for example, "Visual Movement Measurement", Hildreth E.

Author C0, Massachusetts Institute of Technology Press, Cambridge, Mass+1984. ("The Measure
ment of Visual Motion”, l1i
ldreth, E, C,, former T Press, Camb
Ridge, Mass, 1984. ) J, but to summarize, (a) from the input image shown in figure,
This is a technology that calculates which direction the object is apparently moving from the difference in the brightness of each pixel from frame to frame. Optical flow is calculated by focusing on changes in .

The calculation result of the optical flow is (V
The components should be displayed as (x, Vy), etc., but (b
) In the example shown in the figure (however, in the figure (b), only specific pixels are shown), this is expressed by an arrow and shown superimposed on the input image.

In this way, in conventional labeling processing for video images, if objects overlap in a certain frame, a single label is assigned, making it impossible to completely match objects between frames. The problem was that I couldn't do it.

Furthermore, although a technique for calculating optical flow, which indicates the movement of each pixel for each frame, has been known, there has been no example of this technique being applied to labeling processing.

In view of the above-mentioned conventional drawbacks, the present invention provides an image processing device that labels moving images, even when labeled objects overlap in a certain frame of the moving image. and give them separate labels.
It is an object of the present invention to provide an image processing device that can more reliably match objects between frames.

[Means to solve the problem]

FIG. 1 is a diagram explaining the principle of the present invention, in which (a) shows an example of the device configuration, (bl) and (b2) show an example of a two-dimensional histogram of optical flow for one label, and (d) shows an example of a two-dimensional histogram of optical flow for one label. , (C2) shows an example of an image to be correlated, (dl), (d2) shows an example of peak extraction of an optical flow histogram, and (el), (
C2) shows an example of calculating the position of the center of gravity for each label.
shows an example of a correspondence that maximizes the evaluation function E.

The above problems are solved by a moving image labeling apparatus configured as follows.

(1) A device for labeling a moving image, comprising: a device 1 that assigns a temporary label to each frame of the moving image; a device 2 that calculates an optical flow for each frame of the moving image; A device 3 that calculates an optical flow histogram for each label assigned to each frame of the moving image; a device 4 that calculates peak candidates of the calculated histogram; and a device 4 that calculates an average position for each label. a device 5 for storing the processing results for the previous frame, and a device 6 for storing the processing results for the previous frame and the average position and peak candidate of the current frame. 1
It consists of a device 7 that correlates the labels of the previous frame and the current frame, and a device 8 that calculates a correct label image from the above correspondence result.

(2) The device 4 that calculates the peak candidates of the calculated histogram is configured to output the reliability of the candidates.

[Effect]

That is, the present invention focuses on the fact that the optical flow calculated for each pixel of each frame of a moving image is originally different between objects, and uses a label image that labels a moving image of a certain frame and an optical flow of that frame. Based on the flow, a two-dimensional histogram of the optical flow is calculated for each label, and the peak of the histogram is
A quantity representing the average position of each label, for example, the center of gravity and 1
By comparing the above processing results for the previous frame, it is possible to assign original labels to each object even when objects overlap in a certain frame.

Therefore, the image processing apparatus of the present invention has the configuration shown in FIG. 1(a).

First, based on the optical flow calculated by the optical flow calculation device 2 and the label assigned by the labeling device 1, the histogram calculation device 3 calculates a two-dimensional histogram of the optical flow for each label. (
Figure (bl) shows the optical flow calculated for each label, and Figure (b2) is an example of calculating its two-dimensional histogram.From this two-dimensional histogram, the optical flow is calculated as shown in Figure (bl). Two objects, 3g (b2)
In the two-dimensional histogram shown in the figure, it can be seen that two objects J can be recognized.The present invention was made with attention to this point.

However, since the optical flow histogram usually has ambiguous peaks (particularly for pixels in a portion where two objects overlap), this peak information alone is insufficient in accuracy.

Usually, the results obtained by peak search are only candidates that cannot be said to be absolutely correct. For example, the format is as follows.

Candidate 1: (Vx, Vy), the confidence that the peak is correct is 0.8° Candidate 2: (Vx, VyL, the confidence that the peak is correct is 0.6°) The above confidence is, for example, 2 of the optical flow. In the dimensional histogram, the confidence level of the highest pix (pixel number) can be set to 1.0°, for example, and the confidence levels of other peaks can be obtained by determining the confidence level of the other peaks according to the ratio of the respective pixel numbers. .

In the present invention, temporally, the processing result of the previous frame and the above-mentioned column of candidates for each label (candidate 1, candidate 2, . . .
~) and make the most consistent correspondence.

At this time, it is formulated in such a way that an appropriate evaluation function is defined and a correspondence that maximizes its value is searched for.

Figures 1 (cl) and (c2) are sample images for mapping, where (cl) shows the previous frame and (c
2) shows the current frame.

A two-dimensional histogram of optical flow is obtained for each label in this figure, and the result of calculating the peak is (di)
, (d2) (corresponding to figures (d) and (c2)).

Furthermore, the position of the center of gravity for each label is also used for comparison. However, for the previous frame, add a moving valve to the center of gravity,
Set the expected center of gravity position in the current frame. This result is shown in this figure (el), (e2) ((cl), (c2)
(corresponding figure).

In the present invention, these data are correlated with the previous frame, and the most appropriate one is determined from the aforementioned peak candidate sequence.

For this purpose, the following evaluation function E is defined, which takes a larger value as the match is better.

E=Σ f(x) * g(y) * p where f (
x) is the difference between the peak coordinates of i and j; y in g<y> is the difference between the centroid positions of the label to which i belongs and the label to which j belongs.

p is the confidence of j i is the peak in the previous frame. serial number j is the peak in the current frame. serial number.

It is.

The f (x) and g (y) are functions that output a larger value as the input becomes smaller, for example, l/ (1+x
), l/(1+y).

That is, in the example of g (y), it is the difference between the expected value in the next frame of the center of gravity position belonging to the serial number i assigned to the peak in the previous frame, so the smaller the difference, the thicker the (The same object can be searched between frames by making a correspondence that maximizes the evaluation function E using the above function.

The correspondence that maximizes this evaluation function E is determined by the above (cl)
, (c2), the above peak coordinates (
di), (d2) and the center of gravity position (el), (e
2), we get what is shown in figure (f).

At this time, the value of the evaluation function E is E = f (0) * g (0) 0.9 (label l
and label 2) + f (0) * g (1) * 0.9 (pic (0, 1) of label 2 and label 1) + f (0) * g (2) Book 0.9 (label 2 and label 1 peak (1,0) = 0.9 + 0.45 + 0.37 = 1.72. If other measures are taken, the values of f (x) and g (y) become smaller ζ, and the value of the evaluation function E is always smaller than the above value.

In this way, by performing the correspondence in a format that maximizes the evaluation function E, dynamic programming (hereinafter referred to as OP) can be used, and the evaluation process, that is, the maximum value The mapping process that gives

Once the correspondence is determined, the label of the previous frame can be assigned to the peak of the current frame,
This allows even if objects overlap,
Separate labels can be given. Furthermore, since correspondence can be established with the label of the previous frame, it is also possible to track an object having the label.

〔Example〕

Embodiments of the present invention will be described in detail below with reference to the drawings.

The above-mentioned FIG. 1 is a diagram explaining the principle of the present invention, and FIG. 2 is a diagram showing an embodiment of the present invention, and (a) is a diagram schematically showing an example of the configuration of the device. , (b) shows an example of the processing flow for l frame, (cl), (c2)
shows an example of an input image, (di) to (d6) show an example of the configuration of each table in the middle of each processing, and (e) shows an example of an output label image. Apparatus 1 for calculating pix candidates of a dimensional histogram;
2.3.4 and the average position of each label per frame,
For example, devices 6 and 7.8 for associating labels between frames based on the data from the device 1.5 for calculating the center of gravity position are necessary means for carrying out the present invention.

Note that the same reference numerals indicate the same objects throughout the figures.

The configuration and operation of the moving image labeling apparatus of the present invention will be described below with reference to FIG. 1 and FIG. 2.

Even if the present invention is implemented, the labeling process for the grayscale image of each frame of a moving image, the process of calculating the optical flow for each pixel from at least two grayscale images, and the process of calculating the optical flow for each pixel from the optical flow calculation result, the labeling process for each label. The process of calculating a two-dimensional histogram, the process of finding the maximum value of the number of pixels, that is, the peak, from the two-dimensional histogram, and the process of finding the average position of each label, for example, the center of gravity, are particularly different from the conventional method. The details are omitted here because the above techniques are used to label the grayscale images of moving images, and the peak candidates calculated from the two-dimensional histogram of optical flow for each label are The present invention focuses on a means for associating labels between frames based on columns and the average position of each label for each frame, for example, data on which the center of gravity position is calculated. The labeling device will be explained based on processing for one frame.

First, in Figure (a), an input image from a television camera or the like (not shown) is labeled by a labeling device 1 for each object in the image.

The label at this time is temporary, and as described above, only one label is given to overlapping objects. The output result of the labeling device 1 is stored in the device 1 as a label image.

Meanwhile, in parallel with the above labeling process, the optical flow calculation device 2 calculates the optical flow of each pixel based on the input image of two frames (previous frame and current frame), and the calculation result is are two images consisting of the X component and the X component of the velocity vector, and are stored in the device 2.

(See figures (b), (cl), (c2)) Next,
The histogram calculation device 3 calculates a two-dimensional histogram of the calculated optical flow for each label. The calculation results are stored in the work area of the device 3 as a two-dimensional array (the subscript is the velocity component and the value is the number). (Figure 1 (b2) and Figure 2 (b),
(di)) The peak calculation device 4 calculates peak candidates of the two-dimensional histogram for the histogram stored in the work area, and the result (candidate sequence) is the coordinates (velocity) of the peak candidate. component) and its confidence level (P)
For example, each label is stored in the device 4 in the format shown in FIG. 4(d2).

As mentioned above, for example, the confidence level of
The confidence level of the peak (with a large number of peaks) is set to 0.9'', and the confidence level of other peaks is determined according to the ratio of their respective heights (numbers).

In parallel with the above processing, the center of gravity calculation device 5 calculates a quantity representing the average position of each label, for example, the center of gravity position,
The calculation results are stored in the device 5 for each label. ((
b), (d3)) In this embodiment, as described above, the center of gravity is calculated as a quantity representing the average position of each label, but the present invention is not limited to this. It goes without saying that it may be the circumcenter, center of gravity, etc. instead.

Once peak candidate sequences have been found for all labels in the processing target frame, the interframe correspondence device 7 retrieves the barycenter position of the previous frame and the optical flow from the processing result storage device 6. (see figure d4)), from each of the above storage locations, the center of gravity position and optical flow of the current frame are retrieved, and the correspondence between the labels of each frame is determined so as to best match. I do.

At this time, the movement amount determined from the optical flow is added to the center of gravity position of the previous frame to convert it into the expected center of gravity position of the current frame, and then the association is performed.

As described above, this mapping is performed by searching for a mapping that maximizes the value of an appropriate evaluation function E, which takes a larger value as the match is better.

As mentioned above, the evaluation function E is, for example, E=Σ f(
x) * g(y) * p Here, X in f (x) is the difference between the peak coordinates of i and j, and y in (V) is the centroid position of the label to which i belongs and the label to which j belongs. The difference.

p is the confidence of j i is the peak in the previous frame. serial number.

j is the peak in the current frame. serial number.

It is.

In this way, dynamic programming can be used by evaluating, for example, selecting the one that takes the largest value from multiple candidates, and the label between frames can be very quickly calculated. can be mapped.

In this way, if the most consistent correspondence is obtained, it is output to the work area in the device 7 as a correspondence table. ((b), (cl), (c2), (d
5) See figure) Finally, the table updating device 8 updates the contents of the processing result storage device 6 by referring to the corresponding table. (d6)
(See figure) Also, the labeling device 1 calculates the correct label image from the temporarily assigned label image based on the main correspondence table, and outputs the correct label image as the output of the labeling device in the main moving image. . (See figures (b) and (e)) As described above, the present invention provides an image processing device that labels a moving image, in which the optical flow calculated for each pixel of each frame of the moving image is Focusing on the differences between objects, we calculate a two-dimensional histogram of optical flow for each label based on the label image labeled on a video image of a certain frame and the optical flow. with peak candidates.

An evaluation function that takes a larger value as the correspondence between the above processing result of the previous frame and the processing result of the current frame is better for a quantity representing the average position of each label, for example, the center of gravity. The feature of this method is that by comparing the use of E, it is possible to quickly attach the original label to each object even when the objects overlap in a certain frame.

〔Effect of the invention〕

As described above in detail, the moving image labeling device of the present invention includes a device that assigns a temporary label to each frame of a moving image, and a device that calculates an optical flow for each frame of the moving image. , a device that calculates an optical flow histogram for each label assigned to each frame of the video image, a device that calculates peak candidates of the calculated histogram, and a device that calculates the average position of each label. a device for storing each of the processing results for the previous frame; and a device for storing the processing results for the previous frame, and the above-mentioned one from the processing results for the previous frame and the average position and peak candidate of the current frame. It consists of a device that correlates the labels of the previous frame and the current frame, and a device that calculates the correct label image from the above correspondence result, so even when objects overlap in a moving image, They can be given separate labels. Furthermore, since correspondence can be established with each label in the previous frame, it is possible to track the object corresponding to the label.

[Brief explanation of drawings]

FIG. 1 is a diagram explaining the principle of the present invention. FIG. 2 shows an embodiment of the present invention. FIG. 3 is a diagram illustrating conventional labeling processing for moving images. FIG. 4 is a diagram illustrating calculation of optical flow. In the drawing, ■ is a labeling device. 2 is an optical flow calculation device. 3 is a histogram calculation device. 4 is a peak calculation device. 5 is a center of gravity calculation device, 6 is a processing result storage device, and 7 8 is a table update device. are shown respectively. Average position calculation device. is the inter-frame correspondence device (a) Y, number of pixels (cl) (c2) Figure 1 (Part 1) (dl) (el) Figure 1 (Part 2) (d2) (e2) Diagram (Part 1) (cl) Previous frame (c2) Current frame Diagram (Part 3) (d5) (d6) Round (Part 4)

Claims (2)

[Claims] (1) A device for labeling a moving image, which includes a device (1) that assigns a temporary label to each frame of the moving image, and a device (1) that calculates the optical flow for each frame of the moving image. 2), a device (3) that calculates an optical flow histogram for each label given to each frame of the video image, a device (4) that calculates peak candidates of the calculated histogram, and the label. a device (5) for calculating the average position for each frame; a device (6) for storing the above processing results for the previous frame; and a device (6) for storing the processing results for the previous frame and the above processing results for the current frame. Based on the average position and peak candidates, the above 1.
A device (7) that correlates the labels of the previous frame and the current frame, and a device (8) that calculates a correct label image from the above correspondence result.
) A labeling device for moving images, characterized by comprising: (2) The labeling device for a moving image according to claim 1, wherein the device (4) for calculating the peak candidate of the calculated histogram outputs a confidence level of the candidate.