JPH01500238A - Method and device for measuring motion in television images - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed description of the invention] Motion measurement of TV images 1.0 Technical field Knowledge of the speed and direction of motion of all parts of the TV picture is very useful. There are many image processing applications that can be used. These applications include standard format conversion, noise removal, compression methods, in-band compression schemes, and others where some temporal interpolation operation is required. is included.

Information about the panning (horizontal rotation) and motion of relatively large objects in a scene. It would also be very effective.

1.1 Conventional technology Various techniques have been used to measure motion in TV images. These inpositions Another promising method is to measure large motions in a complete scene with sub-vixel accuracy. It appeared that the method was based on phase correction, which was shown to be possible. (J, J., Pearson, D., C.

Hines, S., Golosman, C.D., Kuglin, Video-speed image correlation processor JS, P, 1. E, Volume 119, Digi Application of digital image processing (I OCC 1977)) 1.2 The present invention The object of the invention is to use the known technique to analyze many objects moving in different directions and speeds. The goal is to extend this to the measurement of motion vectors in scenes containing

The invention is particularly defined in the copulatory claims.

Desirable implementation modes of the present invention will be explained in the following two sections.

2.0 Basic phase correlation method The method of calculating the motion vector for each pixel in an image is divided into two steps. Ru. The first step is to determine the main motion vectors present in the scene. Contains a continuous image (or field, depending on the exact application). The second step is this This is an attempt to assign one of the vectors to each pixel. About a certain vixel For example, if the vixel is a very small object or an uncovered background. If it corresponds to a land, it is not possible to assign a motion vector.

2.1 Step 1 - Vector measurement In the first stage of the process, a two-dimensional fast calculation of the luminance components of two successive images is performed. A Fourier transform (FFT) is calculated. At this time, each spatial frequency component in the transformation A vector of length 1 unit is calculated for each image, and its phase angle is is equal to the difference in phase between the frequencies of Fouri on the resulting array of complex numbers. An inverse transform is performed and this operation yields a real array that gives the correlation between the two images. Ru. Mathematically, if G, , G2 is a two-dimensional discrete Fourier of two consecutive images If it is a transformation, the complex number array Z is calculated for each spatial frequency (m, n) using the following formula. calculated. That is, also, the phase correlation is given by the inverse Fourier transform of Z, and this Only these will have real components.

The resulting phase-correlated array can be thought of as a surface, with a certain point ( When the relative displacement between this point is (x, y), its height at x, y) is It is proportional to how well the images correlate. For a simple shift between two images is a Δ function whose correlation surface is centered on the shift vector. The main luck in the scene It is considered that a peak value exists within this plane for each motion vector. These movement vectors Measuring Torr involves looking for large peak values within this plane. Relative peak value Height will reflect the relative size of moving objects. The main method is A new feature point can be found by looking for several peak values instead of just one, and thereby The purpose is to enable detection of many speeds with one calculation.

To measure motion vectors with sub-pixel accuracy, an interpolation in the correlation plane It is necessary to perform the operation.

In order to measure as many velocities as possible in one scene, It is helpful to divide the image into blocks rather than performing correlation operations.

This means that the number of individual peak values that can be accurately detected is blocked by noise. This is because it is limited to about three peaks per peak. Furthermore, if the peak value of velocity divided by a shift vector larger than about 1 pixel per field. If we do so, we can only resolve the velocity peak values. The size of the block is Because the method cannot correlate moving objects between blocks, the expected This is larger than the largest shift amount. 64x 64 pixel size is convenient. On the other hand, it is only necessary to measure a small number of the most important motion vectors. If not, the entire image, or at least very large blocks, may be transformed.

This division of the blocks is carried out only for the purpose of measuring the motion vectors, i.e. It should be recalled that vectors are still allocated pixel by pixel.

2.2 Step 2 - Vector assignment The first step in the process is to provide a set of motion vectors present in the scene. However, it does not tell us anything about which parts of the scene move by which vectors. stomach. The second step is to "try" each possible vector for each pixel. and assigning the vector that yields the best "fit". luck in blocks If dynamic information is required (e.g. block-based bandwidth) of the compression system. case), it is necessary to assign one vector to each pixel, and this The allocation will be done on a block-by-block basis.

Computes the absolute value of the difference between two input images when shifted by the motion vector of interest By doing so, an "error surface" is formed for each detected vector. well arranged The error resulting in areas of the two images that match is small, so these areas It is appropriate to assume that it corresponds to an object moving with a specific motion vector. Ru. Some kind of spatial filtering is applied to the error surface to reduce the noise effect. It is desirable to do so.

If the motion vectors are measured with sub-vixel precision, then when calculating the error surface It may be necessary to use some kind of spatial interpolation operation.

Once all motion vectors measured in the first stage have been estimated, each The cell is assigned the motion vector that produced the smallest error value. acceptable One threshold error level can be set that defines the error level. workman ``Unknown motion'' for pixels higher than this level where Ra has each estimated vector A flag can be attached. Such vixels are probably extraordinary This will correspond to the covered or uncovered background of the motion area.

If the motion vector transforms the input image in blocks rather than as a whole, First, the measurement is performed in the vicinity of the vixel in question. It is worth estimating the vector. is the most vixel contains this vixel. By simply considering the vectors measured in the blocks containing these It is likely that you will have the correct vector assigned to the file. Ideally, direct encirclement The vectors measured in the Bixel should be estimated. This means that the moving object is small. This is important when a block invades an adjacent block.

2.3 Advantages of phase correlation operation There are two basic methods of momentum estimation other than phase correlation operations, namely spatial temporal gradient method and optimal matching.

Both of these methods do not work well for large motions, and they also Although iterative use can help, these techniques still accelerate rapidly. It is not useful for objects that are subject to change and takes a considerable amount of time to recover after a sudden change. can be obtained. Divide the image into small blocks and assign motion vectors to each block. This means that the edges of moving objects are often This means that they will have incorrect velocity vectors assigned to them.

Phase correlation operations overcome the problems in these more "traditional" approaches. small For small movements, all three methods look roughly similar, but for large movements, , the phase correlation method becomes just that.

Performing a correlation operation between large parts of two consecutive images can be done in a very large number of trials. This is equivalent to using the optimal matching method on the calculated vectors. Perform a correlation operation like this The quickest method is to calculate the FFT of both images, multiply them together, and then By converting back the resulting array. Once this attempt is used , performing phase correlation is a little extra effort. This method is The method has the advantage of being insensitive to changes in scene brightness, and also improves correlation. It also tends to produce sharper peak values. The phase correlation method is the cross-correlation method. Although this can be implemented with simpler hardware than that required for This is because it is possible to very roughly quantize the unit amplitude vector on the Z axis. be.

This approach has the inherent advantage of detecting the dominant motion vectors in the scene.

These vectors typically have large background areas and objects of large motion. corresponds to These are the types that the human eye can easily follow and are therefore suitable for This is the darkest part of the scene to be processed correctly.

3.0 Drawings and Description of the Preferred Embodiments The present invention provides a phase correlation method computer. Examined in detail through data simulation and further detailed explanation regarding drawings. It will be. In the drawing, Figures 1 to 4 and 6 to 9 will be explained in further detail below. FIG.

In all these figures, the Z axis indicates the peak value of the correlation, while the X and Y axes indicate the peak value of the correlation. The X-axis is denoted by X-■ and Y-V, respectively, and is measured in vixel/field periods. Direction velocity and Y-axis direction velocity are shown.

FIG. 1 shows a correlation surface for a still image using a 5IN(x)/x interpolator, Figure 2 shows lO pixels per field again using a 5IN(x)/z interpolator. Figure 3 shows the surface for a pair of show the side of Figure 4 corresponds to Figure 2, but is a diagram using a smoother-interpolator. Figure 5 shows a windowed representation of the phased array prior to inversion; Figure 6 shows that the input image has 1 pair of lO pixels per field. shows the correlation surface for those with "band", Fig. 7 corresponds to Fig. 66, but also shows an image with a high cosine hole in the input image. show, Figure 8 shows the correlation surface for a moving object on a stationary background, Figure 9 shows the moving game in the Voith (VoH) J sequence shown below. Figure 1O shows the correlation surface for motion compensation. Show how the velocity vector can be assigned as, Figure 11 shows that both the large vector and the small vector fit one point below. It shows what you get.

The purpose of the computer simulations that have been carried out so far is to use the phase correlation method. The purpose is to find out how useful it actually is. We have created a simulation It is not limited to algorithms that are easy to implement in hardware. , the intention was to find out how good the best algorithms are.

Future simulations will focus on real-time hardware configurations for certain applications. How much should the ``ideal'' algorithm be simplified to make it suitable for ? It's about finding out what you can do.

3.1 Consideration of vector measurement method The first stage of the study looked at the "vector measurement" stage of the algorithm. this stage The purpose of this is to study the illumination intensity of the phase correlation method and that the size of objects, the number of moving objects, the size of objects, the amount of noise, etc. It was important to know how much it depends on the situation.

3.1.1 Measurement accuracy in simple panning The first study was about simple panning. I did it. both relatively large, having the same size but slightly different locations The two image parts obtained from the image were correlated. Both image sessions (ses sion) is just the brightness of 54 pixels square and contains a certain amount of detail. Taken from part of the test slide POND.

Figure 1 shows the correlation surface obtained when two respective image parts are correlated. Ru. Phased array 2 is a 128 x 128 array before performing the inverse FFT. Filled with zeros to form, thus changing the value in half-integer shift to 5in(x)/ It became possible to interpolate using the impulse response of x. This is visible near the peak value. This causes visible "ringing". Figure 2 shows two images made in the same way but Includes a horizontal shift of 10 pixels between image sections. The location of this peak value is moved (L99. 0.02). Therefore, the shift is in lO pixels It was measured with an accuracy of a few hundredths of a pixel. The peak height is attenuated and noise is generated. However, both effects can be due to dark material appearing at the edges of the image.

Similar experiments were attempted at other shift values.

This technique is based on the amount of shift to the instrumental limit of half the image size (in this case, 32 pixels) was found to be able to be accurately measured. A larger size A shift simply occurs at a location that corresponds to an even smaller shift in the opposite direction of the peak value ( For example, a shift of +34 pixels appears as -30). partial shift up Measurements were made to an accuracy of about 0.02 pixels using the interpolation method described above. 5in(x)/ Simply using quadratic interpolation without first using x-interpolation reduces the measurement accuracy to about 0.1 pixel. It was lowered to le.

3.1.2 Effect of different interpolation filter windows 5in(x)/x interpolator The ringing caused by using produce false peaks that can often mask the peaks. This ringing is caused by different interpolation This can be reduced by using children. Before performing the inverse transformation, the interpolator It can be changed by displaying the phased array Z(m,n) in a window. . The effects of different windows were studied. All windows are of the form below. It is. That is, a+(1-a) cos (f/f, -π/2) f≦f0o f>f.

where f is the horizontal or vertical frequency, and fo is the frequency of the lower array before the zero is inserted. This is the highest frequency supported by Figure 5 shows the window display using graphs. It shows the configuration.

The solid line indicates that frequencies @0 to f0 are supported by array 2 and zero is fo to 2f, such an insertion. shows a simple window that allows interpolation of intermediate points. For this reason, 5 in(x)/x impulse response with sharp peaks but “ringing” arise. The dashed line gives rise to a main peak with coarse resolution, but also gives rise to smaller child peaks. This shows a window shaped like Humming (l(aIIlming). The original window display function was used in both horizontal and vertical directions. Against raJ A value of zero would result in a rectangular window that produced Figures 1 and 2, but a value of zero ．． A value of 54 resulted in a Hamming window. According to experiments, approx. A value of 0.8 provides a reasonable compromise between the spurious peak height and the sharpness of each peak. It was shown that Figures 3 and 4 correspond to Figures 1 and 2, except that a This is based on a 0.8 interpolation filter. This filter calculates the height of all peaks has the effect of reducing and slightly widening the interpolation peak position. Never.

The number of situations in which false peaks cause problems is small.

The type of interpolator that is needed to eliminate all false peaks is to accept the main peak value. Creates an unwieldy extension and is therefore not worth using for anything other than simple rectangular windows. .

Another way to avoid false peaks is to perform a peak search operation on the non-interpolated surface. That is true. Once the peak is found, the height of the correlation surface at the half-integer shift value can also be calculated using an interpolator with a response of the type 5in(x)/x . This has the further advantage of looking for smaller points to find the maximum point. . This attempt will be the subject of future computer simulations.

Obtaining some improvement in signal-to-noise ratio by using an “adaptive window” It is possible to

In the basic phase correlation method, the absolute value of all numbers in phased array 2 is set to 1. be done. This assumes that the signal-to-noise ratio of these components is small. , information at frequencies of small amplitude has even more weight than it should have. It means to have. By comparison, when performing cross-correlation, this kind of normalization does not occur, as a result the directional light peak is not wide, and the peak height depends on the scene. That will happen.

A compromise between phase correlation and cross-correlation methods is better than using either method. can bring about results. This results in the normalization of larger amplitudes and smaller amplitudes. This results in a reduction in the amplitude. These ideas follow the simulation work. This will be considered at this stage.

3.1.3 Effect of image windows on different inputs The window displays what should be done with the input images before they are transformed. It was suggested that The reasoning behind this is that the correlation in case of shifts with horizontal components An attempt to correlate left and right edges (for example) when calculating There is a risk of generating loud noise.

Experiments performed with zero guardbands (no other window displays) A small decrease in the speed may occur, but a false peak at zero velocity may form. showed that. Figure 6 shows the same correlation method as in Figure 2, but uses an input image filled with zeros. This shows a method of forming an image with 128 pixels square. false peak It is clearly visible and this peak is probably the difference between the image part and the zero "garter" in the two images. This is due to the sharp edges between the two bands and the correlation operation.

A window of high-closed cosine waves (of the form used in phased array Z) is added to the input image. The effect of using image areas was also considered. When used in conjunction with "gart band" , the false peak at zero velocity was removed (this is the difference between the image and the gartband) (This is consistent with the idea that it was caused by a sharp joint between in signal-to-noise ratio Further improvements were obtained. Figure 7 shows the 64-pixel width guard band and the image area. Correlation surface in the same image part as in Figure 2, but with a raised cosine aperture in It shows. The signal-to-noise ratio is further improved and false peaks at zero speed are eliminated. Had disappeared.

The computational disadvantage of this approach is that the transformation covers a relatively small region of the image. It is to be.

These studies show that the improvement in signal-to-noise ratio exists when the cosine aperture is high and surrounded by zero. We showed that this can be obtained by displaying the input image in a window.

However, this represents a significant increase in the processing power required, and the improvement lies in the signal-to-noise The ratio is very small (less than a factor of 2). Display another window Filling the input image with zeros without doing so is disadvantageous as it introduces false peaks. . Using a high cosine window without filling it with zeros reduces noise slightly. , but without using overlapping blocks to detect motion across the image. This means that it must be done. Therefore, in many applications, It's not worth bothering with some kind of window display.

3.1.4 Effect of noise The method by which noise in the input image affects the results of phase correlation operations was investigated. . A noisy image is created by adding random values (including a certain peak value) to the brightness at each pixel. Produced by adding to degrees. Different (as used to generate Figure 2) A correlation operation is performed between two images shifted by IO pixels with a level of noise. The work was done. 20dB peak noise below peak white level By adding the The height of the correlation peak value was reduced by the number 2.4. Measurement error is up to approximately 0.08 pixels increased. Increase the amount of noise below the peak white level by 10dB , a further decrease occurred by a factor of 2.2. The peak is still detectable and rough.

The measurement error rose to about 0.3 pixels. This means that noisy images are This suggests that this does not pose a major problem for this part of the dynamic measurement process. Text As mentioned in 3.1.2, the noise rejection ability is improved by a certain [adaptive software] for phased arrays. It is possible that this can be increased by using the "window operation method".

A possible way to reduce noise in the correlation surface is some kind of temporal filtering. would be to do so.

For example, a first-order iterative temporal filter applied to the correlation surface of each block is stationary or gradually changes while reducing the level of any noise peak value. Strengthen the peak. The disadvantage of this method is caused by rapidly accelerating objects The peak velocity is attenuated. unduly attenuating pure motion peak values. It should also be possible to select a filter that provides good attenuation of noise. . It is also desirable to detect sudden changes and prohibit temporal filter operation immediately after the change. That's a good thing.

3.1.5 Scenes Containing One or More Motion Vectors Once this method If shown to work well, it can be used for images containing moving objects in the background. I researched about it. The purpose of these studies is that this technique yields several peaks. to determine whether some moving objects can be detected accurately by I had to investigate.

64 pixel square image (Dick and Jean) J) is extracted and a 32x 32 pixel part from another image (“Test A cross on the blackboard (from ``Card F'') was inserted in the middle of this image. inserted picture Image edges can cause artificially high frequencies in the resulting image. As if not.

A second image is "blended" with the background image over a distance of 3 pixels. A similar image was formed, but the inserted image part was only 3 pixels to the left, and Shifted downward by 3 pixels.

FIG. 8 shows the resulting correlation surface. Interpolated peak location ( As before, fit the quadratic interpolated data to the 5in(x)/x interpolated data. ) is exactly 0. It was Ol Bixel. The height ratio is the object (1.6 compared to 4), but the relative height of the peak is Reflects the relative area and background of moving objects. near the peak Noise is caused by the uncovered background surrounding the inserted image portion. It is something.

Since the phase correlation method has no problem in accurately measuring this motion, it can be used in some even more demanding applications. experiment was attempted. The purpose of these experiments was to determine how small a moving object could be. The solution was to find out if it was still detectable. Image areas of various sizes The shift vector of (5,5) is I was made to "exercise" in Kutle. The motion vector of this object is Considering that it moved more than twice its own length, it is quite surprising that it is 2 pixels square. This small object was detected accurately.

Consideration of two moving objects on a stationary background usually involves three pinpoints. (one for the background and two for the object) can be detected. and showed. But in some cases, the velocity peaks are covered by noise (perhaps (due to no background) and false peak values from the 5in(x)/x interpolator The image became blurry. As mentioned earlier, the amount of noise is probably Some form of adaptive window for windows and phased arrays in the force image part It can be reduced by using dow operations.

False peaks from the interpolator can occur if the process of looking for peaks leads to an array that is not interpolated. If it is carried out in a timely manner, there will be no problem.

As a final test, the correlation for two consecutive fields from a real sequence A surface has been generated. A game that moves somewhat rapidly in a roughly stationary background. A portion of the ``Voith'' sequence was selected that showed the Figure 9 shows the obtained phase. It shows the relationship.

Two large peaks were observed at (0,03,−0,42) and (2,38−0,25 ) corresponds to the speed of Measure (using a ruler) to see if the speed of the gate is This suggests that the light period was about 2.40 pixels, and this suggests that the correlation operation It agrees very well with one of the values obtained.

Since the correlation operation was between two consecutive fields, half a pixel vertical A shift is expected, and this means that the detected vertical shift matches very well. The heights of the two large peaks are approximately the same, and the gate and barrier This reflects the fact that the background grounds occupy approximately the same size area.

3.1.6 Summary of motion vector detection research It was shown that it actually works as stated in the literature cited in . Panning and 1 Typical vector measurement accuracy of 0.02 pixels for two moving objects in both directions By first interpolating the correlation surface with a factor of 2, then (X and and y) to fit the quadratic value to the interpolated points on either side of the maximum. It can be obtained by by simply fitting the quadratic values to a non-interpolated correlation surface. Therefore, an accuracy of approximately 0.2 pixels can be obtained. The accuracy of this method is almost It was possible to detect very small objects without being affected by noise.

Measuring the speed of two or more objects in an image area that has been subjected to correlation operations In order to be able to do this, it is desirable to make certain modifications to the basic method. Two possible modifications are windowing the input image portion with a high cosine function, and perhaps filling this with zeros, which reduces noise in the correlation surface can do. Also, so that small amplitude components have relatively small weights, Reduce noise by adding some kind of adaptive window to the phased array. It would also be possible to do so. Problems experienced with false peaks caused by interpolation process is to perform a peak search process on an array that will probably not be interpolated. This can be avoided by Which of these improvements can be incorporated for a specific application? Further simulation operations will be required to determine whether it is worth pursuing.

3.2. Considering vector allocation The purpose of this part of the study is to address the second stage of the motion measurement process, i.e. It was to simulate assigning the main motion vectors to. Current The first problem to face is how many vectors have been measured and assigned. The goal was to find a way to demonstrate that. one between two input images using motion vectors It was decided to do this by interpolating the images in time. this is, In addition to rigorous testing of the method, it is also possible to And what kind of results can you get if you improve the film playing field by It shows what can be done.

Temporal interpolation operations between even fields result in a series of odd fields (some A computer program has been developed that can do this (and vice versa). This program RAM allows the user to specify the size of the input image, i.e. the size of the block on which the correlation operation was performed. change various parameters such as the number of vectors extracted per block, etc. made it possible.

3.2.1 Details of the method used All these considerations are based on a simple rectangular window in the image section and array Z. was carried out using the ``fundamental'' phase correlation method. As mentioned before, the correlation aspect is Interpolated by filling with zeros.

In order to find out how well this method works, we need to use as many motion vectors as possible. The block (64 pixels x 32 lines) is measured instead of the entire image. A correlation operation is performed. A maximum of three vertices were measured per block. be The trial calculation vector menu for a pixel includes the block containing the pixel and It consisted of vectors measured in immediately adjacent blocks. For this reason, the image For the pixels of one block in the middle of , a maximum of 27 vectors are estimated. will be done.

In order to assign one motion vector to one pixel, follow the steps in Section 2.2. As mentioned above, an “error surface” was calculated for each vector. The first task is to Always interpolate the input image using a simple algorithm, resulting in non-integer It is now possible to handle vector lengths. This algorithm consists of four It involves taking a weighted sum of the values of the nearest pixels. There are four points being interpolated. When the interpolated value matches the position of any pixel, the interpolated value is equal to the value of this pixel. The weights were chosen so that Subsequent studies focused on quadratic cubic spline fitting. We improved this algorithm by using the method. (H, S.

Hou and H. C. Andrews [Image Interpolation Methods and Digital Files] "3-dimensional splines for Ruther method" (IEEE Trans, ASSP, No. 85 SP-Volume 26, No. 6, December 1978) The "error surface" for each trial vector has a simple spatial aperture of the form below. Filtered by filter. That is, 1/(dx+dy+1)dx , dy≦20 dx, dy>2 where dx and ciy are the horizontal and Vertical distance Il! (in pixels). This type of filter is mainly Because it was easy to configure, it was widely used and could be refined. intermediate fill It can be shown that other forms of filter manipulation, such as filter manipulation, yield even better results. Ru.

A motion vector is assigned to each pixel, and no upper bound is set on the acceptable error. I couldn't. In an ideal configuration, corrections are made for pixels with large errors. will be studied and an uncovered background of a certain size will be created.

The brightness value of each pixel in the output image is determined by the motion vector for the pixel. Averaging the values in two adjacent fields at the displaced position calculated by.

Figure 1O shows the previous field 21 the next field N and the field being generated. It shows a cold G.

The brightness value of a point in field G is the same as that of points X and N in fields P and N. and 2, where points X and y are displaced from y by ±V/2. , ■ is the vector of velocity assigned at point y.

This vector minimizes the difference in brightness between points X and 2 (after spatial filtering) was selected to be.

3.2.2 Results Even fields from the "Voith" sequence using the method outlined earlier was generated by performing a motion-compensated interpolation operation from odd fields. .

Initial studies performed without spatial filtering in the error plane showed incorrect Vectors were often assigned to pixels due to noise. above spatial The filter solved this problem. Atonement using spatial filters The background immediately adjacent to an object often "lies along" the object. Ru. This problem can probably be resolved by changing the filter format. can. If the background is dark or blurry, try the basics below. There is a problem. That is, since the image information exists only in one of the two input images, A motion vector cannot be assigned to this. This problem requires another attempt , is briefly discussed below.

−H The interpolated image becomes very clear when spatial filtering of the error surface is included. One big problem remains. The lower part of an automobile radiator and its Interpolated image, except parts of the moving gate "pillars" are sometimes obscured. Looks correct for the most part. The problem is that both large and small vectors are valid. This is because a large motion vector is added to a slowly moving region when was discovered.

Figure 11 illustrates such a problem in the case of an automobile radiator. There is. This represents a stationary object 0 against a stationary uniform background. vinegar. ■1 and v2 are both in field G if the background is uniform. is a motion vector that can exist for point P. For example, object 0 is a car. Due to the radiator itself and the black area below it in the “Voith” sequence. The silver surround at the bottom of the radiator against the background given by The image I mentioned here is the BB CRe5ea that was emitted in July 1986. It is shown in Figure 3 of report 198615 of rch Dept. of the gate The reasoning behind its disappearance is a little more difficult to grasp. The gate is It is a periodic structure that moves horizontally by approximately 4.7 pixels, and is itself 14 pixels. Repeat spatially for each file. This means that there are two effective motion vectors for the gate. +4.7 and -9 per image period (ignoring gate edge effects), This means that there are 4 pixels. If an incorrect motion vector is selected If so, the gate "decomposes" in the interpolated image.

The problem is that the error for each vector is a function that increases with increasing vector length. It was alleviated by multiplying the surface. This means that the two vectors have roughly the same alignment. When an error occurs, it means that two vectors of this kind are allocated. for The weighting function used was of the form: That is, However, ■ is the length of the motion vector, and vo is the speed at which the weighting function becomes lO. this The functions were arrived at in a completely arbitrary way. V of 5 pixels per image period. The value of The program led us to believe that the gate was actually moving to the right instead of to the left. It was enough. In order to find out how generally valid this "deception factor" is, we It is necessary to conduct a study using image sequences.

When this feature was incorporated, an output image that was not at all unsightly was produced. The output image is , partly due to insufficient vertical detail available in the input image (odd fields ), and some are used when adding motion vectors with non-integer components. It was slightly soft due to the extremely simple spatial interpolator used. The spatial interpolator is 1 based on the fit with cubic splines, resulting in a large amount of processing time required. This resulted in a sharper output image, at the cost of increased sharpness.

1 from a Voith sequence with interpolation measures using a fully calibrated method. If you examine one field, you will notice that this field is particularly It can be seen that the image looks surprisingly good considering the lack of a 'do' algorithm.

The only remaining problem (apart from the underlying background) is the The motion vectors of various parts of the gate are sometimes assigned incorrectly (despite the cold factor). Also, the spatial filter in the error plane is The problem is that the edge of the land (land) can appear slightly collapsed. for the algorithm It should be possible to resolve these problems with further minor modifications.

For non-hidden background regions, examine the placement of motion vectors in the image. could be detected by. have different vector components perpendicular to the boundary. There must be some blurred or exposed background at the boundaries between the regions. . determine which image information belongs in such regions in the interpolated image. In order to do so, it is necessary to examine two or more input images.

These studies require dividing the input image into many blocks to perform correlation operations. , but this is not necessary in many applications. One of the Voyt sequences section converts the image as 1 large block instead of 16 small blocks. and by extracting four principal vectors from the correlation surface. operation was performed. This interpolated image is obtained using many blocks. It was almost indistinguishable from the others. However, this sequence is 3 It only shows the three main objects (the gate, the car, and the background) and This is a somewhat special case because none of them move very fast. much faster exercise A sequence containing rotating or rotating objects can be So it would probably look much better if we used vectors.

3.2.3 Summary of consideration of motion vector assignment It is possible to properly allocate motion vectors to pixels using a method based on It was shown that The only modification needed is to change the vector assignment to a smaller vector. This was solved by incorporating a weighting factor so that the

An improved filter for the error surface (possibly another kind of spatial filter, or temporal filter) operations into a vector assignment algorithm. Some minor fixes were made. You can also “fine tune” certain parameters. There is a certain range.

Furthermore, the improvements suggested in Section 3.1.6 regarding the vector measurement algorithm are also included. These improvements can also improve the performance of the method as a whole. Ru. If the motion vector is one large block instead of many small blocks, These improvements are particularly important if measured by transforming images Dew. Another useful experiment is to incorporate an “unhidden background” algorithm. This is probably an attempt.

4.0 Hardware configuration The details of incorporating the above method into the hardware are mainly based on motion information. It will depend on the intended use. However, may be reasonable for most configurations. There are several commonalities, some of which are discussed below. The following argument In the discussion, a sampling rate of 13.5 MHz is used.

4.1 Vector measurement hardware The vector measurement part of the hardware is a one-dimensional FF for one data array. It would be based on a circuit that can implement T. Such a circuit requires one In order to realize the FFT's "butterfly" J operation, could be constructed using a multiplier (or FROM) and an adder. . The data to be converted is held in RAM, and FROM is placed between the appropriate RAM addresses. The task is to generate an appropriate address sequence to perform the butterfly operation. Ru. Another FROM will hold the "twiddle factor".

By using a pipelined structure, such a circuit can perform an n-point complex FFT. It would take n·(logzn) clock pulses to do so. This way Such a circuit consists of 8 multipliers, 6 adders, and a FROM which holds the constraint factors. one and four RAMs, each holding a copy of the array to be converted. It turns out. The physical size of such a circuit depends on the number of bits required for computation (which is still undetermined). This number will probably be one or two, depending on the It will fit two 4U boards.

The control board holds the portion of the image (or the entire image) to be converted and sends it to F (one line or digit at a time) to the FT board. If converted If the image is just n pixels square, then 2n'log per 2D FFT, 2n FFTs are required resulting in n total clock periods.

Such a configuration can perform a complex FFT, so two real arrays are It would be possible to arrange the data in such a way that it can be transformed by the operation of 1 To calculate the correlation surface for two blocks, Zn”log2n Two two-dimensional TTRs are required resulting in a total number of clock periods such as .

Consider the simple case where we want to measure velocity for each image. If the image If transformed as one block of 512 elements on each side, the transformation is one FFT This can be done in about 0.35 seconds using blocks. The required conversion speed is 50Hz. 40 ms per image part at the field velocity, resulting in approximately 1O of this ``Una rFFT engine'' will be required. Therefore, there are about 80 multipliers in total. There will be 60 adders.

This requires a large amount of hardware, but it is completely possible to implement. Phase adjustment between conversions Additional hardware would be required to perform the ray multiplication, but this Only about 179 pieces of hardware are required for this.

Additional hardware may be required to perform peak searching and interpolation operations. . Such hardware would probably be a digital digital camera with a 3-pixel square aperture. Similar in size to a tal filter (determines whether one point has one maximum value) (because we need to examine 9 points to do this), instead of a multiplier will be based on a comparator. As such, this configuration requires a significant amount of It does not require any clothing.

4.2 The complexity of this part of the vector allocation hardware equipment is again a special It will depend very much on the application. In a standard converter at some point As such, one motion vector with sub-vixel precision is required for each vixel. , in in-band compression systems, vectors are assigned to pixels for each block. sub-vixel accuracy is not important. Attempts per block The number of computational vectors will also have a direct impact on the size of the hardware. cormorant.

For the worst case, one vector of sub-vixel precision is required per vixel. Let's think about it. Similar to two spatial interpolators, one subtractor and one spatial filter. Similar hardware would be required for each trial vector. This is related to That would mean something like 20 to 30 multipliers with adders. .

If we estimate 4 vectors per pixel, there will be a total of about 100 vectors. A multiplier would be needed.

On the other hand, vector assignment systems based on blocks where sub-vixel precision is not important It will probably be quite simple. Required hardware is slightly less per vector. It could be reduced to a number subtractor and adder.

The cost is small if the transformation is performed in relatively few blocks. However, the vector allocation hardware can be more complex.

If the motion vectors are not required with sub-pixel accuracy, then It is possible to filter the input image by a factor of 2 in both directions, so The cost of hardware could be reduced by a factor of 4.

5. Using 0-dimensional transformations The first step in the motion vector measurement method as described above is to measure two consecutive motion vectors in a sequence. This includes performing two-dimensional phase correlation operations between images. Contains (mXn) pixels. In the case of a single image portion of Fourier transform of length n is performed m times, and for each spatial frequency (m × n), It involves calculating the phase difference and performing the same number of inverse transformations.

The resulting phase correlation surface is then illuminated to find the location of the major peak values. will be met.

Most of this circuit uses one-dimensional filter operations on the image and then We perform a pair of orthogonal-to-dimensional transformations, so that the component of the principal shift parallel to an axis is This can be avoided by measuring. This means that the ``variable separable'' test It could be described as For example, an image contains all pixels in each pixel column. It is possible to first filter vertically by adding . then , the phase between the resulting row of values and the row of values obtained from the previous image filter operation Correlation operations will be performed. The main peak values in the correlation function are This will give the horizontal component of the principal motion vector at . A similar operation is the main This will be done to measure the vertical component of the shift.

This approach results in significantly fewer conversion operations. (nXn) suns For the image part of the pull, a total of 4n transformations are required for two-dimensional operations, but The variable separation method requires four times. The disadvantage of this approach is that the peak in the correlation function It is easy to receive noise and is even more difficult to pick up.

Therefore, although this method cannot unambiguously identify the location of the main peak, This is because there is no indication as to which horizontal and vertical components should be combined. It is. This is only a problem when more than one principal vector is involved. be. At least (n+1) times on axes where n peak locations are not parallel to each other It is possible to show that it can be uniquely identified by calculating the correlation operation. . However, in the case of n times, even if there are less than n axes, there are possible n The probability that there are more than one peak location is small.

For example, in a real case where you want to find 4 peaks, you would probably need 4 correlations. The operation will be sufficient. The probability of assigning an incorrect peak location is also the measured product of coordinates. It decreases with degree. Even if some uncertainty remains in the location of the peak, this by increasing the number of shift vectors attempted during the vector allocation phase. can be overcome.

In summary, the calculation required to find the main peak in the correlation surface for two images is The computational complexity is significantly reduced by performing many one-dimensional correlation operations on the filter images. can be reduced. This filter operation amount is the image along the line perpendicular to the correlation axis. Corresponds to the sum of elements.

This technique requires that correlation operations be performed on axes that are not parallel to each other. In order to uniquely identify the locations of n peaks using Although fewer correlation operations than usual are sufficient, at least (n+1) correlation operations are required. It is necessary to do so.

If four correlation operations are performed on one image part of d pixels square If the input image light and the required The number of Fourier transforms will be eight. For some typical block sizes, This number corresponds to a hardware savings of around a factor of 30. The sacrifices made are As the peak width becomes wider, there will be more noise, and some false peak locations may occur. Is Rukoto.

6.0 Application examples Tree terms are used to apply correlation operations to measure motion in television images. We will explain the novel method of According to computer simulations, books The method works very well (probably better than any other published method) Various adjustments to the zero-line approach have been suggested.

The present invention calculates each motion vector by determining the amount of correlation as a function of selective displacement. Determine the multiple peak correlation values corresponding to the torque, and then determine which of these motion vectors is the best fit in obtaining one from each other in pixels or in blocks. It can be seen that there is a combination of correlation operations on two images that are tested in lock units. Probably.

This method is a significant improvement in that the quality of devices such as standard format converters can be dramatically improved. seems to offer great hope. Furthermore, this method improves the depiction of motion in film. It opens the door to new uses such as goodness.

One useful application of this technique is to improve the quality of motion-adaptive band compression systems. That is. It measures the main motion vectors in a scene and Since the pixels are allocated in blocks (one block is approximately 6 pixels square), Become.

This motion vector measurement method performs a temporal interpolation operation on a series of images to or in any case where it is necessary to follow the movement of an object in a scene. can also be effectively applied. The following sections explain some usages in more detail. Ru.

6.1 Improvement of motion depiction in film When a film is shown on television, each frame is shown twice. (In interlaced systems, once the odd television field one time as an even field). This kind of structure A film shot is shot at a speed of 25 (or 24) frames per second (approximately). Make it send at the correct speed. The process of repeating an image over successive fields is Significantly reduces the level of flicker experienced if the image is shown only once However, this process also introduces motion disturbances; the viewer's eyes cannot see the moving object. When tracking, a double image appears on the retina, which means that each moving object appears at an appropriate point in time. This is because it is not shown in an appropriate position.

The movement disturbance caused by this system is If the image is generated using an interpolation operation with motion compensation, then the can do.

Each vixel in the intermediate image changes the appropriate motion vector by ±0.5. Average the brightness levels of corresponding pixels in the images before and after the image It can be generated by For color films, R, G, and B components The values of are obtained in this way.

A background area that is not hidden or unclear indicates that this area is a suitable motion vector. It cannot be processed in this way because it does not have a file. Processing these areas In order to It is necessary to determine whether it is compliant. This means that at right angles to the boundary To determine by examining the components of the motion vector in the surrounding area that forms Can be done. Areas of unhidden background point away from this area Neighborhood vectors, and blurred background regions are It will have a side vector. Is the image information in the non-hidden region the subsequent image? The information in the blurred areas must come from the previous image. No. to determine the location of the required image information in the subsequent or previous image. In order to It is necessary to

The requested area is not allocated when the motion is complemented later or earlier in time. The area begins or ends with.

These principles apply equally well to other fields or film speeds. . For example, when displaying a film using the 60)1z television system, two Intermediate frames will be generated between eight pairs of film frames. new The images are shown at positions 275 and 415 between one pair of frames, and at positions 1 and 415 between one pair of frames. 11 at positions 75 and 315, and so on.

Generation of an intermediate image at a point other than the middle of two frames can be done by assigning vectors. The above is done at a suitable point between images rather than in the middle of the image. It involves exactly the same process as described in .

The amplitude of the luminance (or color component) signal of each pixel in the generated image is Neighbors such that a proportionally larger weighting operation is given to the closest image values It is a weighted average of the values at appropriate points in the image.

6.2 High-quality format conversion When it is necessary to convert an image from one field speed to another always requires a temporal interpolation operation. By knowing the motion vector of the scene, The process is performed with significantly less disturbance than using traditional methods. enable.

To implement the motion-compensated standard conversion method, the signal must be in encoded form. If so, first decode this signal into luminance and chrominance, and then It is necessary to convert the tarlaced image to a progressive scan standard. This latter option The process itself can be obtained from knowledge of the motion vector of the scene, which is one Appropriate motion vectors with respect to other fields before interpolation of the line where the field fell. This is because it can be displaced by the Alternatively, a vertical temporal filter can be used. However, this method requires a certain amount of vertical detail in the moving area. Resulting in falling out. Once the input image is converted to a sequential format, motion compensation is applied. generate the image at the point the output field velocity requires using a temporal interpolation method .

This process, assuming the conversion to an interlaced standard, will The film's motion depiction, except that it only needs to generate every other line in the image. The process is the same as previously described for revalidation.

6.3 Generation of smooth slow motion sequences Short sequence television It is often necessary to play back John images in slow motion. conventionally , which repeats each field several times (or one field followed by one This is done by successively iterating the fields generated using an interpolation process. be exposed. As a result, a rocking motion occurs. special with very fast field velocity It is possible to obtain smooth slow motion using a camera, but this method poses operational problems.

Smooth slow motion generates extra fields using temporal interpolation operations If possible, this can be accomplished using a conventional camera.

This can be done using the process described earlier for standard method conversion. can. The number of intermediate fields that need to be generated depends on the amount by which the motion is delayed. will exist.

The electronics required to generate the intermediate field are simple slow motion Located at the output of a conventional video tape recorder with playback capabilities. This outfit position measures the number of times each field is repeated and replaces the repeated fields. This will generate an appropriate number of intervening fields.

To obtain clear images, it is necessary to use a camera with a short integration time. becomes. This does not create any major operational problems and is a practical option for future models. All melas will tend to have short integration times.

6.4 Video noise reduction To reduce noise in television images, filter the signal. It is necessary to use

Spatial filtering usually results in an unacceptable loss of image detail. Although a zero temporal filter operation can be very effective, a simple temporal filter operation motion causes severe degradation in the motion regions of the image, which is why such filter operations must be prohibited in these areas; therefore, moving objects This results in noise being generated in the surrounding area.

The motion vector measurement method incorporates a temporal filter method so that it can be extended to the motion domain. You can get into it.

−For order-iteration time filter operations, the frame store is results in shifting each vixel in by the corresponding motion vector. .

If this filtering operation is performed using a widthwise filter, then each store's The read address is the field between the output field and the stored field. The displacement will occur according to the sum of the local motion vectors over the number.

6.5 In-band compression Motion vector measurement methods require knowledge of the local motion vector in the signal. Since it is the key to the possibility of examining redundancy, almost all video bands medium pressure It can be used very advantageously in reduction techniques.

Adaptive subsampling scheme reconstructs dropped samples from previous fields Knowledge of the motion vectors can be used to make it possible to DPCM The method uses the motion vector as a reference for the expected pixel. I can do it. In both of these equations, information about the motion vector is divided into two Obtained at the transmitter by making measurements between consecutive fields and The rest can then be sent to the receiver.

6.6 Addition of color to black and white film Recently, “colorization” on) A process known as J (by Coloriza Lion) ) developed.

This allows a color version of a black and white film to be created. (on the computer Still frames from the film (stored) are “colored” by the artist be done. The computer then considers which areas are in a quiescent state and By coloring the area with the color previously specified for it, subsequent Attempt to color the frame. The motor area is not targeted and requires artist intervention. I need.

If the motion vector in one frame (by correlation operation with the previous frame) each point is assigned to be colored, and each point is associated with the previous frame indicated by a vector. Once the colors of points in the system are assigned, we can extend our method to the motion domain. I can do it. Small areas that require artist intervention are hidden in the background. areas of land and areas of complex movement that are not tracked. (The vector is directly There should never be an area that is allocated second in the image and appears as "unsharp". Therefore, there is no problem with blurred backgrounds. Obtaining this way, the motion vector Adding measurement operations to this process is a task that the artist must perform. This significantly reduces the amount of paint and thus significantly increases the speed of the coloring process. .

6.7 Increase in field speed Interpolation operations can also be used to increase field velocity for display purposes. can.

international search report

Claims (27)

[Claims] 1. In a method for mechanically measuring motion of a television image, Pair each motion vector by determining the correlation operation as a function of the selective displacement. a first step of correlating the two images to determine a plurality of corresponding peak correlation values; and Which of the motion vectors for each of the plurality of fundamental regions of the image moves one of the images to the other? a second step of testing which yields the best fit when obtained from the A method characterized by: 2. The selective displacement amount is two-dimensional, and a two-dimensional correlation operation is performed. The method according to claim 1. 3. Multiple one-dimensional correlation operations using one-dimensional displacements in different directions are used. A method according to claim 1, characterized in that: 4. The number of one-dimensional correlation operations is greater than two, and multiple motion vectors are Components in the different directions mentioned above that match the peak correlation value obtained by the correlation operation The method according to claim 3, characterized in that the vector is determined as a vector having Law. 5. The motion vector has sub-pixel accuracy by interpolating the correlation function. According to any one of claims 1 to 4, the measurement is performed up to the method of. 6. The correlation step is performed individually for a plurality of pixel blocks. The entire set of motion vectors determined by this is used in the second step. 6. The method according to any one of claims 1 to 5, characterized in that: 7. the correlation step is performed individually for a plurality of pixel blocks; In the second step, the motion vector determined in relation to a certain block. Claims characterized in that only The method according to any one of items 1 to 5. 8. The correlation step is performed individually for a plurality of pixel blocks; The motion vector used in the test for a certain block in the second step vector is determined for the block and a plurality of adjacent blocks. 6. A method according to any one of claims 1 to 5, characterized in that: 9. Each block is windowed using a high cosine wave or other window display aperture. Claims 6 to 8, characterized in that Method. 10. A claim characterized in that each windowed block is surrounded by zeros. The method according to item 9. 11. Claim 1, wherein the blocks are overlapping blocks. The method according to any one of Items 6 to 10. 12. Claim 1, characterized in that each basic region consists of one pixel. The method according to any one of items 1 to 10. 13. Claim characterized in that each elementary region is one block of pixels. The method according to any one of items 1 to 10. 14. Claims 1 to 4, wherein the correlation operation is a phase correlation operation. to the method according to any one of paragraphs 13 to 13. 15. The phase correlation function is obtained from the inverse Fourier transform of the product of the Fourier transforms of the two images. 15. A method according to claim 14, characterized in that: 16. The correlation function is temporally filtered before the peak correlation value is determined. Claims 1 to 1 are characterized in that the noise peaks are reduced by The method described in any of Section 5. 17. Said temporal filtering is temporarily inhibited at the point in time when an instantaneous change occurs. 17. A method according to claim 16, characterized in that: 18. an error surface is obtained in said second step for each motion vector; And for each pixel, the motion vector whose error surface has the smallest value is the motion vector for that pixel. Any one of claims 1 to 17, characterized in that it is allocated to a cell. The method described in 19. When the minimum value exceeds a certain predetermined threshold, the motion vector is 19. The method of claim 18, wherein: 20. Each error surface undergoes spatial filtering to reduce noise effects. 20. The method of claim 18 or claim 19. 21. The error surfaces are multiplied by a factor that increases with the vector length for these surfaces. Claims 18 to 20, characterized in that: Method. 22. according to the time of the interpolated image between the images before and after the interpolated image is received; Temporal interpolation of images using motion vectors to interpolate the position of moving objects. Claims 1 to 2 further include the step of performing an inter-operation. The method described in any of Item 1. 23. against an unobscured background that directs the direction in which the motion vector moves away. Image information is incorporated into the image that is interpolated from subsequent images, and motion vectors are directed The image information for the blurred background is interpolated from the previous image to the image. 23. A method according to claim 22, characterized in that it is incorporated. 24. Multiple images are interpolated between consecutive input images to create a slow motion sequence. or increase the field velocity for display purposes. 24. The method according to claim 22 or 23, characterized in that: 25. the image is subjected to a temporal filter operation to reduce noise; A motion pair whose motion is shifted field by field according to the assigned motion vector. Claims 1 to 21 are expanded to include Any method described. 26. The image contains motion vector information for reconstruction of the falling sample of the moving object. Claim No. The method according to any one of Items 1 to 21. 27. A black and white image per frame where the motion vector determines Claim characterized in that the claim is colored by an assigned color that automatically repeats. The method according to any one of items 1 to 22.