JPH06326980A - Movement compensating type processing system of picture signal - Google Patents

Abstract

PURPOSE: To reduce the ability of a video signal processor to be required, without reducing the quality of a conversion processing and resolution so much. CONSTITUTION: A sub-sampling equipment 170 sub-samples the input image of an input video signal 50 so as to generate a sub-sample image. A block comparator 190 compares the block of an image element from a sub-sample image pair so as to generate plural original correlative faces. A correlated face processor 200 generates plural interpolation correlated faces from the original correlated faces by interpolation. A movement vector estimating equipment 210 executes interpolation between the correlative values on the respective interpolation correlative faces so as to detect the point of max. correlation on the interpolation correlative face, and the respective movement vectors are generated from the respective interpolation correlative faces according to the point. A movement vector selector 230 selects the movement vectors to be used for the interpolation of the respective image elements in the output image of an output video signal 60. A movement compensating interpolator 140 interpolates an output image from a pair of the input images of the input video signal corresponding to the pair of the sub-sample images in accordance with the respectively selected movement vectors.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion compensation video signal processing system (apparatus and method).

[0002]

2. Description of the Related Art Motion-compensated video signal processing is based on television standard conversion, film standard conversion, and video (vide conversion).
o) Used for applications such as conversion between standard system and film standard system.

British Patent Application Publication No. GB-A-223
In a motion-compensated television standard conversion, such as the converter described in U.S. Pat. No. 1749, a set of motion vectors is processed in which a pair of consecutive input images is processed to indicate image motion between said pairs of input images. ) Occurs. The processing is performed on separate blocks of the image, so that each motion vector represents the inter-image motion of the contents of the respective block.

Each set of motion vectors is then fed to a motion vector reducer which obtains a subset of the set of motion vectors for each block. The subset is then provided to a motion vector selector that assigns one of the subsets of the motion vector to each pixel (pixel) in each block of the image. The selected motion vector for each pixel is provided to a motion compensated interpolator, which takes into account motion between the input images and operates on successive scans of the input images to produce successive outputs. Interpolate the image.

[0005]

Motion-compensated video signal processing, such as that described above, is for performing the very large number of calculations required to generate and process motion vectors for each pair of input images. It requires powerful and complex processing equipment. This is especially true if the image is in a high definition format or if the processing should be performed on the input video signal so that the output video signal is output in real time. In the above-mentioned case where the output video signal is output in real time, in order to generate a set of motion vectors for each output image within the available time (eg, output field period), a large number of devices of the same device may be used. You will be operating the sets in parallel.

An object of the present invention is to provide a motion-compensated video signal processing device and method in which the above points are improved.

[0007]

The present invention has solved the above-mentioned problems by constructing a motion-compensated video signal processing apparatus as set forth in claim 1.

As described above, the motion-compensated video signal processing requires the video signal processing apparatus to have a high processing capability, especially when it is to be performed on a high-definition video signal in real time. The present invention recognizes that many features of the motion compensation process specifically require it, and provides measures to reduce the processing requirements of these features. This may eliminate or reduce the need for parallel processing to generate and use the motion vectors, thereby reducing the complexity (and corresponding cost and size) of the device.

The measures provided by the present invention are as follows. 1. Interpolating the output image (eg, field or frame) directly from the input image. This obviates the need for progressive scan conversion of the input image (eg field or frame). 2. Generating a motion vector from a sub (down) sampled version of the input image. by this,
Processing costs for block matching are reduced. 3. Interpolating a larger number of correlation surfaces from those generated by block matching. Each interpolated correlation surface can then be used to generate motion vectors for subsequent use in interpolation. 4. Detection of the minimum point (the point with the maximum correlation) in each interpolated correlation plane up to sub-pixel (finer than pixel) accuracy. This helps reduce the resolution loss of the correlation surface caused by the subsampling process. 5. Interpolating pixel values for use in the test block during motion vector selection. Similarly, this helps reduce the loss of resolution of the correlation surface caused by the subsampling process.

Preferably, the input digital video signal has a predetermined resolution, and the apparatus has means for receiving the digital video signal, and the received digital video signal has a resolution lower than the predetermined resolution. And a means for adding the dummy pixel value to the received digital video signal to generate the input digital video signal.

In a preferred and simple embodiment, the dummy pixel value is a pixel value indicating a black pixel.

[0012] Even when the output digital video signal is a high definition video signal, it is possible to view, record, or transmit a display of the output of the device using a conventional definition device. Therefore, it is preferable that the apparatus includes a down converter for generating a second output digital video signal from the output digital video signal described above. The second output digital video signal has a lower resolution than the first-mentioned output digital video signal.

Preferably, the apparatus comprises a first time axis modifier for selecting a pair of input images for use in interpolating a respective output image and a pair of sub-sampled images for the first time. A second selection corresponding to the pair of input images selected by the axis modifier for use in generating a respective set of motion vectors indicative of image motion between the pair of subsample images. And a time axis changer of

In the preferred embodiment, the first time base modifier generates a control signal indicative of the time position of each output image for the pair of input images selected for use in interpolating the image. Means and the second time base modifier includes means for generating a control signal indicative of the time position of each output image for the pair of subsample images selected for use in the generation of motion vectors. .

Although the device is useful when used to process conventional definition (or resolution) video signals, the input digital video signal is preferably a high resolution video signal.

The input digital video signal is preferably an interlaced video signal.

The device is useful for many different video signal formats. However, the input digital video signal is 1125/60, 2: 1 interlaced video signal, 1125/30, 1: 1 non-interlaced video signal, 1250/50, 2: 1 interlaced video signal, 1250/25, 1: 1 non-interlaced video signal, 525/60, 2: 1 interlaced video signal, 525/30, 1: 1 non-interlaced video signal, 625/50, 2: 1 interlaced video signal,
625/25, 1: 1 non-interlaced video signal,
And 1125/24, 3232 pull-down (pull-dow
n) Preferably selected from the group consisting of video signals.

Similarly, although the apparatus is useful when used to process conventional definition (or resolution) video signals, the output digital video signal is preferably a high resolution video signal. The output digital video signal is preferably an interlaced video signal.

The output digital video signal is 1125 /
60, 2: 1 interlaced video signal, 1125/3
0: 1: 1 non-interlaced video signal, 1250 /
50: 2: 1 interlaced video signal, 1250/2
5, 1: 1 non-interlaced video signal, 525/6
0: 2: 1 interlaced video signal, 525/30,
1: 1 non-interlaced video signal, 625/50,
2: 1 interlaced video signal, 625/25, 1:
1 non-interlaced video signal, and 1125/2
It is preferably selected from the group consisting of 4,3232 pulldown video signals.

The apparatus according to the present invention is effectively used particularly in a television standard system conversion, a film standard system conversion, or a conversion between a video standard system and a film standard system.

In a second aspect, the present invention provides a motion-compensated video signal processing method as set forth in claim 16.

The above and other objects, features and advantages of the present invention will be apparent from the following detailed description of the embodiments described with reference to the accompanying drawings.

[0023]

DESCRIPTION OF THE PREFERRED EMBODIMENT FIG. 1 is a schematic block diagram of a motion compensation television standard conversion apparatus. The device includes an input interlaced digital video signal 50 (eg, 1125/6).
Receives 0: 2: 1 high definition video signal (HDVS),
Output interlaced digital video signal 60 (for example, 1
250/50 2: 1 signal).

The input video signal 50 is first supplied to an input buffer / packer 110. For conventional definition input signals, the input buffer / packer 110 formats the image data in high definition (16: 9 aspect ratio) format by adding black pixels as needed. For HDVS input, input buffer /
The packer 110 simply buffers the data.

The data is the input buffer / packer 1
10 to the matrix circuit 120. In the matrix circuit 120, the colorimetry of the input video signal (if necessary) is standard CCIR Recommendation 60.
1 ”(Y, Cr, Cb) colorimetry or the like, which is converted to the colorimetry of the desired output signal.

The input video signal from the matrix circuit 120 is converted into a time base changer and a time base changer.
and delay) 130, and also to subsample time base changer and delayer 180 via subsampler 170. Time axis changer and delay device 1
Reference numeral 30 determines the time position of each field of the output video signal, and selects two fields of the input video signal that are temporally closest to the output field for use in interpolating the output field. To do.
For each field of the output video signal, the two input fields selected by the time axis changer are:
It is delayed appropriately before being fed to the interpolator 140 which interpolates the output field. A control signal t indicating the time position of each output field for the two selected input fields is supplied from the time axis changer and delay device 130 to the interpolator 140.

Sub-sample time base changer and delayer 18
0 behaves similarly, but uses the spatially subsampled video image provided by the subsampler 170. A pair of fields corresponding to the pair selected by the time axis modifier 130 is selected by the subsample time axis modifier and delay unit 180 from the subsampled image. This field pair is the one used in the generation of the motion vector.

The time base modifiers 130 and 180 may operate according to a sync signal associated with the input video signal, the output video signal, or both. If only one sync signal is provided, the timing of the fields of the other of the two video signals is based on the time base changers 130, 18.
It is determined and generated within 0.

Sub-sample time base changer and delayer 18
The pairs of fields of the sub-sampled input video signal selected by 0 are provided to the motion processor 185. The motion processor 185 includes a block matcher (bloc).
k matcher) 190, correlation surface processing apparatus 200, motion vector estimator 210, and motion vector reducer 220
And a motion vector selector 230 and a motion vector post-processing device 240. The pairs of input fields are first provided to the block matcher 190. The block matcher 190 searches for a search block in the earliest of the two selected input fields in time.
Compute a correlation surface showing the spatial correlation between the block) and the (larger) search region of the later of the two input fields in time.

From the correlation surface output by the block matcher 190, the correlation surface processing device 200 generates more interpolated correlation surfaces. These interpolated correlation surfaces are
It is provided to the motion vector estimator 210. The motion vector estimator 210 detects the point of highest correlation in the interpolated correlation plane. (The original correlation surface actually shows the difference between the blocks of the two input fields, which is
This means that the point of maximum correlation is actually the minimum point on the correlation surface, and is hereinafter referred to as the “minimum point”. 3.) To detect the minimum point, additional points on the correlation surface are interpolated, providing some compensation for the loss of resolution caused by using the subsampled image to generate the surface. From the detected minimum point on each correlation surface,
The motion vector estimator 210 uses the motion vector reducer 22.
Generate a motion vector supplied to zero.

The motion vector estimator 210 also provides a confidence test for each generated motion vector.
test) to establish whether the motion vector is significant with respect to the average data level, and to associate a confidence flag indicating the result of the confidence test with each motion vector. The confidence test, known as the "threshold" test, is described in GB-A-2231749 (along with some other features of the device of Figure 1). The trust test is
It will be described in more detail later.

A test for detecting whether each vector is aliased is also performed by the motion vector estimator 21.
Performed by 0. In this test, the correlation surface (away from the exclusion zone around the detected minimum) is examined to find the next lowest minimum. If this second minimum is not located at the edge of the exclusion zone, then the motion vector obtained from the original minimum is said to be potentially aliased (which may be fake). , Flagged.

The motion vector reducer 220, before the motion vector is supplied to the motion vector selector 230,
Operates to reduce the selection of possible motion vectors for each pixel of the output field. The output field is conceptually divided into blocks of pixels. Each block has a position in the output field that corresponds to the position of the search block in the earliest of the selected input fields. The motion vector reducer is
Group together a group of four motion vectors to be associated with each block of the output field. At this time, each pixel in the block is eventually interpolated using a selected one of the group of four motion vectors.

Vectors that have already been flagged as fake are re-qualified during vector reduction if they are identical to unflagged vectors in nearby blocks.

The motion vector reducer 220, as part of its function, passes a "good" motion vector (ie, passes the confidence test and the fake test, or
Alternatively, count the frequency of occurrence of those "good" motion vectors, without considering the position of the blocks of the input field that were used to obtain the motion vectors, again qualified as non-fake. The good motion vectors are sorted in order of decreasing frequency. The most common of the good motion vectors that are significantly different from each other are classified as "global" motion vectors. Three motion vectors that pass the confidence test are selected for each block of output pixels and provided with the zero motion vector to a motion vector selector 230 for further processing. These three selected motion vectors are, in a predetermined priority order, the following (i)-
(Iii) is selected. (I) motion vector generated from the corresponding search block ("local" motion vector) (ii) motion vector generated from the enclosing search block ("adjacent" motion vector) (iii) said global motion vector

The motion vector selector 230 also receives as inputs the two input fields selected by the subsample time base converter and delay unit 180 and used to calculate the motion vector. These fields are moved by the motion selector 230 at the same time as the vectors from which they are derived.
Is properly delayed as supplied to the. This motion vector is selected from the four motion vectors for the block provided by the motion vector reducer 220.

The vector selection process includes detecting the degree of correlation between the test blocks of the two input fields indicated by the motion vector under test.
The motion vector with the highest degree of correlation between the test blocks is selected for use in interpolating the output pixels. By the vector selector,
A "motion flag" is also generated. If the degree of correlation between blocks indicated by the zero motion vector is larger than a preset threshold, this flag is set to "static (stat
ic) ”(no movement) is set.

The vector post-processor reformats the motion vector selected by the motion vector selector 230 to reflect any vertical or horizontal scaling of the image, and reformats this reformatted vector. It is supplied to the interpolator 140. Using the motion vector, the interpolator 140 takes into account any image motion indicated by the motion vector currently supplied to the interpolator 140, and the time axis modifier and delay device 130.
Interpolate one output field from the corresponding two (non-subsampled) interlaced input fields selected by

From the two selected fields supplied to the interpolator, if the motion flag indicates that the current output pixel is within a moving or time varying part of the image. Pixels of the two input fields are combined in a relative ratio according to the time position of the output field for the two input fields (as indicated by the control signal t), so that a larger ratio of the closer input fields is used. To be If the motion flag is set to "static", the temporal weighting is 5 for each input field.
It is fixed at 0%. The output of interpolator 140 is provided to output buffer 150 for output as a high definition output signal and to down converter 160 which uses the motion flag to generate a conventional definition output signal 165.

The downconverter 160 allows a conventional definition device to be used to monitor, transmit and / or record a display of the device's output (eg, a high definition video signal). This is advantageous because the conventional high-definition recording device is significantly cheaper than the high-definition device and is very popular. For example, simultaneous output of conventional definition video and high definition video would be required for each transmission over terrestrial and satellite channels.

The sub-sampling unit 170 performs horizontal and vertical spatial sub-sampling of the input video fields received from the matrix circuit 120, and then supplies these input fields to the time axis changing unit 180. (2:
The input field is first pre-filtered by a half-bandwidth low-pass filter (in this example of 1 horizontal decimation), then 1 along each video line.
Horizontal subsampling is a simple operation in that every third video sample is discarded, thereby reducing the number of samples along each video line by half.

Vertical subsampling of the input field is complicated in the present embodiment by the fact that the input video signal 50 is interlaced. This means that successive lines of video samples in each interlaced field are effectively two video lines that are separated, and that said line in each field is one video line before or after one video line of a complete frame. It means that it is displaced vertically from the line in the field.

One approach to vertical sub-sampling is to perform a progressive scan conversion (to produce consecutive progressively scanned video frames, each having 1125 lines), then the progressively scanned. It may be to subsample the frame by a factor of 2 to perform the vertical subsampling.
However, efficient progressive scan conversion may require a certain amount of motion compensation processing, which is performed by the motion processing apparatus 1.
It may adversely affect the operation of 85. Moreover, real-time progressive scan conversion of high definition video signals may require particularly powerful and complex processing equipment.

A simpler approach to vertical subsampling is shown in FIG. In FIG. 2, the input field is first low pass filtered in the vertical direction (to reduce potential aliasing) and then down (for even fields) or (for odd fields). ) A filtering operation is performed that effectively displaces each pixel vertically by half the video line. The resulting displaced field is roughly equivalent to a vertically subsampled progressively scanned frame by a factor of two.

In summary, therefore, the result of the subsampling operation described above is that motion processor 185 operates on a pair of spatially subsampled input fields in the horizontal and vertical directions by a factor of two. This is a four-step process required for motion vector estimation.
It is reduced by the rate of.

FIG. 3 is a schematic diagram of the correlation surface 300. The correlation surface shows the difference between the search block of the earliest of the two input fields from which it is generated and the (larger) search area in the slower of the two input fields. . Therefore, the peak in the correlation is represented by the minimum point 310 on the correlation surface 300. The position of the minimum point 310 on the correlation surface 300 determines the magnitude and direction of the motion vector obtained from the correlation surface.

In the apparatus of FIG. 1, each motion vector is generated by detecting the minimum point on its respective correlation surface. In total, for each pair of input fields provided to motion processor 185, 8000 correlation surfaces are provided to vector estimator 210 for use in generating 8000 motion vectors.

In order to reduce the processing requirements of the device of FIG.
Only one-fourth of the total correlation surface is the block matching unit 190.
Generated by comparing the blocks of the two subsample input fields supplied to The correlation surface to be used to generate the motion vector is then interpolated from the correlation surface generated by block matching. This is 20
The 00 "original" correlation surface is the block matcher 19
0 means that it is supplied to the correlation surface processor 200, which then generates 8000 “interpolated” correlation surfaces from the 2000 original correlation surfaces. To do. The 8000 interpolation correlation surfaces are used for motion vector estimation.

4, 5 and 6 show the correlation surface processing device 2
2 schematically shows the interpolation of the correlation surface performed by 00.

Referring to FIG. 4, each original correlation surface 4
00 compares the search block at a particular position in the earlier one of the pair of subsample input fields with the (larger) search area of the other one of the pair of subsample input fields, ( Block matcher 190). As shown in FIG. 5, the search block is centered at each position (eg, position 410) within the grid pattern 420 located on each input field. The original correlation surface generated from the search block is the lattice 4
It has corresponding respective positions 410 on the 20.

As mentioned above, in order to increase the number of correlation surfaces from 2000 generated by block matching to 8000 required by the motion vector estimator 210,
The interpolation processing is performed in the correlation surface processing device 200, so that each of the original correlation surfaces 400 to the four interpolation correlation surfaces 43
0,440,450,460 are generated. (In practice, the filtering process is used so that each interpolated correlation surface depends on the number of surrounding original correlation surfaces.

The interpolated correlation surfaces 430, 440, 450,
The center 460 of the effective position of the original correlation surface 400 is an effective position 460 that is slightly displaced horizontally and vertically from the position of the original correlation surface 400.
ons). The displacement is shown in FIG. 4 as a fraction of the horizontal and vertical spacing of the original correlation surface grid (ie, the search block grid 420). In particular, four interpolated correlation surfaces generated from the original correlation surface 400 (hereinafter, the correlation surface may be referred to as CS) 430, 4
The displacements of 40, 450 and 460 are as follows. Interpolation correlation surface 430: (-1/4 horizontal, -1/4 vertical) Interpolation correlation surface 440: (-1/4 horizontal, + 1/4 vertical) Interpolation correlation surface 450: (+ 1/4 horizontal) , +1/4 vertical) Interpolation correlation surface 440: (+1/4 horizontal, -1/4 vertical)

The effect of interpolating the correlation surface using the displacements described above is shown in FIG. FIG. 6 shows a grating 470 at the effective position 480 of the interpolated correlation surface. For comparison, the grating 420 indicating the position of the original correlation surface is also shown (in broken lines) in FIG.

The use of interpolated correlation surfaces to generate motion vectors will be described with reference to FIGS. 7 and 8A and 8B. In particular, FIG. 7 is a schematic view of a portion 500 of an image in which a vertical bar 510 rotates clockwise, and FIGS. 8A and 8B are respectively for representing the movement of the bar 510. 3 shows an original correlation surface and an interpolated correlation surface generated from the image.

In FIG. 7, the horizontal component of the top movement of the bar 510 is + va and the horizontal component of the bottom movement of the bar 510 is -va . Through the zero at the center of rotation 520 of the bar 510, the horizontal component varies continuously along the length of the bar.

As shown in FIG. 8A, three original correlation surfaces 530, 540 and 550 are the parts 50 of the image.
Respective cross sections, originating from 0 and passing through each of these original correlation surfaces, are shown. In the correlation surface 530, the minimum point 535 represents the movement of the top of the bar 510,
Corresponds to the horizontal component of + va movement. In the correlation surface 540, the minimum point 545 is at the point exhibiting zero movement and thus represents the movement of the center 520 of the bar 510. Finally, in the correlation surface 550, the minimum point 555 is at the point showing the horizontal component of -va and thus represents the movement of the bottom of the bar.

As described above, the original correlation surface 53
0,540,550 are not used by the vector estimator 210 in the generation of motion vectors. Instead, an interpolated correlation surface of half the horizontal and vertical spacing of the original correlation surface is generated for use in motion vector interpolation. Five such interpolated correlation surfaces 560,57
A cross section through 0,580,590,600 is shown in FIG. 8 (b). The five interpolated correlation surfaces 560, 570, 5
Reference numerals 80, 590, and 600 denote the respective minimum points 565, 575, 585, which indicate the following horizontal motion components.
It has 595 and 605. Minimum point 565: + va (horizontal movement of top of bar 510) Minimum point 575: + va / 2 (horizontal movement of point between top of bar 510 and center 520) Minimum point 585: 0 (bar 510 The center 52
Horizontal movement of 0) Minimum point 595: -va / 2 (center 52 of bar 510)
Horizontal movement of point between 0 and bottom) Minimum point 605: -va (horizontal movement of bottom of bar 510)

Five interpolated correlation surfaces 560, 570, 58
0,590,600 thus represents the movement of the bar 510 rotating at twice the vertical spatial resolution of the original correlation surfaces 530,540,550.

FIG. 9 and FIG. 10 following it are schematic block diagrams showing respective halves of the correlation surface processing apparatus 200 in which the correlation surface is interpolated.

In the apparatus shown in FIGS. 9 and 10, the input data 610 indicating the original correlation surface is supplied from the block matching unit 190 to a number of correlation surface delay units 620 in serial form. Each correlation surface delay device 620 delays the input data 610 by a period equivalent to the transmission time of data indicating one correlation surface. This is the correlation surface delay unit 62
It means that the data at each input and output of 0 represents the same point in two adjacent correlation planes.

The input data 610 has two delays that delay the data by a period equivalent to the transmission time of a complete row of the correlation surface (ie, the correlation surface generated from a complete row of the search block). It is also supplied to the row delay devices 630 and 640. This means that the data at the input and output of the row delay (630 or 640) represents the same point in the two correlation planes at corresponding positions in two adjacent rows. The output of each of the row delay units 630 and 640 is 4 in series.
It is supplied to one correlation surface delay device 620.

The input data 610, the data at the output of each of the correlation surface delays 620, and the data at the output of each of the row delays 630, 640 are filtered by the respective filter coefficients before being added by adder 650. C00, C01, ...
., C42 is multiplied. Addition output 66 of adder 650
0 represents consecutive points in one interpolated correlation surface. By using the filtering arrangement shown in FIGS. 9 and 10, each interpolated correlation surface is obtained by a filtered combination of data indicating the corresponding positions in the 15 surrounding original correlation surfaces.

By generating the required 8000 correlation planes using the apparatus shown in FIGS. 9 and 10, than is required when 8000 correlation planes are generated directly by block matching. Significantly less data processing hardware is required.

When the correlation surfaces interpolated by the apparatus of FIGS. 9 and 10 are provided to the motion vector estimator 210, they are examined and the point of minimum difference in each plane (the point of maximum correlation).
To detect. Each motion vector is generated from the point of the minimum difference. The second lowest lowest point away from the exclusion area around the actual lowest point is also detected from the correlation surface. A confidence test is then performed to determine the significance of the original minimum for the rest of the correlation surface.
e) is detected. Only those motion vectors that pass the confidence test are made available for subsequent use by the interpolator 140.

The fact that the input fields supplied to the block matcher are spatially subsampled means that the calculation of the original correlation surface requires less processing than would otherwise be required. It means to finish. However, the sub-sampling also has the result that the spatial resolution of the original correlation surface and the interpolated correlation surface is reduced by a corresponding amount.

FIG. 11 is a schematic block diagram of the motion vector estimator 210 showing how the reduction in spatial resolution of the correlation surface caused by the spatial subsampling of the input field can be overcome. There is. The motion vector estimator 210 is the block matching unit 1.
90 receives the correlation surface in digital form. The motion vector estimator 210 includes a maximum correlation detector 212, a correlation surface interpolator 214 and a revised maximum detector 216. The revised maximum detector 216 supplies the motion vector from the correlation surface to the motion vector reducer 220.

The operation of the motion vector estimator 210 is shown in FIG.
Will be described with reference to. FIG. 12 passes through the correlation surface (A
-Shows an array of points on the correlation surface, along with a cross section (along line A). The maximum correlation detector 212 is the block matching device 1
It receives the correlation surface from 90 and finds the point of maximum correlation, such as point 218, called the "original" maximum. Maximum correlation detector 212 outputs the value of point 218 and its position on the correlation surface.

The correlation surface interpolator 214 receives the position of the original maximum point 218 together with the data representing the original correlation surface, and interpolates eight additional points surrounding the point 218 using a two-dimensional interpolator. To do. The interpolated correlation value at the interpolated point is a revised maximum detector 21 for detecting which one of the interpolated correlation values exhibits a maximum greater than the original maximum.
6, together with the original maximum value. In the example shown in FIG. 12, since the interpolation point 219 shows the maximum larger than the point 218, the motion vector determined by the point 219 is generated and given to the motion vector reducer 220.

As described above, the motion vector reducer 22
0 operates to reduce the selection of possible motion vectors for each pixel of the output field before the motion vectors are provided to the motion vector selector 230. The output field is conceptually divided into blocks of pixels, each block having a position in the output field that corresponds to the position of the search block in the earlier of the selected input fields. The motion vector reducer assembles a group of four motion vectors to be associated with each block of the output field, each pixel in the block eventually being selected from among the four motion vectors of the group. It is interpolated using one. The four motion vectors (the zero motion vector and the other three vectors) for each block are provided to the vector selector 230.

The motion vector selector 230 corresponds to each block for use in interpolation of each output pixel.
Select one of each of the two motion vectors. This selection is done by comparing the test blocks of pixels indicated by each of the four motion vectors and selecting the motion vector with the highest degree of correlation between the test blocks. To make the use of spatially subsampled input fields less impact on this process, the pixel values in the test block are interpolated from the surrounding pixels, as described below.

13 and 14 show pixel interpolation performed by the motion vector selector 230.

As described above, the motion vector selector 23
0 is the local and global motion vector from the motion vector reducer 220, the two sub-sampled input fields from which the motion vector was generated, and the time base modifier 180 And the control signal t. For each pixel 700 in the current output field 705,
The motion vector selector determines each of the four possible motion vectors for the pixel in the previous input field 7
Test by comparing the block of pixels specified by the motion vector in each of the 10 and subsequent input fields 720. This comparison is done by calculating the sum of absolute luminance differences between corresponding pixels in the two blocks. The lower the sum, the higher the correlation between the blocks. However, input field 7
Every other pixel is missing from the block because 10,720 is sub-sampled horizontally by a factor of 2 and vertically by a factor of 2, and therefore the test performed during motion vector selection. Therefore, interpolation is used to reconstruct the missing pixels. In FIG. 14, the missing pixels to be interpolated are P1 to P5.
Represented by the existing pixels A, B, C,
It is generated from D as follows. P1 = (A + B) / 2 P2 = (A + C) / 2 P3 = (A + B + C + D) / 4 and so on. The example of converting one interlaced video format to another interlaced video format has been described.
Suitable for conversion between numerous interlaced and non-interlaced film and video signal formats including: 1125/60, 2: 1 interlaced video signal, 1125/30, 1: 1 non-interlaced video signal, 1250/50, 2: 1 interlaced video signal, 1250/25, 1: 1 non-interlaced video signal, 525/60, 2: 1 interlaced video signal, 525/30, 1: 1 non-interlaced video signal, 625/50, 2: 1 interlaced video signal,
625/25, 1: 1 non-interlaced video signal,
And 1125/24, 3232 pull-down (pull-dow
n) Video signal.

While the illustrated embodiments of the invention have been described in detail herein with reference to the accompanying drawings, the invention is not limited to these exact embodiments, but rather is defined by the claims. Various changes and modifications can be made without departing from the scope and spirit of the invention.

[0074]

In summary, therefore, a number of the features of the present invention include standard conversion to high definition video signals and standard conversion from high definition video signals without significantly degrading the quality and resolution of the conversion process. It helps to reduce the processing requirements for format conversion and standard format conversion from high definition video signals to high definition video signals. These features are as follows. 1. Interpolation of output fields from interlaced input fields. This eliminates the need for progressive scanning of input fields. 2. Generation of motion vectors from the spatially subsampled version of the input field. As a result, the processing cost for block matching is reduced by a rate of 4 in this embodiment. 3. Interpolate 8000 correlation surfaces from 2000 original correlation surfaces. Although 8000 motion vectors can be generated and made available in subsequent interpolations, the block matching process is only performed 2000 times. 4. Perform minimum detection in each interpolated correlation plane up to sub-pixel accuracy. This helps reduce the loss of correlation surface resolution caused by the subsampling process. 5. Pixel values are interpolated for use in the test block during motion vector selection. This also helps reduce the loss of correlation surface resolution caused by the subsampling process.

As a result of applying the measures described above, in combination with the effect of reducing the above-mentioned processing requirements, the device is arranged for real-time conversion between high definition video signal formats (in this example). The effect that it can be obtained.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of a motion compensation television standard conversion apparatus using the present invention.

FIG. 2 is a schematic diagram illustrating vertical subsampling of an interlaced video field.

FIG. 3 is a schematic diagram showing an example of a correlation surface.

FIG. 4 is a diagram (part 1) for schematically explaining interpolation of a correlation surface.
Is.

FIG. 5 is a diagram (part 2) schematically explaining interpolation of a correlation surface.
Is.

FIG. 6 is a diagram (part 3) schematically explaining the interpolation of the correlation surface.
Is.

FIG. 7 is a schematic view of a portion of an image including a rotating bar.

8 is a schematic diagram of an original correlation surface and an interpolated correlation surface obtained from the image of FIG. 7.

FIG. 9 is a schematic block diagram of a half of the correlation surface processing device.

10 is a view continuing from FIG. 9 and is a schematic view of another half of the correlation surface processing apparatus.

FIG. 11 is a schematic block diagram of a motion vector estimator.

FIG. 12 is a diagram illustrating an array of points on the correlation surface.

FIG. 13 is a diagram (part 1) schematically explaining pixel interpolation performed during motion vector selection.

FIG. 14 is a diagram (part 2) schematically explaining pixel interpolation performed during motion vector selection.

[Explanation of symbols]

 140 Interpolator (Motion Compensation Interpolator) 160 Down-converter 170 Sub-Sampler 180 Time Axis Changer and Delayer 185 Motion Processor 190 Block Matcher (Block Comparator) 200 Correlation Surface Processor 210 Motion Vector Estimator 220 Motion Vector Reducer 230 Motion vector selector 240 Motion vector post-processing device

Continued Front Page (31) Priority Claim Number 9307410: 2 (32) Priority Date April 8, 1993 (33) Priority Claim Country United Kingdom (GB) (31) Priority Claim Number 9307411: 0 (32) Priority Date April 8, 1993 (33) Priority claim United Kingdom (GB) (31) Priority claim number 9307442: 5 (32) Priority date April 8, 1993 (33) Priority claim United Kingdom (GB) (31) Priority claim number 9307448: 2 (32) Priority date April 8, 1993 (33) Priority claiming country United Kingdom (GB) (31) Priority claim number 9307473: 0 (32) Priority date 1993 4 March 8 (33) Priority claim United Kingdom (GB) (72) Inventor Martin Rex Dricot United Kingdom Hampshire, Basingstoke, Basing, Ringfield Claus 6 (72) Inventor Carl William Walters UK Berkshire, Redede Packaging, Great Norris Street 139

Claims (16)

[Claims] 1. A sub-sampler for sub-sampling an input image of an input digital video signal to generate a corresponding sub-sampled image and a block of pixels from a pair of said sub-sampled images to compare a first plurality. Block comparators for generating original correlation surfaces, each of the original correlation surfaces including an array of correlation values indicating a correlation between blocks of the respective pixels; and interpolating from the original correlation surfaces. Means for generating a second plurality of interpolated correlation surfaces, the correlation surface generating means wherein the number of the second plurality is greater than the number of the first plurality, and the correlation in each interpolated correlation surface. Means for interpolating between the values to detect a point of maximum correlation in the interpolated correlation surface, and according to the detected point of maximum correlation in the interpolated correlation surface, from each interpolated correlation surface Interpolation of each pixel of the output image of the output digital video signal by comparing the means for generating each motion vector with the test block in the pair of sub-sampled images indicated by the motion vector under test. Motion vector selection means for selecting a motion vector for use in each test block, each test block including a pixel of the respective sub-sample image and a test value interpolated from the pixel, and each of the selected A motion compensation video signal processing device, comprising: a motion compensation interpolator that interpolates the output image from a pair of input images of the input digital video signal corresponding to the pair of sub-sampled images according to a motion vector. . 2. The motion-compensated video signal processing device according to claim 1, wherein the input digital video signal has a predetermined resolution, the means for receiving the digital video signal, and the received digital video signal. Motion compensation video signal, comprising means for adding the dummy pixel value to the received digital video signal to generate the input digital video signal when the resolution is lower than the predetermined resolution. Processing equipment. 3. The motion-compensated video signal processing device according to claim 2, wherein the dummy pixel value is a pixel value indicating a black pixel. 4. The motion-compensated video signal processing device according to claim 1, further comprising a down converter for generating a second output digital video signal from the first-mentioned output digital video signal, the second output. A motion-compensated video signal processing apparatus, wherein the digital video signal has a lower resolution than the output digital video signal described above. 5. The motion-compensated video signal processing device according to claim 1, wherein a first time axis modifier for selecting a pair of input images for use in interpolating each output image, and a sub-sampled image. Corresponding to the pair of input images selected by the first time axis modifier,
For use in generating each set of motion vectors that indicate the motion of the image between the pair of subsampled images,
A motion-compensated video signal processing device, comprising: a second time axis changer for selecting. 6. The motion-compensated video signal processing device according to claim 5, wherein the first temporal axis changer outputs each of the pairs of input images selected for use in interpolation of the image. Means for generating a control signal indicative of a temporal position of the image, wherein the second time axis modifier comprises the time of each output image for the pair of sub-sampled images selected for use in the generation of the motion vector. A motion-compensated video signal processing device comprising means for generating a control signal indicating a position. 7. The motion-compensated video signal processing device according to claim 1, wherein the input digital video signal is a high-resolution video signal. 8. The motion-compensated video signal processing device according to claim 1, wherein the input digital video signal is an interlaced video signal. 9. The motion-compensated video signal processing device according to claim 1, wherein the input digital video signal is 1125/60, a 2: 1 interlaced video signal, 1125/30, a 1: 1 non-interlaced video signal. Signal, 1250/50, 2: 1 interlaced video signal, 1250/25, 1: 1 non-interlaced video signal, 525/60, 2: 1 interlaced video signal, 525/30, 1: 1 non-interlaced video signal Signal, 625/50, 2: 1 interlaced video signal, 625/25, 1: 1 non-interlaced video signal,
And a 1125/24, 3232 pull-down video signal selected from the group consisting of: 10. The motion-compensated video signal processing device according to claim 1, wherein the output digital video signal is a high-resolution video signal. 11. The motion-compensated video signal processing device according to claim 1, wherein the output digital video signal is an interlaced video signal. 12. The motion-compensated video signal processing device according to claim 1, wherein the output digital video signal is 1125/60, a 2: 1 interlaced video signal, 1125/30, a 1: 1 non-interlaced video signal. Signal, 1250/50, 2: 1 interlaced video signal, 1250/25, 1: 1 non-interlaced video signal, 525/60, 2: 1 interlaced video signal, 525/30, 1: 1 non-interlaced video signal Signal, 625/50, 2: 1 interlaced video signal, 625/25, 1: 1 non-interlaced video signal,
And a 1125/24, 3232 pull-down video signal selected from the group consisting of: 13. A television standard format conversion apparatus comprising the motion-compensated video signal processing apparatus according to claim 1. 14. A film standard format converter comprising the motion-compensated video signal processor according to claim 1. 15. A standard format conversion apparatus for converting between a film standard format and a television standard format, including the motion-compensated video signal processing apparatus according to claim 1. 16. Sub-sampling an input image of an input digital video signal to generate a corresponding sub-sampled image, and comparing blocks of pixels from the pair of sub-sampled images to compare a first plurality of originals. Generating a correlation surface, each of the original correlation surfaces including an array of correlation values indicating a correlation between blocks of the respective pixels, and a second plurality of interpolation values from the original correlation surface by interpolation. Of the second plurality of numbers is greater than the first plurality of numbers, and the interpolated correlation is obtained by interpolating between correlation values in each interpolated correlation surface. Detecting a point of maximum correlation in the plane, generating a respective motion vector from each of the interpolated correlation planes according to the detected point of maximum correlation in the interpolated correlation plane, and Selecting a motion vector for use in interpolating each pixel of the output image of the output digital video signal by comparing the test blocks in the pair of sample images indicated by the motion vector under test And each test block includes a pixel of the respective sub-sampled image and a test value interpolated from the pixel, and the sub-sampled image of the sub-sampled image according to the selected motion vector. Interpolating the output image from a pair of input images of the input digital video signal corresponding to a pair, the motion-compensated video signal processing method.