JPH0865676A - Image dncoding device - Google Patents

Abstract

PURPOSE: To decrease the number of times of calculation for the interpolation of a luminance value by performing approximation with use of a picture element value at the position closest to this coordinate in a high definition reference image when the luminance value of the reference image at the position whose coordinate value is not an integer is required. CONSTITUTION: A high definition reference image 406 in which the luminance value at the point whose (x) and (y) coordinates are respectively the integer multiple of 1/m1 and 1/m2 (m1 and m2 are positive integers) in a reference image 404 is obtd. by interpolation is provided. Therefore, in the high definition reference image 406, a picture element exists at the point whose (x) and (y) coordinates are respectively the integer multiple of 1/m1 and 1/m2. When a coordinate value is required in motion estimating processing, the approximation is performed with the luminance value of the picture element existing at the position closest to this coordinate in the high definition reference image 406. Thus, since the interpolating processing is previously performed by a reference image high definition circuit 405, the calculation amt. can be reduced.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention eliminates the constraint that all pixels belonging to the same patch have a common motion vector.
The present invention relates to an image coding apparatus that executes a motion compensation method in which horizontal and vertical components of a pixel motion vector can take values other than integer multiples of the distance between adjacent pixels.

[0002]

2. Description of the Related Art In high-efficiency coding of moving images, it is known that motion compensation utilizing the similarity between temporally adjacent frames has a great effect on information compression. The motion compensation method which is the mainstream of the current image encoding technology is MPEG1 and MP which are international standards of the moving image encoding method.
This is block matching with half-pixel accuracy that is also adopted in EG2. In this method, an image to be encoded is divided into a large number of blocks, and the motion vector of each block is calculated in the horizontal and vertical directions with a half length of the distance between adjacent pixels being the minimum unit. This processing is expressed using mathematical formulas as follows. The predicted image of the frame (current frame) to be encoded is P (x, y), and the reference image (P
R (x, y) is a decoded image of a frame that is temporally close to and has already been encoded. Also, x and y
Is an integer, and it is assumed that a pixel exists at a point whose coordinate value is an integer in P and R. At this time, the relationship between P and R is

[0003]

[Equation 1]

It is represented by However, assuming that the image is divided into n blocks, Bi represents a pixel included in the i-th block of the image, and (ui, vi) represents a motion vector of the i-th block. The most commonly used motion vector (ui, vi) estimation method is (ui, v).
i) has a certain search range (for example, -15 ≦ ui, vi ≦ 1
5), which is a method of searching for the one that minimizes the prediction error Ei (ui, vi) in the block. When the average absolute error is used as the evaluation criterion of the prediction error, Ei (ui,
vi) is the original image of the frame to be coded I
As (x, y),

[0005]

[Equation 2]

It is represented by However, Ni is the number of pixels included in the i-th block. The process of evaluating the prediction error for each different motion vector and searching for the motion vector with the smallest error is called matching process. All (ui, v
The calculation of Ei (ui, vi) for i) and the search for the minimum value is called a full search.

In block matching with half-pixel precision, u
i and vi are calculated with the minimum unit of half the distance between pixels, that is, 1/2 in this case. Therefore, it is necessary to obtain the luminance value of a point where the coordinate value is not an integer and a pixel does not actually exist in the reference image (hereinafter, such a point is referred to as an interpolation point). As processing at this time, co-linear interpolation using four peripheral pixels is often used. When this interpolation method is described by a mathematical expression, the brightness value R (x + p, y) at the interpolation point (x + p, y + q) of the reference image is defined with p and q (0 ≦ p, q <1) as the decimal component of the coordinate value.
y + q) is

[0008]

(Equation 3)

It is represented by

In a system for performing block matching with half-pixel accuracy, first, a full search with one-pixel accuracy is performed over a wide search range to roughly estimate a motion vector, and then a motion vector is estimated.
A two-step search is widely used in which a full search with half-pixel accuracy is performed on a very narrow range (for example, a range of vertical and horizontal ± half pixels) around the motion vector. When performing the search in the second stage, a method of first obtaining the brightness value at the interpolation point in advance and then performing the search is often used. An example of the processing of this method is shown in FIG. In this example, a block of 4 pixels vertically and horizontally is used. In the reference image, a point where a pixel originally exists even if the coordinate value is an integer is represented by “◯”, and an interpolation point for which a new brightness value is obtained is represented by “x”. In addition, the pixels of the block of the original image of the current frame are represented by “□”. The motion vector obtained in the search in the first stage is (uc, vc). Matching example 101 shows the manner of matching when the motion vector is (uc, vc) in the first-stage search. The prediction error is evaluated between the overlapping "○" and "□". In Examples 102, 103, and 104, the motion vector is (uc + 1/2, vc), (uc + 1/2,
vc + 1/2), (uc-1 / 2, vc-1 / 2). In these examples, the prediction error is evaluated between the overlapping "x" and "□". As can be seen from this figure, when the search range in the second stage search is ± 1/2 pixels in the vertical and horizontal directions, the brightness value of 65 interpolated points (the number of “×” in FIG. 1) is calculated to obtain the motion. The matching process for eight vectors can be covered. At this time, all the interpolated points for which the brightness value is obtained are used for matching. When the interpolation calculation is performed on the reference image for each matching, it is necessary to perform the interpolation 4 × 4 × 8 = 128 times. The number of interpolation operations can be reduced in this way because the same interpolation point of the reference image is used multiple times.

As described above, block matching with half-pixel precision is widely used at present, but MPEG1 and M
An application that requires a higher information compression rate than PEG2 requires a more sophisticated motion compensation method. The disadvantage of block matching is that all pixels within a block must have the same motion vector. Therefore, in order to solve this problem, a motion compensation method has recently been proposed which allows adjacent pixels to have different motion vectors. The motion compensation based on the spatial transformation, which is an example of this method, will be briefly described below.

In motion compensation based on spatial transformation, the relationship between the predicted image P and the reference image R is

[0013]

[Equation 4]

It is represented by However, assuming that the image is divided into n small areas (patches), Pi represents a pixel included in the i-th patch of the image. Also, the conversion function f
i (x, y) and gi (x, y) represent the spatial correspondence between the image of the current frame and the reference image. At this time, Pi
The motion vector of the pixel (x, y) within is (x−fi (x,
y), y-gi (x, y)). by the way,
Block matching can also be interpreted as a special case of motion compensation based on spatial transformation, where the transformation function is constant. However, when the term motion compensation based on spatial transformation is used in this specification, block matching is not included therein.

The form of the conversion function is affine transformation

[0016]

(Equation 5)

(See Nakaya et al., "Basic study of motion compensation based on triangular patch", IEICE technical report, IE90-106, Hei 2-03), bilinear transformation

[0018]

(Equation 6)

Example using (GJ Sullivan and RL
Baker, "Motion compensation for video compression
using control grid interpolation ", Proc. ICASSP '9
1, M9.1, pp.2713-2716,1991-05), perspective transformation

[0020]

(Equation 7)

Example using (V. Seferdis and M. Ghanbari,
"General approach to block-matching motion estim
ation '', Optical Engineering, vol. 32, no. 7, pp.
1464-1474, 1993-07) has been reported. Where a
ij, bij, and cij are motion parameters estimated for each patch. In order to obtain the same predicted image as that obtained by the encoding device on the receiving side, the image encoding device transmits to the receiving side, as motion information, information that allows the motion parameter of the conversion function to be specified for each patch in some way. Good. For example, if the affine transformation is used as the transformation function and the patch shape is 3
It is assumed to be a prism. In this case, the six motion parameters can be reproduced on the receiving side even if the six motion parameters are directly transmitted or the motion vectors of the three vertices of the patch are transmitted (co-linear transformation). In case of using, a similar process can be performed by adopting a square patch). In the following, the case where the affine transformation is used as the transformation function will be described, but this explanation also applies when other transformations are used.
It can be applied almost as it is.

Even if the conversion function is determined, various variations can be considered for the motion compensation based on the spatial conversion, and one example thereof is shown in FIG. In this example, the motion vector is constrained to continuously change at the boundary of the patch. In the following, it is considered that the reference image 201 is used to synthesize the predicted image of the original image 202 of the current frame.
For this reason, the current frame is first divided into a plurality of polygonal patches, and the patch-divided image 208 is obtained. The vertices of the patch are called grid points, and each grid point is shared by multiple patches. For example, the patch 209 has grid points 210, 2
11 and 212, and these grid points also serve as the vertices of other patches. After dividing the image into a plurality of patches in this way, motion estimation is performed. In the example shown here, motion estimation is performed between each grid point and the reference image. As a result, the reference image 203 after motion estimation
Then each patch is transformed. For example, patch 2
09 corresponds to the modified patch 204. This is because, as a result of the motion estimation, it is estimated that the grid points 205, 206, and 207 have moved to 210, 211, and 212, respectively. The predicted image is synthesized by calculating the conversion function for each pixel in the patch and obtaining the luminance value of the corresponding point from the reference image according to equation 4. In this example, this is calculated from the motion vectors of the three vertices to the value of 6 in Equation 5.
This is realized by calculating the motion parameters and calculating Equation 5 for each pixel. As described above, the motion vector of the lattice point or the motion parameter may be transmitted as the motion information, but in this example, only one lattice point also serves as the vertices of a plurality of patches. The former is more efficient.

It has been pointed out that the motion estimation based on the matching is effective in the motion compensation based on the spatial transformation as well as the block matching. An example of a motion compensation algorithm based on matching is shown below. This method is called hexagonal matching and is effective when the motion vector continuously changes at the patch boundary as in the above example. This method is composed of the following two processes.

(1) Rough motion estimation of grid points by block matching (2) Correction of motion vector by correction algorithm In the process of (1), block matching is performed on a block (any size) including grid points. Then, the motion vector of this block is used as the rough motion vector of the grid point. The purpose of this processing is merely to obtain a rough motion vector of the grid point, and it is not always necessary to use block matching. The state of the process (2) is shown in FIG. This figure shows a part of the patches and grid points in the reference image (corresponding to the image 203 in FIG. 2). Therefore, changing the position of the grid point in this figure means changing the motion vector of the grid point. When the motion vector of the grid point 301 is modified, first, the motion vectors of the grid points 303 to 308, which correspond to the vertices of the polygon 302 formed by all patches related to this grid point, are fixed. In this way, when the motion vector of the grid point 301 is changed within a certain range (for example, the grid point 301 is moved to the position of the grid point 309), the prediction error in the patch included in the polygon 302 also changes. Then, the motion vector that minimizes the prediction error in the polygon 302 within the search range is the grid point 301.
Registered as the corrected motion vector of In this way, the correction of the grid point 301 is completed, and after moving to another grid point, the same correction is continued. The prediction error can be further reduced by once making corrections to all grid points and then making corrections again from the first grid point. It has been reported that a suitable number of repetitions is 2 to 3 times.

A typical search range in the correction algorithm is ± 3 pixels vertically and horizontally. In this case, for one grid point, 49 times of matching are performed within the polygon 302 by one modification. On the other hand, since one patch participates in the correction algorithm of three grid points, the prediction error is evaluated 147 times in total for each pixel in the patch. Further, if this correction is repeatedly performed, the number of error evaluations will increase each time. As a result, each time the error evaluation is performed, the calculation of the interpolation is performed on the target pixel, resulting in a huge amount of calculation.

The problem of the interpolation operation in the motion compensation based on the spatial transformation is troublesome because there is an essential difference in the following points as compared with the similar problem in the block matching with half pixel accuracy. In motion compensation based on spatial transformation,
Even if the horizontal and vertical components of the motion vector of the grid point are 1/2
Even if it is limited to an integral multiple of, the horizontal / vertical components of the motion vector of each pixel are similarly not an integral multiple of 1/2.
Further, in general, since the component after the decimal point of the motion vector of each pixel can take an arbitrary value, it is rather rare that the luminance value of the same interpolation point of the reference image is used multiple times in the matching process.

[0027]

When motion estimation based on matching is performed in motion compensation based on spatial transformation, there arises a problem that the amount of calculation required for interpolation of luminance values becomes enormous. It is an object of the present invention to reduce the number of calculations for brightness value interpolation and realize motion estimation processing with a small amount of calculation.

[0028]

Before the motion estimation process, the x and y coordinates in the reference image are 1 / m 1 and 1 respectively.
A high-definition reference image is prepared by interpolating the luminance value at a point that is an integer multiple of / m2 (m1 and m2 are positive integers). Therefore, in the high-definition reference image, pixels are present at points where the x coordinate and the y coordinate are integer multiples of 1 / m1 and 1 / m2, respectively. When the luminance value of the reference image at a position where the coordinate value is not an integer is required in the motion estimation process, the luminance value of the pixel existing at the position closest to this coordinate in the high-definition reference image should be approximated. Thus, the purpose of reducing the number of interpolation operations is achieved.

[0029]

In the processing for creating the high-definition reference image, it is necessary to perform the interpolation calculation m1 × m2-1 times for each original image. However, once this high-definition processing is completed, there is no need to perform further interpolation calculations in the motion estimation processing. In the example of the motion compensation based on the spatial conversion described in “Prior Art”, the motion estimation process requires 147 or more interpolation operations per pixel. Therefore, if m1 = m2 = 2, the number of interpolation calculations can be reduced to about 1/50, which leads to a significant reduction in the amount of calculations.

[0030]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS As a first embodiment, a method of performing motion estimation processing after refining the entire reference image is shown. First, a high definition reference image R'is created. Assuming that co-linear interpolation (Equation 3) is used for the interpolation method of the luminance value, R'is represented by the following expression.

[0031]

[Equation 8]

However, s and t are integers, and 0 ≦ s
It is assumed that <m1, 0 ≦ t <m2. In R ', x,
It is assumed that a pixel exists at a point where y, s, and t are all integers. The point of s = t = 0 originally corresponds to the pixel existing in the reference image, and the luminance values of the other points are obtained by interpolation. In the following, for simplicity, m
An example will be given for the case where 1 = m2 = m (m is a positive integer).

FIG. 4 shows an example of an image coding apparatus utilizing a high definition reference image. The arrows in the figure represent the flow of data, and address signals and the like are omitted. In this apparatus, the motion estimation unit 401 is in charge of motion estimation processing. The reference image 404 is processed by the reference image high definition circuit 405 that performs high definition processing, and then the high definition reference image 40 is processed.
6 is stored in the frame memory 407 and the approximate brightness value information 408 is provided to the matching processing circuit 409. On the other hand, the original image 402 of the current frame is also stored in the frame memory 403.
And is used for motion estimation in the matching processing circuit 409. The motion information 415 output from the matching processing circuit is transmitted to the receiving side, but is also used in the encoding device for combining predicted images in the predicted image combining circuit 410. The difference between the combined predicted image and the original image 411 of the current frame is obtained by the subtractor 412, and the prediction error 41
3 is encoded by the prediction error encoder 414 and transmitted as prediction error information 416. In the conventional method, the calculation of the conversion function, the interpolation processing, and the evaluation of the prediction error are all performed by the matching circuit, but in the present embodiment, the interpolation processing is performed by the reference image high-definition circuit in advance. The calculation amount is reduced by. Moreover, by using the high-definition reference image, the calculation accuracy required for the conversion function can be reduced, and the processing can be further simplified.
This is because the result of motion estimation is not affected as long as the pixels used for approximation do not differ in the high-definition reference image when an error occurs in the calculation of the conversion function. Note that not all the pixels of the high-definition reference image whose luminance value is obtained by interpolation here are used in the matching process. This point is different from the example of block matching taken up in "Prior Art".

In FIG. 5, it is assumed that m = 2 and the brightness value is interpolated by 1
An example of the reference image high-definition circuit 405 when the next interpolation is used is shown. Also in this figure, the arrows show the flow of data,
The same reference numbers as in FIG. 4 refer to the same things. It is assumed that the input reference image signal 404 gives pixel luminance values from left to right line by line from top to bottom. This signal is supplied to two pixel delay circuits 502 and 503 and one line delay circuit 501.
The luminance values 504 to 507 of the four pixels vertically and horizontally adjacent to each other can be obtained by adding to the circuit constituted by the above. For each of these brightness values, an integrator 50
8 to 511 are used to multiply the weighting coefficient according to the interpolation position, and the result is added to the adders 512 to 514. By adding this result to the adder 515 and the shift register 516, an operation of dividing by 4 and then rounding off after the decimal point is realized. As a result of the above processing, the brightness values 517 to 520 for four pixels of the high-definition reference image R ′ can be obtained as the output 406.

FIG. 6 shows an example of a circuit in the matching processing circuit 409 which obtains an approximate value of the brightness value at the interpolation point of the reference image by using the high definition reference image R '. The same reference numbers as in FIG. 4 refer to the same things. Here, it is assumed that the coordinate values 601 and 602 in binary fixed point representation are given by calculating the conversion function. Also,
Assuming that m is 2 as in FIG. 5, the high-definition reference image R ′
Are stored in the frame memory 407. The coordinate values 601 and 602 pass through an adder 603 that adds 1/4 and a circuit 604 that rounds down the second decimal place and the second decimal place is rounded off and converted into an integral multiple of 1/2. . The resulting coordinate values 605,
Reference numeral 606 corresponds to the coordinate value of the point where the pixel exists in the high definition reference image R ′. This coordinate value is converted into an address of the frame memory 407 by the coordinate-address conversion circuit 607 to obtain the target approximate brightness value information 408. In this example, the components below the third digit after the decimal point of the conversion function are not used at all. Therefore, the calculation error in the range that does not affect the second digit or more after the decimal point of the conversion function does not affect the motion estimation result. This is because, as described above, the use of the high-definition reference image reduces the calculation accuracy required for the conversion function.

While the number of interpolation operations is reduced in the above-mentioned embodiment, a memory for storing an image having a size four times as large as the reference image is required to store the high definition reference image.
Therefore, an example of a method in which the required memory capacity is small, although the number of interpolation operations is larger than that in the above example, will be shown. In this method, the original image of the current frame and the required portion of the reference image are captured little by little, and the reference image is refined and used for motion estimation. It is assumed that the distance between adjacent pixels is 1 in the horizontal and vertical directions for both the original image and the reference image of the current frame. Note that, here, assuming that hexagonal matching is used in the motion estimation method, a circuit that executes a correction process in hexagonal matching will be mainly described. Rough motion estimation of grid points, which is the other process of hexagonal matching,
As described in "Prior Art", block matching is performed on a block including a grid point.

FIG. 7 shows the positions of grid points 703 to 711 in a part of the original image of the current frame. The distance between grid points is Ng vertically and horizontally, and the search range of the motion vector for each grid point is horizontal.
If it is ± Ns in the vertical direction, the reference image is 2Ng in length and width.
+ 2Ns range 701 and vertical and horizontal 2 of the original image of the current frame
If the pixels included in the Ng range 702 (hatched range) are used, the correction processing of hexagonal matching with respect to the grid point 703 can be performed (actually, a narrower range may be used, but in order to simplify the processing. To use a square area). Therefore, the device that performs the correction process can execute the subsequent processes independently of the external frame memory by reading the brightness values of the pixels included in these ranges in advance. In this case, the grid point 70
If the correction processing of the grid point 708 is performed before the correction processing of No. 3, some of the pixels in the ranges 701 and 702 have already been read by the apparatus that performs the correction processing. Therefore, at this time, as shown in FIG. 8, the range 801 of the reference image and the range 802 of the original image of the current frame.
It suffices to add and read only (in FIG. 8, the same reference numerals as those in FIG. 7 refer to the same elements). At the time of this additional reading, part of the pixel information used for the motion estimation processing of the grid point 708 is not necessary, so the information of the ranges 801 and 802 may be written in the memory in which this information was stored. In this way, the processing can be simplified by newly reading only necessary information each time the grid point for motion estimation moves from left to right.

FIG. 9 shows a motion estimator 9 of the apparatus for correcting hexagonal matching according to the method shown in FIGS. 7 and 8.
An example of No. 09 is shown. In this figure, the arrows indicate the flow of data, and the same reference numerals as in FIG. 4 indicate the same things. The motion estimation unit 909 has the same configuration as the motion estimation unit 401 of FIG. The original image 402 and the reference image 404 of the input current frame are respectively stored in the frame memory 90.
It is stored in 1 and 903. On the other hand, first, rough motion estimation of grid points is executed by the circuit 902, and information on the coordinates of grid points in the reference image is stored in the grid point coordinate memory 904 in accordance with the obtained motion vector. Then, the correction processing unit 905 performs correction processing in hexagonal matching. In the following, following the example of FIG.
The correction process for the grid point 703 when the process for 8 is performed will be described. The correction processing unit 905 includes a high definition circuit 907 and a matching processing circuit 906. First, the high definition circuit 907 reads out luminance value information of pixels in a newly required range (range 801 in the example of FIG. 8) from the frame memory 903 in which reference images are stored.
Interpolation processing is performed on this information to create a high-definition reference image in the range required for motion estimation, and this is provided to the matching processing circuit 906. Similarly, the matching processing circuit reads the luminance value information of a newly required range (range 802 in the example of FIG. 8) from the frame memory 901 of the original image of the current frame. The matching processing circuit has a frame memory that stores the high-definition reference image and the original image of the current frame in the range necessary for the correction processing by itself, and performs processing using this memory. The matching processing circuit is newly required in the coordinate information of the grid point in the reference image from the grid point coordinate memory 904 (in the example of FIG. 8,
Coordinate information of grid points 704, 706, and 711. This is because the coordinate information of the grid points 707, 708, 710 is used in the previous processing. ) Is read and correction processing of hexagonal matching is performed. In accordance with this processing result, the coordinates of the grid point in the corrected reference image (the coordinates of the grid point 703 in the example of FIG. 8) are written in the grid point coordinate memory 904. With the above, the correction of the grid point 703 is completed, and the correction processing unit moves to the correction processing of the grid point 704 in FIG. When all the correction processing is completed, the information stored in the grid point coordinate memory 904 is converted into a motion vector for each grid point in the motion vector calculation circuit 908 and output as motion information 415. Further, in order to calculate the prediction error of the predicted image, the information 411 on the original image of the current frame is also output.

FIG. 10 shows an example in which parallel processing is introduced in the processing of FIG. The same reference numbers as in FIG. 9 refer to the same things. In this example, there are a plurality of correction processing units that perform correction processing in hexagonal matching and share the processing.
In order to read the luminance value information from the frame memories 901 and 903 which store the reference image and the original image of the current frame, a common data bus 1001 and address bus 1002 are required.
To use. In addition, information is read from the grid point coordinate memory 904 that stores the coordinates of the grid points in the reference image,
When writing information, a common data bus 1005 and address bus 1004 are used. Through these buses,
A circuit 902 that roughly estimates the movement of the lattice points and circuits 905 and 1003 that corrects the hexagonal matching transfer information. The correction processing units 905 and 1003 have the same configuration. The correction processing can be executed at a higher speed by adding a correction processing unit having the same configuration. Since the correction processing unit can perform the processing almost independently except when reading the brightness value information and reading and writing the grid point coordinate information, it is possible to perform the processing in parallel while avoiding the collision of access to the memory. .

In the embodiment shown in FIGS. 8, 9 and 10, about one pixel (2 + 2Ns) of the reference image is subjected to the correction process.
/Ng).times.(m.times.m-1) interpolation operations are required, which is about (2 + 2Ns / Ng) times as many as the embodiment shown in FIG. However, since a memory that stores the entire high-definition reference image is not required, the memory capacity required as a whole can be reduced.

Considering the ease of multiplication and division in the circuit, m is preferably a power of 2. If m is reduced, the circuit scale can be reduced. However, on the other hand, the approximation accuracy of the coordinates (motion vector) in the motion estimation is deteriorated, and the magnitude relationship of the prediction error is easily reversed in the calculation of Expression 2, so that the motion estimation result is distorted and the prediction characteristic is deteriorated. On the other hand, when m is increased, the opposite phenomenon occurs. Considering the circuit scale, the value of m is preferably 4 or less. Further, considering the prediction characteristics, the value of m is preferably 2 or more. Therefore, considering the balance between the two, 2 and 4 are suitable as the value of m. An appropriate value of m may be selected according to the allowable prediction error and the circuit scale.

Performing motion estimation using a high-definition reference image having a pixel density of m times the length and width means that the values of the conversion functions fi (x, y) and gi (x, y) in Equation 3 are 1 / m. It means that it is limited to an integral multiple (the minimum unit of the conversion function is 1 / m of the distance between adjacent pixels). However, this is just a limitation in motion estimation, and it is not necessary to comply with this limitation also in combining predicted images. On the other hand, in motion compensation based on spatial transformation, a mismatch between the predicted images on the transmission side and the reception side (due to the difference in the calculation accuracy of the conversion function on the transmission side and the reception side, the prediction images synthesized by the two sides do not match. In order to prevent such a problem, it is necessary to set some standard for the calculation accuracy of the conversion function when the predicted images are combined. As one of the methods of setting this reference, there is a method of providing a minimum unit in a conversion function when synthesizing a predicted image as in the case of motion estimation. However, for the reason described above, the minimum unit at this time is not necessarily the same 1 / m as the minimum unit in motion estimation. Generally, even if the motion estimation is completed and the motion parameter is fixed, the prediction error decreases if the calculation accuracy of the conversion function in the synthesis of the predicted images is increased. Therefore, the prediction error can be reduced by making the minimum unit of the motion vector smaller when the predicted images are combined than when the motion estimation is performed.
When the prediction image is combined, the brightness value interpolation operation is performed only once for each pixel, so even if this operation is somewhat complicated, it does not affect the overall processing amount as much as in the case of motion estimation. .

Obviously, the following modifications are included in the present invention.

(1) As a function used for interpolation of luminance values,
In this specification, colinear interpolation is taken up, but other functions may be used. When the function becomes complicated, the effect of reducing the number of interpolation operations becomes large.

(2) Although the affine transformation is mainly used in the present specification as the type of the transformation function, other transformation functions may be used. The present invention is effective as long as the pixels in the same patch do not have to follow a common motion vector, and the vertical and horizontal components of the pixel motion vector can take values other than integer multiples of the distance between adjacent pixels.

(3) The shape of the patch may be any shape as long as it specifies a set of pixels, and is not limited to the triangular shape described in this specification.

(4) Regarding motion compensation based on spatial transformation,
In this specification, the method in which the motion vector continuously changes at the boundary of the patch is taken up. However, a method of allowing discontinuity at a patch boundary, such as a method of directly transmitting a motion parameter for each patch, may be used.

(5) As the motion estimation algorithm, block matching and hexagonal matching are taken up in this specification, but this may be a system based on other matching. The present invention is effective as long as the prediction error is evaluated many times.

(6) In the motion compensation based on the spatial transformation, the motion information transmitted as in the example taken in this specification may not be the motion vector of the lattice point. The motion information may be any information that specifies a conversion function for each patch, as in the example taken in item (4) above.

(7) In the embodiment, the case where m1 = m2 is described, but both may be different.

(8) In the present specification, the method of fixing the patch structure of the current frame and deforming the patch of the reference image has been described. Conversely, the patch structure of the reference image is fixed and the patch of the current frame is deformed. It may be a method of allowing.

(9) Although the number of reference images used to synthesize one predicted image is one in the present specification, a method using a plurality of reference images may be used.

[0053]

According to the present invention, there is no constraint that all pixels belonging to the same patch have a common motion vector, and the horizontal / vertical components of the pixel motion vector have values other than integer multiples of the distance between adjacent pixels. In the possible motion compensation type motion estimation processing, it is possible to reduce the number of times the calculation of the interpolation of the luminance value is performed.

[Brief description of drawings]

FIG. 1 is a diagram showing an example of a two-step search process in block matching with half-pixel accuracy.

FIG. 2 is a diagram showing an example of motion compensation processing based on space conversion.

FIG. 3 is a diagram showing a process called a hexagonal matching process as an example of a motion estimation process in motion compensation based on space conversion.

FIG. 4 is a diagram showing an example of an image encoding device that utilizes a high-definition reference image.

FIG. 5 is a diagram showing an example of a circuit for increasing the definition of a reference image by interpolating a brightness value.

FIG. 6 is a diagram showing an example of a circuit that obtains a luminance value in a high-definition reference image from a calculation result of a conversion function.

FIG. 7 is a diagram showing a range of pixels used in correction processing of hexagonal matching.

FIG. 8 is a diagram showing a range of pixels that is newly required when a correction process is performed subsequent to an adjacent grid point in the correction process of hexagonal matching.

FIG. 9 is a diagram showing a method in which a reference image is refined and used for motion estimation while gradually capturing necessary portions of an original image and a reference image of a current frame.

FIG. 10 is a diagram showing a case where parallel processing is introduced into a method used for motion estimation while gradually capturing necessary portions of an original image and a reference image of a current frame.

[Explanation of symbols]

101, 102, 103, 104 ... Examples of matching processing in block matching with half pixel accuracy, 201 ... Reference image, 202 ... Original image of current frame, 203 ... Patch and grid point of reference image after motion estimation, 204, 209 ... Patch, 205-207, 210-212, 301, 30
3 to 309, 703 to 711 ... Lattice points, 208 ... Patches and lattice points of the original image of the current frame, 302 ... Polygons to be matched in the correction process, 401, 909 ...
Motion estimation units 402, 411 ... Original image of current frame, 4
03, 407, 901, 903 ... Frame memory, 40
4 ... Reference image, 405 ... Reference image high-definition circuit, 406
... High-definition reference image, 408 ... Approximate brightness value, 409 ... Matching processing circuit, 410 ... Predicted image synthesizing circuit, 412 ...
Subtractor, 413 ... Prediction error, 414 ... Prediction error encoder, 415 ... Motion information, 416 ... Prediction error information, 501
... line delay circuits, 502, 503 ... pixel delay circuits, 5
04 to 507 ... Luminance values of pixels of the reference image, 508 to 51
1 ... Multiplier, 512-515, 603 ... Adder, 516
... shift register, 517 to 520 ... luminance value of pixel of high-definition reference image, 601, 602 ... fixed point representation of coordinate value in reference image (calculation result of conversion function), 604 ...
Rounding circuit below the second decimal place (binary representation), 60
5, 606 ... Coordinate values in high-definition reference image, 607 ...
Coordinate-address conversion circuit, 701 ... Range used for correction in reference image, 702 ... Range used for correction in original image of current frame, 801 ... Range added in reference image, 802 ... Original image of current frame , 902 ... Circuit for performing rough motion estimation of grid points in hexagonal matching, 904 ... Coordinate memory of grid points in reference image, 905, 1003 ... Hexagonal matching correction processing unit, 906 ... Matching processing circuit,
Reference numeral 907 ... High-definition circuit for a part of the reference image, 908 ... Motion vector arithmetic circuit, 1001 ... Luminance value information data bus, 1002 ... Luminance value information address bus, 1004 ...
Address bus for coordinate information of grid points, 1005 ... Data bus for coordinate information of grid points.

Claims (7)

[Claims] 1. Motion compensation in which all pixels belonging to the same patch are not constrained to have a common motion vector, and the horizontal and vertical components of the pixel motion vector can take values other than integer multiples of the distance between adjacent pixels. By means of executing the method and by interpolating the luminance value of a point where no pixel exists in the reference image, the pixel density is multiplied by m1 and m2 in the horizontal and vertical directions (m1 and m2 are positive integers), respectively. A means for synthesizing a high-definition reference image and a memory for storing the high-definition reference image are provided, and the brightness value at a point where no pixel exists in the reference image is closest to this point in the high-definition reference image. An image coding apparatus characterized by approximating with a luminance value of a pixel located at. 2. Motion compensation in which all pixels belonging to the same patch are not constrained to have a common motion vector, and the horizontal / vertical components of the pixel motion vector can take values other than integer multiples of the distance between adjacent pixels. The pixel density in the horizontal and vertical directions is m1 and m2 times (m1 and m2 respectively) by interpolating the brightness value of a point where no pixel exists by using the means for executing the method and a part of the reference image as a target. A part of the high-definition reference image and a memory for storing a part of the high-definition reference image, and a pixel does not exist in the part of the reference image. The brightness value at
An image coding apparatus characterized by approximating with a luminance value of a pixel located closest to this point in a part of the high definition reference image. 3. The horizontal and vertical components of a motion vector at each pixel when a predicted image is combined are limited to integer multiples of 1 / d1 and 1 / d2 (d1 and d2 are positive integers) of the distance between adjacent pixels, respectively. The image coding apparatus according to claim 1 or 2, further comprising: means for satisfying m1≤d1 and m2≤d2. 4. The image coding apparatus according to claim 1, wherein the values of m1 and m2 are integers of 2 or more, respectively. 5. The values of m1 and m2 are 2 to the power w1, respectively.
The image coding device according to claim 1, wherein the image coding device is a power of 2 (w1 and w2 are positive integers). 6. The image coding apparatus according to claim 1, 2 or 3, wherein the values of m1 and m2 are two. 7. The image coding apparatus according to claim 1, wherein the value of m1 and m2 is 4.