JP2011182181A - Motion vector arithmetic apparatus by block, or method for the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform block matching for a block whose block boundary line is shifted by a half pixel from the boundary between pixels in an intermediate frame. <P>SOLUTION: An extraction unit 7 also extracts a block matching candidate obtained by shifting a block of a first reference block or a second reference block by one pixel so that the block boundary line is located on the boundary between the pixels for blocks of the first and second reference frames, and the block boundary line is set at a position shifted by a half pixel from the boundary between the pixels for a specific block in the intermediate frame in determination of the block matching candidate. A determination unit 9 performs the block matching to the extracted block matching candidate, and determines combination in which difference becomes minimum as a motion vector of the specific block in the intermediate frame. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a motion vector computing device for each block, and more particularly to improvement in accuracy.

  Patent Document 1 discloses a symmetric motion estimation method. As shown in FIG. 1, the frame Q to be generated is divided into grid blocks, and a specific block B0 is located at a block B1 at a position that becomes a geometric object from the preceding and following images P1, P2 with B0 as the origin. , B2 is extracted, and the motion vector having the smallest difference is estimated as the motion vector of the block B0.

  Details of block matching in the symmetric motion estimation method will be described with reference to FIG. FIG. 2 shows a case where the block size is 4 * 4 pixels and the search range is 12 * 12 pixels. In this case, the blocks of the previous frame (0, 0) and the subsequent frame (9, 9) are compared. Next, the blocks of the previous frame (1, 0) and the subsequent frame (8, 9) obtained by shifting the previous frame and the subsequent frame by one are compared. Thereafter, similarly, block matching is performed by shifting both the previous frame and the subsequent frame by one pixel.

  In such symmetric motion estimation, when performing block matching with ½ pixel accuracy, as shown in FIG. 3, block data shifted by ½ pixel is generated for each of the preceding and following frames, and in the intermediate frame, Block matching is sequentially performed at positions where there is no deviation.

No. 2528103

  However, the method for generating block data shifted by 1/2 pixel for each of the preceding and following frames as described above has the following problems. Block data shifted by 1/2 pixel in the previous and next frames must be generated. Therefore, not only a circuit that extracts a region that is slightly larger than a normal block is required, but also a generation unit that generates block data shifted by 1/2 pixel from the extraction result. Furthermore, block data shifted by 1/2 pixel is usually generated by calculating the average of surrounding pixels. Therefore, image blur occurs accordingly.

  An object of the present invention is to provide a block-by-block motion vector calculation apparatus or method that solves the above-described problems, is simple in configuration, and can perform block matching with little image blur.

(1) A block-specific motion vector calculation device according to the present invention calculates a local motion vector, which is a block-specific motion vector for these intermediate frames, from a first reference frame and a second reference frame. A device, A) image storage means for storing images of the first reference frame and the second reference frame, B) search range determination means for determining a search range for the first reference frame and the second reference frame, C) The image within the search range of the determined first reference frame and the second reference frame is a block composed of predetermined pixels, and the first reference frame and the second reference are centered on a specific block of the intermediate frame. A block matching candidate is a block in which the blocks of the frame are geometrically symmetrical. Extraction means for sequentially extracting, and D) determining means for performing block matching on the extracted block matching candidates and determining a combination having the smallest difference as a motion vector of the specific block in the intermediate frame, E) For the blocks of the first and second reference frames, the extracting means has a block boundary line located at the boundary between the pixel and the pixel, and for the specific block in the intermediate frame, the block boundary line is the pixel and the pixel. Block matching in which the block of the first reference frame and / or the block of the second reference frame is shifted by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by 1/2 ⁿ pixels from the boundary of Candidates are also extracted.

Thus, for the blocks of the first and second reference frames, the block boundary line is located at the pixel-to-pixel boundary, and for the specific block in the intermediate frame, the block boundary line extends from the pixel-to-pixel boundary. Block matching candidates obtained by shifting the block of the first reference frame and / or the block of the second reference frame by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by 1/2 ⁿ pixels are also obtained. By extracting, block matching can be performed at a position where the block boundary line is shifted by 1/2 ⁿ pixels from the boundary between pixels in the intermediate frame. Therefore, highly accurate block matching is possible.

(2) In the block-by-block motion vector arithmetic apparatus according to the present invention, the extraction means shifts a pixel that generates 2 * (1/2 ⁿ ) -precision block data for the blocks of the first and second reference frames. Block data generating means is included. Where n is an integer. Therefore, block matching with (1/2 ^{n + 1} ) pixel accuracy is enabled by the (1/2 ⁿ ) pixel accuracy generation circuit.

  (3) In the motion vector computing unit for each block according to the present invention, the n is 2, and the pixel-shifted block data generating means generates block data with half-pixel accuracy for the blocks of the first and second reference frames. Generate. Thereby, block matching with 1/4 pixel accuracy is possible.

  (4) In the block-specific motion vector calculation apparatus according to the present invention, the n is 3, and the pixel-shifted block data generation means is configured such that a block boundary line is a pixel and a pixel for the blocks of the first and second reference frames. Block data shifted by 1/4 pixel, 1/2 pixel, and 3/4 pixel from the boundary is generated. This enables block matching with 1/8 pixel accuracy.

(5) In the block-specific motion vector calculation method according to the present invention, a block-specific motion vector that calculates a local motion vector that is a block-specific motion vector for these intermediate frames from the first reference frame and the second reference frame. In the calculation method, when images of the first reference frame and the second reference frame are given, search ranges are determined for the first reference frame and the second reference frame, and the determined first reference frame and second reference frame are determined. Positions of the first reference frame and the second reference frame that are geometrically symmetric with respect to a specific block of the intermediate frame, with an image within a search range of two reference frames as a block composed of predetermined pixels The relevant blocks are sequentially extracted as block matching candidates, and the extracted In a block-based motion vector calculation method for performing block matching on a lock matching candidate and determining a combination that minimizes a difference as a motion vector of the specific block in the intermediate frame, the first and first 2 For the block of the reference frame, the block boundary line is positioned at the boundary between the pixels, and for the specific block in the intermediate frame, the block boundary line is shifted by 1/2 ⁿ pixel from the boundary between the pixels. As set, block matching candidates are also extracted by shifting the block of the first reference frame and / or the block of the second reference frame by 2 * (1/2 ⁿ ) pixels.

Thus, for the blocks of the first and second reference frames, the block boundary line is located at the pixel-to-pixel boundary, and for the specific block in the intermediate frame, the block boundary line extends from the pixel-to-pixel boundary. Block matching candidates obtained by shifting the block of the first reference frame and / or the block of the second reference frame by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by 1/2 ⁿ pixels are also obtained. By extracting, block matching can be performed at a position where the block boundary line is shifted by 1/2 ⁿ pixels from the boundary between pixels in the intermediate frame. Therefore, highly accurate block matching is possible.

  (6) A block-specific motion vector calculation apparatus according to the present invention calculates a block-specific motion vector calculation that calculates a local motion vector that is a block-specific motion vector for these intermediate frames from a first reference frame and a second reference frame. 1) image storage means for storing images of the first reference frame and the second reference frame; 2) search range determination means for determining a search range for the first reference frame and the second reference frame; 3) Means for extracting a block composed of predetermined pixels from the determined image within the search range of the first reference frame and the second reference frame, wherein the first block is centered on a specific block of the intermediate frame. When a virtual reference frame having a geometrically symmetrical positional relationship is defined, the intermediate frame and the temporary frame are defined. When the block of the second reference frame located on an extension line that is n times away from the temporal relationship with the reference frame is shifted by one pixel block with respect to the first reference frame, the second reference frame is equivalent to n pixels. Extraction means for moving blocks and extracting them as block matching candidates 4) Perform block matching on the extracted block matching candidates and determine the combination that minimizes the difference as the motion vector of the specific block in the intermediate frame And 5) the extraction means is configured to set the block boundary line in the intermediate frame at a position shifted from a pixel-to-pixel boundary or the block of the first reference frame or the second reference. Blocks with shifted frames are extracted as block matching candidates.

  Therefore, a block in which the block boundary line in the intermediate frame is shifted from the boundary between pixels is also extracted. Thereby, block matching at a position deviated from the boundary between pixels can be performed. Therefore, highly accurate block matching is possible.

  (7) In the block-specific motion vector arithmetic apparatus according to the present invention, the extraction means includes pixel-shifted block data generation means for generating block data with decimal x precision, wherein the decimal x is arbitrary. is there. Therefore, in the intermediate frame, block matching shifted by an amount smaller than the decimal x precision is possible.

  (8) In the block-by-block motion vector computing device according to the present invention, the extraction means is configured such that the block of the first reference frame or the timing of shifting the second reference frame is not synchronized, or the second reference The block of the first reference frame is set such that the block boundary line in the intermediate frame is set at a position shifted from the boundary between the pixels by extracting a block obtained by shifting the block of the frame by one pixel as a block matching candidate. Alternatively, a block in which the second reference frame is shifted is extracted as a block matching candidate.

  Therefore, a block in which the block boundary line in the intermediate frame is shifted from the boundary between pixels is also extracted. Thereby, block matching at a position deviated from the boundary between pixels can be performed. Therefore, highly accurate block matching is possible.

  (9) In the block-specific motion vector calculation method according to the present invention, a block-specific motion vector that calculates a local motion vector that is a block-specific motion vector for these intermediate frames from the first reference frame and the second reference frame. 1) a step of determining a search range for the first reference frame and the second reference frame when the images of the first reference frame and the second reference frame are given; and 2) the determined first Extracting a block composed of predetermined pixels from an image within a search range of one reference frame and a second reference frame, wherein the first reference frame is geometrically centered on a specific block of the intermediate frame; The intermediate frame and the virtual reference when a virtual reference frame having a symmetric positional relationship is defined When the block of the second reference frame located on the extension line separated by n times the temporal relationship with the frame is shifted by one pixel block for the first reference frame, the block for n pixels for the second reference frame 3) Extraction step for extracting as a block matching candidate, 3) Perform block matching on the extracted block matching candidate, and determine a combination that minimizes the difference as a motion vector of the specific block in the intermediate frame 4) In the extraction step, one of the block of the first reference frame or the second reference frame is set such that a block boundary line in the intermediate frame is set at a position shifted from a pixel-to-pixel boundary. Blocks that are shifted only as block matching candidates Extract.

  Therefore, a block in which the block boundary line in the intermediate frame is shifted from the boundary between pixels is also extracted. Thereby, block matching at a position deviated from the boundary between pixels can be performed. Therefore, highly accurate block matching is possible.

  In this specification, the “intermediate frame” means a frame located between the first reference frame and the second reference frame.

It is a figure explaining the conventional symmetrical motion estimation. It is a figure explaining the conventional symmetrical motion estimation of integer precision. It is a figure explaining the conventional symmetrical motion estimation of half pixel precision. It is a functional block diagram of the motion vector computing device 1 by block. In the present invention, an outline of a motion vector obtained is shown. It is a figure which shows the detailed hardware constitutions of the motion vector calculating apparatus 1 according to block. It is a figure which shows the pixel coordinate of the search range of 8 * 8 pixel. It is a figure which shows the relationship between a shift register and PE array. It is a time chart of shift processing and PE processing for the previous frame. It is a time chart of the shift process and PE process about a back frame. It is a figure explaining symmetrical type motion estimation of half pixel accuracy in the present invention. It is a figure explaining symmetrical type motion estimation of half pixel accuracy in the present invention. It is a figure which shows the detailed hardware constitutions of the symmetrical motion estimation apparatus of 1/4 pixel precision in this invention. It is a figure which shows the positional relationship with the intermediate | middle frame in other embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

(1. Function block diagram)
As shown in FIG. 4, the block-specific motion vector calculation apparatus 1 calculates a local motion vector, which is a block-specific motion vector for these intermediate frames, from the first reference frame and the second reference frame. An arithmetic unit, which includes an image storage unit 3, a search range determination unit 5, an extraction unit 7, and a determination unit 9.

The image storage means 3 stores images of the first reference frame and the second reference frame. Search range determining means 5 determines a search range for the first reference frame and the second reference frame. The extracting means 7 sets the image within the search range of the determined first reference frame and second reference frame as a block composed of predetermined pixels, and centers the specific block of the intermediate frame on the first reference frame and the second reference frame. Blocks in which the blocks of the two reference frames have a geometrically symmetrical positional relationship are sequentially extracted as block matching candidates. In the block matching candidate determination, the extracting unit 7 determines that a block boundary line is located at a pixel-to-pixel boundary for the blocks of the first and second reference frames, and a block boundary for the specific block in the intermediate frame. The block of the first reference frame and / or the block of the second reference frame is set to 2 * (1/2 ⁿ ) pixels so that the line is set at a position shifted by 1/2 ⁿ pixels from the boundary between the pixels. Block matching candidates that have been shifted are also extracted. The determining unit 9 performs block matching on the extracted block matching candidates, and determines a combination that minimizes the difference as a motion vector of the specific block in the intermediate frame.

In this way, the extraction unit 7 not only shifts the block of the first reference frame and the second reference frame at the same time, but also extracts candidates that are shifted only one of them as block matching candidates, so that Thus, block matching is possible for a block whose block boundary line is shifted by ½ ⁿ pixels from the boundary between pixels.

Specifically, for example, as shown in FIG. 5, when the shift process is performed symmetrically, only one of the block of the first reference frame or the second reference frame is shifted, and thereby, in the intermediate frame, Block matching is possible for blocks where the block boundary is shifted by half a pixel from the pixel boundary (2. Configuration of each part)
A detailed block diagram of the block-by-block motion vector computing device 1 is shown in FIG. The block-by-block motion vector computing device 1 includes a control unit 13, shift register arrays 15 and 25, a PE array 24, a minimum value detection circuit 27, an SRAM 31, and a DRAM 33. Synchronous processing is performed by applying a common clock to each unit other than the DRAM 33. The DRAM 33 is synchronized with another clock.

  The control unit 13 performs shift control as will be described later on the shift register arrays 15 and 25. Further, the image data is read from the DRAM 33 and stored in the SRAM 31.

  The DRAM 33 stores one frame of image data of the previous frame and the subsequent frame. The SRAM 31 stores image data for the search range of the previous frame and the subsequent frame from the DRAM 33 in a FIFO manner. When the image data for one block in the search range is given to the previous frame from the SRAM 31, the register array 15 stores and holds the image data and gives it to the PE array 24. When the image data for one block in the search range is given to the subsequent frame, the shift register 25 stores and holds the image data and gives it to the PE array 24.

  The shift register arrays 15 and 25 shift one block of image data pixel by pixel. Data lacking due to the shift processing is provided from the SRAM 31. When there is a read request from the SRAM 31, the DRAM 33 provides data.

  The PE array 24 calculates the luminance difference of each pixel given from the shift registers 15 and 25, and obtains the sum total for one block. When data is given from the PE array 24, the minimum value detection circuit 25 compares the stored value with the stored value and holds the data if the value is small.

(3. About motion vector calculation processing)
The shift process shown in FIG. 6 will be described in detail. Hereinafter, a case where the search range is 8 * 8 pixels and one block is 4 * 4 pixels will be described as an example.

FIG. 7 shows coordinates specifying each pixel in the search range of 8 * 8 pixels. FIG. 8 shows a connection relationship between a shift register and a PE array in a 2-parallel type in which a search range is 8 * 8 pixels and block matching of integer precision and half-pixel precision is performed simultaneously. The shift register for the previous frame is composed of shift registers SRF00 to SRF07, SRF10 to SRF17, SRF20 to SRF27, and SRF30 to SRF37, and the final register of each stage is connected to the top register of the next stage. ing. Thereby, the shift registers SRF00 to SRF37 are connected in series.

  The shift registers are SRB00 to SRB08, SRB10 to SRB18, SRB20 to SRB28, and SRB30 to SRB38. The final register of each stage is connected to the first register of the next stage. Thus, the shift registers SRB00 to SRB38 are connected in series.

  The processing elements PEA00 to PEA03, PEA10 to PEA13, PEA20 to PEA23, and PEA30 to PEA33 are for integer precision block matching. Processing elements PEB00-PEB03, PEB10-PEB13, PEBA20-PEB23, and PEB30-PEB33 are for half-pixel precision block matching.

  Although each processing element is not shown, all outputs from the processing elements PEA00 to PEA03, PEA10 to PEA13, PEA20 to PEA23, and PEA30 to PEA33 are connected to the minimum value detection circuit 27 shown in FIG. The difference sum for one block is obtained. The same applies to the processing elements PEB00 to PEB03, PEB10 to PEB13, PEBA20 to PEB23, and PEB30 to PEB33 shown in FIG.

  The connection relationship between the processing elements PEA00 to PEA03 and PEB00 to PEB03 and the shift register will be described. The processing element PEA00 is connected to the shift registers SRF00 and SRR34, and calculates the difference between them. The processing element PEB01 is connected to the shift registers SRF00 and SRR33, and calculates the difference between them. The same applies to the relationship among the shift registers SRF01 to SRF03, SRR33 to SRR30, processing elements PEA01 to PEA03, and processing elements PEB01 to PEB03. The same applies to the other stages (columns).

  Thus, when the data of the search range of the previous frame is sequentially given to the shift register SRF37 and the data of the search range of the subsequent frame is sequentially given to the SRR 38, the data is transferred to the shift registers connected to the SRF 37 and SRR 38, respectively. The value of is sent. As will be described later, a comparison operation by the processing element is performed every predetermined number of shifts. The differences from the processing elements corresponding to each block are summed up to obtain the difference sum value in the block.

  The relationship between the shift process and the calculation process by the processing element will be described with reference to FIGS. FIG. 9 shows the front screen, and FIG. 10 shows the rear screen. The coordinates of the pixels stored in each register when the shift direction is arranged in the vertical direction and the data is shifted every clock in the horizontal direction are shown respectively. It is shown. In FIG. 9 and FIG. 10, only the numerical values are described as coordinates. Since the subsequent frame performs block matching with half-pixel accuracy, it is necessary to send dummy data. Therefore, the shift process is started for the previous frame with a delay of 3 clocks from the subsequent frame. Specifically, at time T4 when the pixel (4, 7) is stored in the shift register SRR38 of the subsequent frame, the pixel (0, 0) is stored in the shift register SRF37 of the previous frame. Each time the clock advances, the pixel data is transferred sequentially.

  At time Tp1, as shown in FIG. 9, the shift registers SRF00 to SRF03 include the pixels (0, 0) to (3, 0), and the shift registers SRF10 to SRF13 include the pixels (0 , 1) to (3, 1) are the shift registers SRF20 to SRF23, the pixels (0, 2) to (3, 2) are the shift registers SRF30 to SRF33, and the pixels (0, 3) to (3, 3 ) Is stored. Further, at time Tp1, as shown in FIG. 10, the shift registers for the rear frame include pixels (4, 4) to (7, 4) in shift registers SRR34 to SRR31, and pixels in shift registers SRR24 to SRR21. (4,5) to (7,5) are the shift registers SRR14 to SRR11, the pixels (4,6) to (7,6) are the shift registers SRR04 to SRR01, and the pixels (4,7) to (7 , 7) is stored. Therefore, when the comparison by the processing elements PEA00 to PEA03, PEA10 to PEA13, PEA20 to PEA23, and PEA30 to PEA33 (see FIG. 8) connected to these shift registers is performed at time Tp1, the previous processing shown in FIG. Block matching is performed between the blocks of pixels (0, 0) to (3, 3) in the frame and the blocks of subsequent frames (4, 4) to (7, 7).

  At time Tp1, as shown in FIG. 10, pixels (5, 4) to (NA) are included in the shift registers SRR33 to SRR30, and pixels (5) are included in the shift registers SRR23 to SRR20, as shown in FIG. , 5) to (NA), the pixels (5, 6) to (NA) are stored in the shift registers SRR13 to SRR10, and the pixels (5, 7) to (NA) are stored in the shift registers SRR03 to SRR00. Therefore, when the comparison by the processing elements PEB00 to PEB03, PEB10 to PEB13, PEBA20 to PEB23, and PEB30 to PEB33 (see FIG. 8) connected to these shift registers is performed at time Tp1, the previous processing shown in FIG. The block of pixels (0, 0) to pixels (3, 3) in the frame is compared with the subsequent frames (5, 4) to (8, 7). Thereby, block matching can be performed for a motion vector shifted by half a pixel in an intermediate frame.

  At time Tp2, which is one clock further from time Tp1, as shown in FIG. 9, pixels (1, 0) to (4, 0) are included in shift registers SRF00 to SRF03 as shift registers for the previous frame. SRF10 to SRF13 include pixels (1,1) to (4,1), shift registers SRF20 to SRF23, pixels (1,2) to (4,2), shift registers SRF30 to SRF33, pixel (1 , 3) to (4, 3) are stored. Further, at time Tp1, as shown in FIG. 10, the shift registers for the subsequent frames include pixels (3, 4) to (6, 4) in the shift registers SRR34 to SRR31 and pixels in the shift registers SRR24 to SRR21. (3, 5) to (6, 5) are the shift registers SRR14 to SRR11, the pixels (3, 6) to (6, 6) are the shift registers SRR04 to SRR01, and the pixels (3, 7) to (6 , 7) is stored. Therefore, when the comparison by the processing elements PEA00 to PEA03, PEA10 to PEA13, PEA20 to PEA23, and PEA30 to PEA33 (see FIG. 8) is performed at time Tp2, the pixel (1,0) of the previous frame shown in FIG. Block matching is performed between the block of the pixel (4, 3) and the blocks of the subsequent frames (3, 4) to (6, 7). Further, at time Tp1, as shown in FIG. 10, the shift registers for the rear frame include pixels (4, 4) to (7, 4) in shift registers SRR33 to SRR30, and pixels in shift registers SRR24 to SRR20. (4,5) to (7,5) are the shift registers SRR14 to SRR10, the pixels (4,6) to (7,6) are the shift registers SRR04 to SRR00, and the pixels (4,7) to (7 , 7) is stored. Therefore, when the comparison by the processing elements PEB00 to PEB03, PEB10 to PEB13, PEBA20 to PEB23, and PEB30 to PEB33 (see FIG. 8) is executed at time Tp2, the pixel (1,0) of the previous frame shown in FIG. The block of pixel (4, 3) and the subsequent frames (4, 4) to (7, 7) are compared. Thereby, block matching can be performed for a motion vector shifted by half a pixel in an intermediate frame. Hereinafter, also at time Tp3, as shown in FIGS. 11E and 11F, block matching is performed. The same applies to time Tp4. After time Tp4, a shift operation of 5 clocks is performed for the shift register of the subsequent block. For the shift register of the previous block, a shift operation of 4 clocks is performed. As a result, the line is shifted by one line in FIG. Specifically, at time Tp11, as shown in FIG. 9, the shift registers SRF00 to SRF03 include pixels (0, 1) to (3, 1) as shift registers SRF10 to SRF13. In addition, the pixels (0, 2) to (3, 2) are included in the shift registers SRF20 to SRF23, the pixels (0, 3) to (3, 3) are included in the shift registers SRF30 to SRF33, and the pixel (0, 4). ~ (3, 4) are stored. Further, at time Tp11, as shown in FIG. 10, the shift registers for the rear frame include pixels (4, 3) to (7, 3) in shift registers SRR34 to SRR31, and pixels in shift registers SRR24 to SRR21. (4,4) to (7,4) are the shift registers SRR14 to SRR11, the pixels (4,5) to (7,5) are the shift registers SRR04 to SRR01, and the pixels (4,6) to (7 , 6) is stored. Therefore, when the comparison by the processing elements PEA00 to PEA03, PEA10 to PEA13, PEA20 to PEA23, and PEA30 to PEA33 (see FIG. 8) is executed at time Tp11, the pixel (0, 1) of the previous frame shown in FIG. Block matching of the block of .about.pixel (3, 4) and the block of subsequent frames (4, 3) to (7, 6) is performed.

  Further, at time Tp11, as shown in FIG. 10, the shift registers for the subsequent frame include pixels (5, 3) to (NA) in the shift registers SRR33 to SRR30, and pixels (5) in the shift registers SRR24 to SRR20. , 4) to (NA), the pixels (5, 5) to (NA) are stored in the shift registers SRR14 to SRR10, and the pixels (5, 6) to (NA) are stored in the shift registers SRR04 to SRR00. Therefore, when the comparison by the processing element is executed at time Tp11, the block of the previous frame pixel (0, 1) to pixel (3, 4) and the rear frame (5, 3) to ( A comparison of 8,6) is made. Thereby, block matching can be performed for a motion vector shifted by half a pixel in an intermediate frame. Thereafter, block matching is performed in the same manner at times Tp12 to Tp14. Thereafter, similarly to the case of time Tp4, after time Tp14, a shift operation of 5 clocks is performed for the shift register of the subsequent block, and a shift operation of 4 clocks is performed for the shift register of the previous block.

  By repeating such processing, block matching with half-pixel accuracy is also performed on the data within the search range.

  As described above, the block is read from the search range memory by shifting the previous frame or the subsequent frame by one pixel, so that the intermediate image is generated without interpolating the block image shifted by a half pixel for the previous frame and the subsequent frame. A half-pixel motion vector for the frame is obtained. In addition, since the conventional half-pixel motion vector generates block images by interpolation for the previous frame and the subsequent frame, image blurring may occur, which may reduce the motion vector accuracy. In the present embodiment, such a problem can be solved.

  About the obtained motion vector, it is good also as a motion vector in the block of integer precision instead of the position which shifted | deviated the half pixel in the intermediate | middle frame. Even with such processing, the block position of the frame to be stored is shifted by a half pixel, but it is a slight shift, and the motion vector can be obtained with higher accuracy than the interpolation based on the block image by interpolation for the previous frame and the subsequent frame. This is because there are cases where it is done.

(4. Other embodiments)
In the present embodiment, the case where the rear frame is shifted in the x direction by one pixel has been described, but it may be shifted in the y direction. Further, although comparison is not performed at a position shifted by half a pixel in the y direction, this may also be performed. Further, not the rear frame but the previous frame may be shifted.

  Further, in the present embodiment, block matching with half pixel accuracy is performed in the intermediate frame without generating block images shifted by half pixels for the first reference frame and the second reference frame. Therefore, by providing a block image generation unit shifted by half a pixel, block matching with ¼ accuracy becomes possible. FIG. 13 shows a hardware configuration in the case of 1/4 pixel accuracy. It differs from FIG. 6 in that it has a selector 61 for selecting the minimum value among the values obtained from the 1/2 pixel accuracy calculator 51 and the minimum value detection circuits 53 and 55.

That is, for the blocks of the first and second reference frames, the block boundary line is located at the boundary between the pixels, and for the specific block in the intermediate frame, the block boundary line is ½ from the boundary between the pixels. Extracting block matching candidates by shifting the block of the first reference frame and / or the block of the second reference frame by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by ⁿ pixels. Can do. Here, n is a natural number of 1, 2,.

  In the present embodiment, the case where the present invention is applied to an apparatus for determining a motion vector for an intermediate frame has been described. However, motion compensation may be performed using the obtained motion vector.

  In the present embodiment, the case where the search range is 8 * 8 pixels and 1 block 4 * 4 has been described as an example, but the present invention is not limited to this.

  In the present embodiment, the case of the 2-parallel type has been described, but parallel processing such as a 4-parallel type is not particularly limited. Further, the shift register and the processing element can be configured as a systolic array type.

  In the present embodiment, the case where the shift process is performed pixel by pixel has been described, but the method of the shift calculation process is not particularly limited, such as shifting a plurality of pixels at once.

  In this embodiment, when the first reference frame and the intermediate frame are temporally the same as the second reference frame and the intermediate frame, that is, the first reference frame and the second reference frame As an example, the case where is geometrically symmetrical with respect to the intermediate frame has been described. However, the present invention is not limited to this, and the first reference frame and the intermediate frame are temporally different from the second reference frame and the intermediate frame. As a result, the second reference frame is different from the first reference frame. It may be located on an extension line of a virtual reference frame that is geometrically symmetrical with respect to the intermediate frame.

  The second reference frame is located on the extension line of the virtual reference frame that is geometrically symmetrical in time with respect to the first reference frame and the intermediate frame. For example, when the intermediate frame is time t = 0, the previous frame Is the time t = -1, and the subsequent frame is t = 2. In this case, the virtual reference frame is time t = 1. In such a symmetric search, if the search position of the previous frame is shifted by (1,1) vector with respect to the intermediate frame, it is shifted by (−2, −2) in the subsequent frame. Specifically, for example, in addition to shifting the previous frame by (1,1) vectors and further calculating the combination candidate when the subsequent frame is shifted by (-2, -2) vectors, 1,1) A combination candidate when the vector is shifted or a combination candidate when only the subsequent frame is shifted by (−2, −2) vector without shifting the previous frame is extracted as a candidate. In this way, the present invention can be similarly applied to a case where the relationship with the virtual reference frame is a relationship that is separated by an integer multiple in time.

  A case where temporal distances from the first reference frame and the second reference frame are different will be described with reference to FIG. The temporal relationship between the intermediate frame fc and the virtual reference frame fk when the virtual reference frame fk is defined in which the first reference frame f1 is geometrically symmetrical with respect to the specific block bx of the intermediate frame fc. The second reference frame f2 may be located on an extension line separated by n times. In this case, when the block for one pixel is shifted for the first reference frame, the block for n pixels may be moved for the second reference frame. For the matching candidate, only one of the block of the first reference frame or the second reference frame is shifted so that the block boundary line in the intermediate frame is set at a position shifted from the boundary between the pixels. What is necessary is just to extract a block. For example, when the intermediate frame is t = 0, the first reference frame is t = −1, and the second reference frame is t = 2, a virtual target frame fk where t = 1 is defined, The extraction calculation process may be sequentially performed so that the block of the second reference frame is located at the same position.

  When the temporal distance to such an intermediate frame is different, if a symmetric search is performed, if the search position of the previous screen is shifted by (1,1) vectors with respect to the intermediate image, -2, -2) Block candidates are extracted so as to be shifted.

  As described above, in the case of the relationship with the virtual reference frame that is separated by an integer multiple in time, not only the shifting timing but also the shifting pixel amount can be changed. Specifically, for example, combination candidates when only the subsequent frame is shifted by (-1, -1) vectors and combination candidates when the subsequent frame is shifted by (-2, -1) vectors are extracted.

  You may make it change both the timing to shift, and the amount to shift.

  Note that the shift amount in the intermediate frame is determined by the shifted pixels due to the temporal distance relationship between the previous frame and the subsequent frame. Therefore, when an intermediate frame with a desired shift amount is desired, the combination of the previous frame and the subsequent frame determined as the reference frame may be determined. For example, when it is desired to obtain a matching candidate shifted by 2/3 pixels in the intermediate image, in the above example, the previous frame is left as it is, and the subsequent frame is shifted by one image. When it is desired to obtain matching candidates that are shifted by 3 pixels, the previous frame may be shifted without changing the subsequent frame. In this way, a shifted block matching candidate can be extracted from the temporal distance between frames.

In the above-described embodiment, the pixel shifted block data generation unit has been described as generating block data with 2 * (1/2 ⁿ ) pixel accuracy. However, shifted pixels that generate block data with decimal x pixel accuracy are described. You may make it provide a production | generation means. In the present invention, since block matching candidates are extracted by shifting either the previous frame or the subsequent frame, block matching at a position finer than the shifted pixel generation means is possible.

  In the description of FIG. 14, the case where the first reference frame is the previous frame and the second reference frame is the subsequent frame has been described.

  In the above embodiment, each block may be configured by describing an algorithm performed in each block in a hardware description language.

  In the above embodiment, a case has been described in which all hardware processing is performed to enable parallel processing in order to realize the functions shown in FIG. However, the present invention is not limited to this, and a part thereof may be realized by software. In this case, a part of the program may be processed by an operating system (OS).

Claims (9)

A block-by-block motion vector calculation device that calculates a local motion vector, which is a block-by-block motion vector for these intermediate frames, from a first reference frame and a second reference frame,
Image storage means for storing images of the first reference frame and the second reference frame;
Search range determining means for determining a search range for the first reference frame and the second reference frame;
The image within the search range of the determined first reference frame and the second reference frame is a block composed of predetermined pixels, and the first reference frame centered on a specific block of the intermediate frame is the second reference. An extraction means for sequentially extracting blocks that are geometrically symmetrical to the blocks of the frame as block matching candidates;
A determination unit that performs block matching on the extracted block matching candidates and determines a combination that minimizes a difference as a motion vector of the specific block in the intermediate frame;
With
For the blocks of the first and second reference frames, the extracting means is such that a block boundary line is located at a pixel-pixel boundary, and for a specific block in the intermediate frame, the block boundary line is a pixel-pixel boundary. Block matching candidates obtained by shifting the block of the first reference frame and / or the block of the second reference frame by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by 1/2 ⁿ pixels from Extracting also,
A block-by-block motion vector computing device. In the motion vector computing device according to block of claim 1,
The extraction means includes pixel shifted block data generation means for generating block data with 2 * (1/2 ⁿ ) accuracy for the blocks of the first and second reference frames.
Where n is an integer,
A block-by-block motion vector computing device. The motion vector computing device for each block according to claim 2,
The n is 2, and the pixel-shifted block data generation means generates half-pixel precision block data for the blocks of the first and second reference frames;
A block-by-block motion vector computing device. In the motion vector computing device according to block of claim 1,
N is 3;
The pixel-shifted block data generating means generates block data in which the block boundary line is shifted by 1/4 pixel, 1/2 pixel, and 3/4 pixel from the pixel-to-pixel boundary for the blocks of the first and second reference frames. To do,
A block-by-block motion vector computing device. A block-specific motion vector calculation method for calculating a local motion vector, which is a block-specific motion vector for these intermediate frames, from a first reference frame and a second reference frame,
Given the images of the first reference frame and the second reference frame, determine a search range for the first reference frame and the second reference frame;
The image within the search range of the determined first reference frame and second reference frame is a block composed of predetermined pixels, and the first reference frame and the second reference frame are centered on a specific block of the intermediate frame. Blocks in which the blocks are geometrically symmetric are sequentially extracted as block matching candidates,
In the motion vector calculation method for each block, block matching is performed on the extracted block matching candidates, and a combination that minimizes the difference is determined as a motion vector of the specific block in the intermediate frame.
At the time of extracting the block matching candidate, the block boundary line is located at the boundary between the pixel and the pixel for the blocks of the first and second reference frames, and the block boundary line is the pixel and pixel for the specific block in the intermediate frame. Block matching in which the block of the first reference frame and / or the block of the second reference frame is shifted by 2 * (1/2 ⁿ ) pixels so as to be set at a position shifted by 1/2 ⁿ pixels from the boundary of Extracting candidates,
A block-by-block motion vector calculation method. A block-by-block motion vector calculation device that calculates a local motion vector, which is a block-by-block motion vector for these intermediate frames, from a first reference frame and a second reference frame,
Image storage means for storing images of the first reference frame and the second reference frame;
Search range determining means for determining a search range for the first reference frame and the second reference frame;
Means for extracting a block composed of predetermined pixels from the determined image within the search range of the first reference frame and the second reference frame, the first reference frame centering on a specific block of the intermediate frame; Block of the second reference frame located on an extension line separated by n times the temporal relationship between the intermediate frame and the virtual reference frame when defining a virtual reference frame having a geometrically symmetrical positional relationship Extraction means for moving the block by n pixels for the second reference frame and extracting it as a block matching candidate when the block is shifted by 1 pixel for the first reference frame,
A determination unit that performs block matching on the extracted block matching candidates and determines a combination that minimizes a difference as a motion vector of the specific block in the intermediate frame;
With
The extraction means sets a block of the first reference frame or a block shifted from the second reference frame as a block matching candidate so that a block boundary line in the intermediate frame is set at a position shifted from a pixel boundary. Extracting as a
A block-by-block motion vector computing device. The motion vector computing device for each block according to claim 6,
The extraction means includes pixel shifted block data generation means for generating decimal x-precision block data.
However, the decimal number x is arbitrary,
A block-by-block motion vector computing device. The motion vector computing device for each block according to claim 6,
The extraction unit extracts, as a block matching candidate, a block in which the block of the first reference frame or the second reference frame is not synchronized, or a block in which the block of the second reference frame is shifted by one pixel. Thus, the block of the first reference frame or the block shifted from the second reference frame is extracted as a block matching candidate so that the block boundary line in the intermediate frame is set at a position shifted from the pixel boundary. To do,
A block-by-block motion vector computing device. A block-by-block motion vector calculation method for calculating a local motion vector, which is a block-by-block motion vector for these intermediate frames, from a first reference frame and a second reference frame, comprising the following steps:
Determining a search range for the first reference frame and the second reference frame, given images of the first reference frame and the second reference frame;
Extracting a block composed of predetermined pixels from an image within the search range of the determined first reference frame and second reference frame, wherein the first reference frame is centered on a specific block of the intermediate frame; Block of the second reference frame located on an extension line separated by n times the temporal relationship between the intermediate frame and the virtual reference frame when defining a virtual reference frame having a geometrically symmetrical positional relationship An extraction step of moving a block by n pixels for the second reference frame and extracting it as a block matching candidate when the block is shifted by 1 pixel with respect to the first reference frame,
Performing block matching on the extracted block matching candidates, and determining a combination that minimizes a difference as a motion vector of the specific block in the intermediate frame;
With
In the extracting step, a block in which only one of the block of the first reference frame or the second reference frame is shifted so that the block boundary line in the intermediate frame is set at a position shifted from the boundary between pixels. Extracting as block matching candidates,
A block-by-block motion vector calculation method.

Images